Everything LLM teams need, to build something your users love. Get started here.

The Problem ❌

Monitoring and testing LLMs is notoriously challenging. Specifically:

1. Product insights & monitoring - how can I see how different customers are using my LLM stack?

   What are users talking to my bot about? What questions are users asking my search system?

2. Testing - I just made a change to my code. How can I confirm it’s making the change I intended? How can I make sure I haven’t introduced unexpected side effects?

   Will making my bot faster make it less reliable? Will switching LLM providers make our system too slow for our customers

3. E2E analytics - Product 🤝 LLM metrics. How can I see how changes to my LLM stack are impacting my primary product metrics?

   I just changed my RAG pipeline, is that improving my conversion rates? Which prompt gives us the best activation rate? What are high vs. low value customers talking to my bot about?

4. Errors - I’m using 4+ providers across my LLM stack, each with its own logging system and dashboard. How can I quickly view the root cause of my errors and get all the context I need in one place?

We’ve seen teams waste weeks of dev time building internal tools to solve these problems instead of building what their users want.

The Solution 🛠️

Lytix brings insights, testing, and E2E analytics to your LLM stack with minimal changes to your existing codebase.

Product insights 🔎 & monitoring 👀

1. Add a tracer function to any aspect of your LLM stack you’d like monitored

   

2. [Optional] - in the Lytix dashboard, define some ‘themes’ you’d like Lytix to start tagging your sessions as. For ex. ‘questions about math’

   

3. If you do not define an intent - Lytix will auto-tag each session with an intent that best summarizes the session.

4. You can always explore previous sessions and re-configure your themes to change how they appear in your analytics stack

E2E analytics 📊 ( for ex. PostHog 🦔)

1. Save your PostHog API key in your Lytix account (under settings)

2. Whenever Lytix sees a new session, we’ll tag it (with the intents you’ve configured), and import them into Posthog with all the associated metadata

3. In PostHog, you’ll see a “lytix_event” → this event describes your users’ sessions with your LLM stack. Bring this into your funnel analysis, retention charts, etc., to include your LLM sessions in your product insights

   

Errors 🚨

1. Wherever you throw an error, use the new Lytix LError class.

   async def backgroundProcess():
       logger = LLogger("background-logger", {"userId": '124'})
       logger.info("Some process is starting")
       raise LError("An unexpected error")
   
   

2. Anytime the error is thrown in production, you’ll find all relevant logs and metadata in your Lytix dashboard

   

To try it out yourself, head over to https://docs.lytix.co/introduction. You’ll also get 2 founders on deck as analysts to ask any questions about how your customers are using your products 🚀.

And get in touch if you:

1. Are about to launch - and would like to chat about best practices
2. Recently launched - and are thinking about effective analytics and monitoring
3. Or have a hair-on-fire problem you need to be taken care of!