LLM Observability in the Wild – Why OpenTelemetry Should Be the Standard

Summary

A few days ago I hosted a live conversation with Pranav, co-founder of Chatwoot, about issues his team was running into with LLM observability.The short version: building, debugging, and improving AI agents in production gets messy fast. There's multiple competing standards for default libraries for LLM observability. And many such libraries like OpenInference which claim to be based on OpenTelemetry don't strictly adhere to it's conventions. This introduces problems for users who are trying to get better observability across their stack.Here’s a write-up of what we covered and what I think it means for anyone shipping LLM features into real products. Feel free to watch the complete videoVIDEOPranav and I go way back to our YC days in 2021, and it's always interesting to see how our paths have evolved. Chatwoot has built something really compelling - an open-source customer support platform that unifies conversations across every channel you can imagine: live chat, email, WhatsApp, social media, you name it. All in a single dashboard.But here's where it gets interesting. They've built an AI agent called "Captain" that can work across all these channels. You build the logic once, and it can handle support queries whether they come through email, live chat, or WhatsApp. Pretty neat, right?The problem started showing up in production in the most unexpected ways. Sometimes their AI would randomly respond in Spanish when it absolutely shouldn't. Other times, responses just weren't quite right, and they had no visibility into why.This is where Pranav's journey into LLM observability began, it mirrors what I've been seeing across many companies building LLM applications. You need to understand:What documents were retrieved for a RAG query?Which tool calls were made?What was the exact input and output at each step?Why did the AI make certain decisions?Without this visibility, you're essentially flying blind in production.The Standards ProblemHere's where things get really i...