What makes 5% of AI agents work in production?

Summary

Most founders think they’re building AI products. They’re actually building context selection systems.This Monday, I moderated a panel in San Francisco with engineers and ML leads from Uber, WisdomAI, EvenUp, and Datastrato. The event, Beyond the Prompt, drew 600+ registrants, mostly founders, engineers, and early AI product builders.We weren’t there to rehash prompt engineering tips.We talked about context engineering, inference stack design, and what it takes to scale agentic systems inside enterprise environments. If “prompting” is the tip of the iceberg, this panel dove into the cold, complex mass underneath: context selection, semantic layers, memory orchestration, governance, and multi-model routing.Here’s the reality check: One panelist mentioned that 95% of AI agent deployments fail in production. Not because the models aren’t smart enough, but because the scaffolding around them, context engineering, security, memory design, isn’t there yet.One metaphor from the night stuck with me:“The base models are the soil; context is the seed.”I’ve been obsessed with semantic layers for a while now, not because they’re flashy, but because they’re where founders quietly build trust, utility, and differentiation into LLM systems. I’ve seen too many teams conflate prompting with product. This panel felt like a moment where the real engineering work started getting its due.Below are the takeaways, not just quotes, but patterns I see repeating in serious AI teams. If you’re building at the infra, tooling, or vertical AI layer, this is the scaffolding you’ll need to get right.Several panelists echoed the same insight: fine-tuning is rarely necessary. Retrieval-augmented generation (RAG), when done well, is enough. But most RAG systems today are too naive.The failure mode:Index everything → retrieve too much → confuse the modelIndex too little → starve the model of signalMix structured + unstructured data → break embeddings or flatten key schemaSo what does advanced context ...