Bitter lessons building AI products

Summary

I was catching up with a friend who had also been building AI products for a few years now. We were lamenting (and laughing) on the graveyard of seemingly-failed projects that now have turned into rapid successes. AI features that were multi-quarter grinds a couple years ago can now be shipped in a matter of weeks.We realized that we had just actually learned "the Bitter Lesson" - building AI products edition.The Bitter Lesson - from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The version of this we – and many others – have learned is that you shouldn't try to make AI work for your existing roadmap with fancy clever engineering – much of it will be obsolete with the next major model upgrade (trained with more compute and more data). Instead, you should aim to understand model capabilities and how you can best pivot your roadmap accordingly.Some questions I now ask:What unique value are we delivering that others can't easily replicate?How can we leverage evolving model capabilities to take advantage of that?Are we building something that can take advantage of better models, or are we building around existing model deficiencies?2+ years into building AI features, we’ve also changed a few things about how we build:Ditching demos - we have customers use alpha features early to validate if the feature is even viable.Spot capability shifts - what does a new model release unlock for you and your team?Kill projects faster - don’t let sunk cost fallacy keep you building if you realize something isn’t working.But this wasn’t obvious - and we made a lot of mistakes to get here.The first failed attempt at building the Notebook AgentThe Notebook Agent has been a huge hit with customers, but what folks may not realize is that the core idea of what we built was an idea originally from early 2023. Our team was convinced that this was the future - and we were right, just a bit too early.We spent ...