Just make it scale: An Aurora DSQL story

Summary

At re:Invent we announced Aurora DSQL, and since then I’ve had many conversations with builders about what this means for database engineering. What’s particularly interesting isn’t just the technology itself, but the journey that got us here. I’ve been wanting to dive deeper into this story, to share not just the what, but the how and why behind DSQL’s development. Then, a few weeks ago, at our internal developer conference — DevCon — I watched a talk from two of our senior principal engineers (PEs) on building DSQL (a project that started 100% in JVM and finished 100% Rust). After the presentation, I asked Niko Matsakis and Marc Bowes if they’d be willing to work with me to turn their insights into a deeper exploration of DSQL’s development. They not only agreed, but offered to help explain some of the more technically complex parts of the story.In the blog that follows, Niko and Marc provide deep technical insights on Rust and how we’ve used it to build DSQL. It’s an interesting story on the pursuit of engineering efficiency and why it’s so important to question past decisions – even if they’ve worked very well in the past.Note from the authorBefore we get into it, a quick but important note. This was (and continues to be) an ambitious project that requires a tremendous amount of expertise in everything from storage to control plane engineering. Throughout this write-up we've incorporated the learnings and wisdom of many of the Principal and Sr. Principal Engineers that brought DSQL to life. I hope you enjoy reading this as much as I have.Special thanks to: Marc Brooker, Marc Bowes, Niko Matsakis, James Morle, Mike Hershey, Zak van der Merwe, Gourav Roy, Matthys Styrdom.A brief timeline of purpose-built databases at AWS Since the early days of AWS, the needs of our customers have grown more varied — and in many cases, more urgent. What started with a push to make traditional relational databases easier to manage with the launch of Amazon RDS in 2009 quickly expan...