Data engineering and software engineering are converging

Summary

TL;DR: · If you’re an engineer building realtime analytics or AI-powered features, you need the right data infrastructure coupled with the right developer experience (DX). · A great DX for data infrastructure should empower both software devs and data engineers, while taking inspiration from the best of modern web development (git-native, local-first, everything as code, CI/CD friendly, etc). · MooseStack by 514 offers a fully open source implementation of a DX layer for ClickHouse. Data engineering and software engineering are converging. For years, data infrastructure was built for analysts. Warehouses, lakes, BI dashboards—all SQL-first, point-and-click workflows. But today, analytics isn’t just about reporting or data science. Real-time data is at the center of modern user experiences and AI-readiness. SaaS apps are surfacing analytics and AI directly in their UX to drive adoption, engagement, and retention. Enterprises are accelerating their business with AI-powered automations for faster insights, predictions, and operations. Engineering teams are on the hook to ship data-backed functionality with the same discipline as any other application code. If you’re coming from the software engineering world, you probably start with a transactional database like Postgres, MySQL or Mongo. The tooling is great, and the developer experience is mature—but those systems are built for transactions, not analytics. As cardinality and scan sizes grow, queries bog down. Dashboards spin. AI chat slows to a crawl. Alternatively, if you’re coming from the data engineering world, you’re probably on managed analytics platforms like Snowflake, BigQuery, or Databricks. These work well for batch ETL and reporting, but fall short when you need freshness, concurrency, or sub-second response times. They’re also full of rough edges for developers. You don’t get a real local environment. Iteration cycles are slow. They’re just not built for the modern software development lifecycle. So we’re...