Why Metaflow?

Summary

Why MetaflowIn the past, data scientists and ML engineers had to rely on a medley of point solutions and custom systems to build ML and data science applications.Applications can be built quicker and more robustly if they stand on a common, human-friendly foundation. But what should the foundation cover?Data may come in different shapes and sizes and may be loaded from various data stores. However, no matter what data is used, accessing and processing it shouldn't be too cumbersome.Some applications require a tremendous amount of compute power - think computer vision - while some do with less. Regardless of the scale, all applications need to perform computation reliably. Thanks to cloud computing, data scientists and ML engineers should be able to utilize elastic compute resources without friction.Consider an application that loads data, transforms it, trains a bunch of models, chooses the best performing one, runs inference, and writes the results to a database. Multi-steps workflows like this are a norm in ML. A workflow orchestrator is needed to make sure all steps get executed in order, on time.Rarely a real-world application is built and deployed only once. Instead, a typical application is built gradually, through contributions by many people. The project needs to be tracked, organized, and versioned, which enables systematic and continuous improvement over time.To produce real business value, DS/ML applications can't live in a walled garden. They must be integrated with the surrounding systems seamlessly: Some applications enhance data in a database, some power internal dashboards or microservices, whereas some power user-facing products. There are many such ways to deploy ML in production. The more valuable the application, the more carefully it needs to be operated and monitored as well.For many data scientists and ML engineers, the most rewarding part of the project is modeling. Using their domain knowledge and expertise, the modeler should be able to cho...