LLM Workflows then Agents: Getting Started with Apache Airflow

Summary

This repository contains an SDK for working with LLMs from Apache Airflow, based on Pydantic AI. It allows users to call LLMs and orchestrate agent calls directly within their Airflow pipelines using decorator-based tasks. The SDK leverages the familiar Airflow @task syntax with extensions like @task.llm , @task.llm_branch , and @task.agent . To get started, check out the examples repository here, which offers a full local Airflow instance with the AI SDK installed and 5 example pipelines. To run this locally, run: git clone https://github.com/astronomer/ai-sdk-examples.git cd ai-sdk-examples astro dev start If you don't have the Astro CLI installed, run brew install astro (or see other options here). If you already have Airflow running, you can also install the package with any optional dependencies you need: pip install airflow-ai-sdk[openai,duckduckgo] Note that installing the package with no optional dependencies will install the slim version of the package, which does not include any LLM models or tools. The available optional packages are listed here. While this SDK offers the optional dependencies for convenience sake, you can also install the optional dependencies from Pydantic AI directly. Table of Contents: Features LLM tasks with @task.llm : Define tasks that call language models (e.g. GPT-3.5-turbo) to process text. Define tasks that call language models (e.g. GPT-3.5-turbo) to process text. Agent tasks with @task.agent : Orchestrate multi-step AI reasoning by leveraging custom tools. Orchestrate multi-step AI reasoning by leveraging custom tools. Automatic output parsing: Use function type hints (including Pydantic models) to automatically parse and validate LLM outputs. Use function type hints (including Pydantic models) to automatically parse and validate LLM outputs. Branching with @task.llm_branch : Change the control flow of a DAG based on the output of an LLM. Change the control flow of a DAG based on the output of an LLM. Model support: Support f...