Airport for DuckDB

Summary

The Airport extension brings Arrow Flight support to DuckDB, enabling DuckDB to query, modify, and store data via Arrow Flight servers. A DuckDB extension is a plugin that expands DuckDB’s core functionality by adding new capabilities. To understand the rationale behind the development of this extension, check out the motivation for creating the extension. Getting started The Airport extension is a DuckDB community extension. To install it, run the following SQL inside DuckDB: INSTALL airport FROM community; To load the extension you can then execute: If you wish to build the extension from source see these instructions. What can I do with the Airport extension that I can’t do with DuckDB now? With the Airport extension you can: Query data that DuckDB can’t normally access—either because it’s non-tabular or in an unsupported format. Even external APIs. It all depends what the server allows. Add custom scalar or table returning SQL functions not available in DuckDB. Provide User Defined Functions (UDFs) for DuckDB that execute remotely. Serve data with fine-grained access control, filtering both rows and columns based on user permissions. Access and provide Data-as-a-Service. What is Arrow Flight? From the Apache Arrow Documentation: Arrow Flight is an RPC framework for high-performance data services based on Apache Arrow and is built on top of gRPC and the Arrow IPC format. Flight is organized around streams of Arrow record batches, being either downloaded from or uploaded to another service. A set of metadata methods offers discovery and introspection of streams, as well as the ability to implement application-specific methods. Methods and message wire formats are defined by Protobuf, enabling interoperability with clients that may support gRPC and Arrow separately, but not Flight. However, Flight implementations include further optimizations to avoid overhead in usage of Protobuf (mostly around avoiding excessive memory copies). What is an Apache Arrow “Flight”? A...