OpenTelemetry Protocol with Apache Arrow

Summary

OpenTelemetry Protocol with Apache Arrow - Phase 2 AnnouncementWe are excited to announce the next phase of the OpenTelemetry Protocol with Apache Arrow project (OTel-Arrow). We began this project several years ago with the goal of bridging between OpenTelemetry data and the Apache Arrow ecosystem. Apache Arrow is a framework designed for zero-copy exchange of structured data between column-oriented data producers and consumers.We believe that having OpenTelemetry data accessible to external systems through Apache Arrow will lead to powerful integrations, with the potential for new telemetry systems and applications to emerge. For large streams of telemetry, we know that column-oriented data handling is substantially more efficient, with improved data compression and performance.We are choosing to investigate this phase of the project in Rust. With the help of the OpenTelemetry Governance Committee, we’ve defined a project scope that entails studying the potential for Rust-based OpenTelemtry pipelines without “being” a Collector. We will investigate both the performance of Rust pipelines as well as how to successfully integrate our work with the OpenTelemetry Collector’s Golang-based ecosystem.This project aims to answer our original hypothesis. We are investigating what happens and what is possible if we use a zero-copy, column-oriented paradigm from end to end, starting in the SDK and carried through a pipeline.We couldn’t help but notice: the Rust ecosystem around Apache Arrow is large and vibrant, and we think it would be a missed opportunity not to explore this space. We are particularly interested in connecting the Apache DataFusion library with OpenTelemetry pipelines, and we believe that OpenTelemetry users will benefit from more direct integrations with their data lake.We are motivated to build an end-to-end OTAP pipeline in Rust, where telemetry data is placed into Arrow record batches as it is produced. The OTel-Arrow project expects to align closely with...