F5 collaborates with Microsoft on Phase 2 of OpenTelemetry with Apache Arrow

Phase 1 of OpenTelemetry with Apache Arrow set out with ambitious goals. By introducing innovations to the OpenTelemetry Protocol (OTLP), this phase optimized telemetry data transport like never before. Compression rates improved by 30%-70% compared to the standard OTLP protocol, a remarkable achievement that laid the foundation for the upcoming work.

But Phase 1 was just the tip of the iceberg. It proved the power of combining OpenTelemetry and Apache Arrow, and set the stage for an even larger mission: transforming the entire OpenTelemetry pipeline into a high-performance, columnar data processing powerhouse.

In Phase 2, F5 and Microsoft are diving deeper into performance optimization by developing a fully native OpenTelemetry pipeline powered by Rust and Apache Arrow with the goal of building Rust pipelines for the Golang collector. This will lead to dramatic improvements in CPU utilization, memory efficiency, throughput, and latency—all of which contribute to a more efficient data processing ecosystem.

Our goal is also to balance optimal data compression, accelerated data processing, and efficient integration with modern, columnar-oriented observability, setting this initiative apart from other projects primarily focused on compression alone, such as STEF. 

The decision to go "all in" with Rust has stirred excitement—and a bit of controversy—among the technical community. After all, OpenTelemetry’s Collector project is traditionally built using Golang. So why Rust?

Rust and Apache Arrow are like best friends in the data processing world. Both technologies complement each other in profound ways, making Rust the ideal choice for Phase 2 development. Here are just a few reasons: 

• Memory safety and performance without Garbage Collector: Rust’s strong commitment to memory safety ensures that data pipelines are both robust and secure, minimizing risks like data corruption or segmentation faults. Combine this with Rust’s performance characteristics—low overhead, zero-cost abstractions—and you have a perfect match for efficiently handling the high-speed, high-volume telemetry data embraced by Apache Arrow.
• Columnar data processing: Apache Arrow is the de facto framework for zero-copy, column-oriented data processing. It is widely adopted across modern data lakes, supporting systems like Parquet, Delta Lake, and Iceberg. Native integration with Arrow means you can exchange data seamlessly and efficiently with powerful analytics engines like Apache DataFusion and Arrow-enabled data lakes.
• Zero-copy design: Apache Arrow’s zero-copy paradigm aligns beautifully with Rust’s emphasis on direct memory access. Together, these technologies eliminate serialization and deserialization overhead, making it possible to handle large data streams efficiently without wasting CPU cycles.
• A thriving ecosystem: The Rust ecosystem around Apache Arrow is thriving. By choosing Rust for Phase 2, F5 and Microsoft tap into this vibrant community, unlocking integrations with powerful libraries and frameworks, such as Apache DataFusion. These connections open doors to new applications, especially integrations with large-scale data lakes used for analytics and machine learning. Although this project is still in its early stages, we’ve already attracted interest from several companies specializing in observability-focused databases.

With Rust and Apache Arrow at the helm, OpenTelemetry with Apache Arrow isn’t merely tackling performance metrics—it’s setting the stage for broader adoption across telemetry and data ecosystems.

Imagine a world where telemetry pipelines are seamlessly embeddable in data lakes, offering direct connections to column-oriented analytics platforms. This end-to-end, zero-copy pipeline will allow telemetry data to flow seamlessly from production to downstream systems, unlocking integrations previously unimaginable within OpenTelemetry.

By safeguarding embeddability, prioritizing strict memory controls, and enabling thread-per-core runtime support, Phase 2 ensures that this Rust-based paradigm will not only outperform expectations but also remain accessible to developers and enterprises alike

Despite the buzz around Rust, Phase 2 doesn’t mean leaving Golang behind. In fact, F5 and Microsoft are taking careful steps to ensure compatibility between Go and Rust pipelines. Adapters and existing OpenTelemetry Collector components written in Golang will continue to be supported and maintained. This ensures interoperability, enabling organizations to choose the best toolset for their workflows without being locked into a single language.

This unifying approach aligns with OpenTelemetry’s commitment to growing the ecosystem, not fracturing it. By bridging Rust and Go pipelines as part of an interconnected workflow, the collaboration ensures that OpenTelemetry remains accessible to its diverse user base.

The OpenTelemetry Governance Committee has endorsed Phase 2, recognizing the importance of exploring Rust-based innovations while maintaining harmony within the broader OpenTelemetry ecosystem. Contributions from F5, like the “Beaubourg” library have already provided a strong foundation to build upon.

As F5 and Microsoft collaborate on this endeavor, they’re not just optimizing performance—they’re opening doors to exciting possibilities for the telemetry world. This investment in innovation speaks volumes about OpenTelemetry’s commitment to remaining on the cutting edge of open-source telemetry systems.