BLOG

F5 collaborates with Microsoft on Phase 2 of OpenTelemetry with Apache Arrow

Laurent Querel Thumbnail
Laurent Querel
Published June 09, 2025

F5 is collaborating with Microsoft on Phase 2 of the open-source OpenTelemetry project with Apache Arrow (OTAP). As part of this project, OpenTelemetry data will be consumed, processed, and exported using Apache Arrow record batches from the SDK to the agent, collector, and backend. Building upon the success of Phase 1, which saw dramatic improvements in telemetry data compression rates, this next chapter aims to push the boundaries of OpenTelemetry’s even further with a Rust-based, Arrow-native data processing pipeline. 

F5 and Microsoft’s collaboration on Phase 2 of the OpenTelemetry with Apache Arrow project will develop a high-performance telemetry pipeline integrated with the Apache Arrow ecosystem, with a range of applications for small- and large-scale uses. We are creating new telemetry pipeline libraries in the Rust language, and we aim to deliver this functionality through integration with the existing OpenTelemetry Collector.

This collaboration is poised to deliver innovative performance enhancements for telemetry systems, marking a significant milestone not only for OpenTelemetry but also for the broader Apache Arrow and Rust ecosystems. We’re confident in our approach and are now actively implementing this new Rust-based pipeline and conducting extensive performance benchmarking to clearly demonstrate the advancements.  

A look back: Phase 1 success

Phase 1 of OpenTelemetry with Apache Arrow set out with ambitious goals. By introducing innovations to the OpenTelemetry Protocol (OTLP), this phase optimized telemetry data transport like never before. Compression rates improved by 30%-70% compared to the standard OTLP protocol, a remarkable achievement that laid the foundation for the upcoming work.

But Phase 1 was just the tip of the iceberg. It proved the power of combining OpenTelemetry and Apache Arrow, and set the stage for an even larger mission: transforming the entire OpenTelemetry pipeline into a high-performance, columnar data processing powerhouse.

Phase 2: Enter Rust and Arrow

In Phase 2, F5 and Microsoft are diving deeper into performance optimization by developing a fully native OpenTelemetry pipeline powered by Rust and Apache Arrow with the goal of building Rust pipelines for the Golang collector. This will lead to dramatic improvements in CPU utilization, memory efficiency, throughput, and latency—all of which contribute to a more efficient data processing ecosystem.

Our goal is also to balance optimal data compression, accelerated data processing, and efficient integration with modern, columnar-oriented observability, setting this initiative apart from other projects primarily focused on compression alone, such as STEF. 

The decision to go "all in" with Rust has stirred excitement—and a bit of controversy—among the technical community. After all, OpenTelemetry’s Collector project is traditionally built using Golang. So why Rust?

Why Rust and Apache Arrow Are the perfect pair

Rust and Apache Arrow are like best friends in the data processing world. Both technologies complement each other in profound ways, making Rust the ideal choice for Phase 2 development. Here are just a few reasons: 

  • Memory safety and performance without Garbage Collector: Rust’s strong commitment to memory safety ensures that data pipelines are both robust and secure, minimizing risks like data corruption or segmentation faults. Combine this with Rust’s performance characteristics—low overhead, zero-cost abstractions—and you have a perfect match for efficiently handling the high-speed, high-volume telemetry data embraced by Apache Arrow.
  • Columnar data processing: Apache Arrow is the de facto framework for zero-copy, column-oriented data processing. It is widely adopted across modern data lakes, supporting systems like Parquet, Delta Lake, and Iceberg. Native integration with Arrow means you can exchange data seamlessly and efficiently with powerful analytics engines like Apache DataFusion and Arrow-enabled data lakes.
  • Zero-copy design: Apache Arrow’s zero-copy paradigm aligns beautifully with Rust’s emphasis on direct memory access. Together, these technologies eliminate serialization and deserialization overhead, making it possible to handle large data streams efficiently without wasting CPU cycles.
  • A thriving ecosystem: The Rust ecosystem around Apache Arrow is thriving. By choosing Rust for Phase 2, F5 and Microsoft tap into this vibrant community, unlocking integrations with powerful libraries and frameworks, such as Apache DataFusion. These connections open doors to new applications, especially integrations with large-scale data lakes used for analytics and machine learning. Although this project is still in its early stages, we’ve already attracted interest from several companies specializing in observability-focused databases.

Beyond performance: Expanded horizons

With Rust and Apache Arrow at the helm, OpenTelemetry with Apache Arrow isn’t merely tackling performance metrics—it’s setting the stage for broader adoption across telemetry and data ecosystems.

Imagine a world where telemetry pipelines are seamlessly embeddable in data lakes, offering direct connections to column-oriented analytics platforms. This end-to-end, zero-copy pipeline will allow telemetry data to flow seamlessly from production to downstream systems, unlocking integrations previously unimaginable within OpenTelemetry.

By safeguarding embeddability, prioritizing strict memory controls, and enabling thread-per-core runtime support, Phase 2 ensures that this Rust-based paradigm will not only outperform expectations but also remain accessible to developers and enterprises alike

A unified approach: Rust meets Go

Despite the buzz around Rust, Phase 2 doesn’t mean leaving Golang behind. In fact, F5 and Microsoft are taking careful steps to ensure compatibility between Go and Rust pipelines. Adapters and existing OpenTelemetry Collector components written in Golang will continue to be supported and maintained. This ensures interoperability, enabling organizations to choose the best toolset for their workflows without being locked into a single language.

This unifying approach aligns with OpenTelemetry’s commitment to growing the ecosystem, not fracturing it. By bridging Rust and Go pipelines as part of an interconnected workflow, the collaboration ensures that OpenTelemetry remains accessible to its diverse user base.

Industry endorsement and contributions

The OpenTelemetry Governance Committee has endorsed Phase 2, recognizing the importance of exploring Rust-based innovations while maintaining harmony within the broader OpenTelemetry ecosystem. Contributions from F5, like the “Beaubourg” library have already provided a strong foundation to build upon.

As F5 and Microsoft collaborate on this endeavor, they’re not just optimizing performance—they’re opening doors to exciting possibilities for the telemetry world. This investment in innovation speaks volumes about OpenTelemetry’s commitment to remaining on the cutting edge of open-source telemetry systems.

What’s next?

For those tracking the progress of OpenTelemetry with Apache Arrow, Phase 2 represents a leap forward in efficiency, scalability, and flexibility. By using Rust and Apache Arrow to their fullest potential, this collaboration has the potential to redefine how telemetry data is produced, processed, and consumed.

As the technical community awaits results from this ambitious phase, one thing is clear—OpenTelemetry with Apache Arrow is helping shape the future of distributed observability systems.

Stay updated for more insights about Phase 2 developments and how this project is paving the way to a high-performance, columnar data future. Let’s build big—and build it fast.

F5 and Microsoft’s collaboration encapsulates the spirit of open-source innovation, and we’re excited to see how these efforts elevate OpenTelemetry for us all. Together, we aren’t just adopting modern tools—we're redefining what’s possible. To find out more, read the Microsoft blog post