F5 is collaborating with Microsoft on Phase 2 of the open-source OpenTelemetry project with Apache Arrow (OTAP). As part of this project, OpenTelemetry data will be consumed, processed, and exported using Apache Arrow record batches from the SDK to the agent, collector, and backend. Building upon the success of Phase 1, which saw dramatic improvements in telemetry data compression rates, this next chapter aims to push the boundaries of OpenTelemetry’s even further with a Rust-based, Arrow-native data processing pipeline.
F5 and Microsoft’s collaboration on Phase 2 of the OpenTelemetry with Apache Arrow project will develop a high-performance telemetry pipeline integrated with the Apache Arrow ecosystem, with a range of applications for small- and large-scale uses. We are creating new telemetry pipeline libraries in the Rust language, and we aim to deliver this functionality through integration with the existing OpenTelemetry Collector.
This collaboration is poised to deliver innovative performance enhancements for telemetry systems, marking a significant milestone not only for OpenTelemetry but also for the broader Apache Arrow and Rust ecosystems. We’re confident in our approach and are now actively implementing this new Rust-based pipeline and conducting extensive performance benchmarking to clearly demonstrate the advancements.
Phase 1 of OpenTelemetry with Apache Arrow set out with ambitious goals. By introducing innovations to the OpenTelemetry Protocol (OTLP), this phase optimized telemetry data transport like never before. Compression rates improved by 30%-70% compared to the standard OTLP protocol, a remarkable achievement that laid the foundation for the upcoming work.
But Phase 1 was just the tip of the iceberg. It proved the power of combining OpenTelemetry and Apache Arrow, and set the stage for an even larger mission: transforming the entire OpenTelemetry pipeline into a high-performance, columnar data processing powerhouse.
In Phase 2, F5 and Microsoft are diving deeper into performance optimization by developing a fully native OpenTelemetry pipeline powered by Rust and Apache Arrow with the goal of building Rust pipelines for the Golang collector. This will lead to dramatic improvements in CPU utilization, memory efficiency, throughput, and latency—all of which contribute to a more efficient data processing ecosystem.
Our goal is also to balance optimal data compression, accelerated data processing, and efficient integration with modern, columnar-oriented observability, setting this initiative apart from other projects primarily focused on compression alone, such as STEF.
The decision to go "all in" with Rust has stirred excitement—and a bit of controversy—among the technical community. After all, OpenTelemetry’s Collector project is traditionally built using Golang. So why Rust?
Rust and Apache Arrow are like best friends in the data processing world. Both technologies complement each other in profound ways, making Rust the ideal choice for Phase 2 development. Here are just a few reasons:
With Rust and Apache Arrow at the helm, OpenTelemetry with Apache Arrow isn’t merely tackling performance metrics—it’s setting the stage for broader adoption across telemetry and data ecosystems.
Imagine a world where telemetry pipelines are seamlessly embeddable in data lakes, offering direct connections to column-oriented analytics platforms. This end-to-end, zero-copy pipeline will allow telemetry data to flow seamlessly from production to downstream systems, unlocking integrations previously unimaginable within OpenTelemetry.
By safeguarding embeddability, prioritizing strict memory controls, and enabling thread-per-core runtime support, Phase 2 ensures that this Rust-based paradigm will not only outperform expectations but also remain accessible to developers and enterprises alike
Despite the buzz around Rust, Phase 2 doesn’t mean leaving Golang behind. In fact, F5 and Microsoft are taking careful steps to ensure compatibility between Go and Rust pipelines. Adapters and existing OpenTelemetry Collector components written in Golang will continue to be supported and maintained. This ensures interoperability, enabling organizations to choose the best toolset for their workflows without being locked into a single language.
This unifying approach aligns with OpenTelemetry’s commitment to growing the ecosystem, not fracturing it. By bridging Rust and Go pipelines as part of an interconnected workflow, the collaboration ensures that OpenTelemetry remains accessible to its diverse user base.
The OpenTelemetry Governance Committee has endorsed Phase 2, recognizing the importance of exploring Rust-based innovations while maintaining harmony within the broader OpenTelemetry ecosystem. Contributions from F5, like the “Beaubourg” library have already provided a strong foundation to build upon.
As F5 and Microsoft collaborate on this endeavor, they’re not just optimizing performance—they’re opening doors to exciting possibilities for the telemetry world. This investment in innovation speaks volumes about OpenTelemetry’s commitment to remaining on the cutting edge of open-source telemetry systems.
For those tracking the progress of OpenTelemetry with Apache Arrow, Phase 2 represents a leap forward in efficiency, scalability, and flexibility. By using Rust and Apache Arrow to their fullest potential, this collaboration has the potential to redefine how telemetry data is produced, processed, and consumed.
As the technical community awaits results from this ambitious phase, one thing is clear—OpenTelemetry with Apache Arrow is helping shape the future of distributed observability systems.
Stay updated for more insights about Phase 2 developments and how this project is paving the way to a high-performance, columnar data future. Let’s build big—and build it fast.
F5 and Microsoft’s collaboration encapsulates the spirit of open-source innovation, and we’re excited to see how these efforts elevate OpenTelemetry for us all. Together, we aren’t just adopting modern tools—we're redefining what’s possible. To find out more, read the Microsoft blog post.