Collaboration with OpenTelemetry

[felixge]

Hello,

my name is Felix and I'm writing you in my capacity as a maintainer in the OpenTelemetry Profiling SIG.

I read your announcement post of perforator with great interest. It's a really fantastic article covering a lot of ground and novel ideas (e.g. minicores).

Towards the end of your article you mention:

We plan to contribute our experience to developing a universal profile format within OpenTelemetry.

I'd like to invite you to join one of our upcoming SIG meetings and/or our online discussion channels: https://github.com/open-telemetry/community#sig-profiles

As you're probably aware, the profiling SIG is also working on an ebpf profiler, so perhaps there are opportunities for collaborating on the core profiling technologies in addition to the profiling format ideas you mentioned.

Meanwhile: Thank you for open sourcing perforator and advancing the state of art in profiling! ❤

[BigRedEye]

Thank you very much for your kind words and invitation to collaborate!

We have been following the excellent work on the eBPF profiler with great interest. By the time it was released as open source in 2024, we had already developed significant portions of the Perforator project, so we believe our work, particularly features like sPGO profiles, can provide additional value to the ecosystem.

We look forward to participating in the discussions and sharing our experiences with continuous profiling to contribute to the community's efforts.

[aalexand]

The profiler implementation is one thing, it would also be great to have you a part of the OTel profiling proto discussions and make sure that your learnings are taken into account.

[felixge]

We look forward to participating in the discussions and sharing our experiences with continuous profiling to contribute to the community's efforts.

We recently discussed your protobuf format, but realized it would be much more productive if we could ask questions to the people who worked on it. How is your availability to join one of upcoming meetings in the near future?

[itrofimow]

We'd be happy to join the discussion @felixge , and we should have someone available at the times profiling meeting takes place

Are there any specific questions you would like to be discussed?

[felixge]

Are there any specific questions you would like to be discussed?

We are very interested in any kind of quantitive analysis between your profiling format and pprof when it comes to data sizes and encoding/decoding efficiency.

In your article you mention the problem of pprof labels breaking the per-stack trace aggregation. We agree that this is an issue, and have addressed it the OpenTelemetry format by referencing stack traces from a lookup table. However, your format seems to have taken a much more radical approach of encoding everything in a column oriented fashion as well. We're very interested in learning more about the motivation for this, and what benefits you have observed.