Send metrics via any protocol in the OTLP Registry

[kasparkivistik]

Please describe the feature request.
The idea is to make it available for any consumer of the micrometer-otlp-registry to provide their own metrics sending implementation. The OTLP Registry can still only support HTTP/protobuf as it does today but, for example, its implementation can be described by a public interface.

Rationale
The purpose of the change is to allow for metrics sending to be more flexible on the consumer side. I'm not advocating any changed to the OTLP protocols, rather my intention would be to open metrics sending up.
And example on my end would be that I desire to push all metrics to a Kafka topic before they are sent via HTTP/protobuf to OTLP collector. This way I have greater control over my metrics that might be coming from a number of microservices.

Additional context
Slack discussion: https://micrometer-metrics.slack.com/archives/C01C4Q8FHC2/p1731676707725909

[jonatan-ivanov]

People who give this issue a 👍🏼 or 🚀, what is your use-case?

Right now, the use-case seems to be sending the data through Kafka which is not supported by the specs if I remember correctly. But I'm curious if people have other use-cases or you are co-workers and has the same use-case.

[kasparkivistik]

@jonatan-ivanov they are co-workers 😅

For the time being I did a local workaround for the issue. Let's see what the community might think of it

[shakuzen]

Could people who wanted this feature please try out what we've delivered in 1.15.0-M2 and give feedback? See, for example, the added section in the documentation. There's a strong possibility this will change somewhat before the 1.15.0 GA release, and your feedback will help ensure we end up with a better feature in the end.

[shakuzen]

We're still hoping for some feedback on this before GA if anyone can try it out and let us know what you think.

[kasparkivistik]

We're still hoping for some feedback on this before GA if anyone can try it out and let us know what you think.

Looks good on my end. Made my use of OtlpMeterRegistry significantly simpler! @shakuzen

[pirgeo]

Looks like a great idea, maybe even one that could extend to other registries (i.e. make the export independent from the serialization)!
Are there any plans to include that in the Spring Boot autoconfiguration, e.g. here? It would be really cool if you only had to specify the OtlpHttpMetricsSender bean in your Spring Boot app to change the export path, I understand currently there is no way to use this in Spring Boot, or am I missing something?

[shakuzen]

Are there any plans to include that in the Spring Boot autoconfiguration, e.g. here?

I'm sure the nice folks on the Spring Boot team would consider it. I was waiting to get feedback on this and consider any additional changes before I asked them to change any auto-configuration for it.

currently there is no way to use this in Spring Boot, or am I missing something?

You can always provide your own OtlpMeterRegistry bean, but I guess that defeats a lot of the benefit of the auto-configuration. You could still use the OtlpConfig that Spring Boot auto-configures, for instance, though. I could see the Boot auto-configuration taking an optional OtlpMetricsSender and configuring that on the registry it creates, if present.

maybe even one that could extend to other registries (i.e. make the export independent from the serialization)!

I did consider doing that with this, but it hasn't really been asked for much in other registries and so I didn't want to expand the scope more than made sense. We can make this generic, but other registries would need to be updated to use it or else it wouldn't offer any benefit. Most other registries also tend to be a single protocol and format for the backend. Thus, most desired customization can be done via the long existing HttpSender or the registry-specific SDK if there is one.

[pirgeo]

That makes sense to me. I think most registry autoconfigurations don't allow you to overwrite the HttpSender without creating a ChooseYourRegistry bean today, but that could be addressed in the Spring Boot autoconfig code, if desired. For Micrometer standalone it already works with the HttpSender interface.
Overall, this change looks good to me!

[shakuzen]

I've proposed a breaking API change to address this concern with the current API. Feel free to leave feedback on the PR or here if there is any concern with the changes. #5994

[shakuzen]

I've opened another PR for refining the API introduced for this - see #6025.