MSO Integration with Observatorium

[cmwylie19]

MSO greatly simplifies the process of creating independent, self-contained, Prometheus-based monitoring stacks through Kubernetes CRDs, but we still need to integrate with Observatorium.

As avid consumers of MSO, we need to send these metrics to an aggregator, which is where Observatorium comes into play to allow us to run and operator a multi-tenant, easy to operator availability system.

The problem:
Currently MSO doesn't have specific functionality to configure integration with external centalized monitoring facilities (for an example Observatorium),  instead relying on remoteWrite sections in the CR . While it is flexible approach it has some of the potential shortcomings , such as credential information is passed in text format to the respective CR , there are no means to refresh Authentication bearer token. As tokens inevitably expire (else they are inherently insecure). Some proposed options are to use a Token Refresher to ensure the bearer token is not expired. The token refresher would essentially proxy the request to the observatorium instance with a Bearer token that is not expired, therefore, in the RemoteWrite spec of the MSO would have the address of the token refresher.

(With Observatorium , authentication is just one thing that creates obvious frictions and there is need for embedding dynamic functionality of the refreshing token , there could be other cases that something else needed to be done that can not be fullfilled by just static config. )

Desired Solutions
MSO to be equipped to fully integrate and communicate with Observatorium and have the Authentication happen in the background.

Possible Implementations

• A separate operant that populates remote writes (responsible for providing all communication needs to observatorium including auth, refreshing tokens, creating a secret that will emit remote write and do all plumbing ).
• The Observability Operator handles this with a Fetch function

[simonpasquier]

My assumption is that the remote write configuration already support oauth2 authentication so token-refresher isn't needed anymore but I might be wrong :)

[jan--f]

@cmwylie19 Can you check of v0.0.9 works for you as expected? #140 updated prometheus-operator so that it supports auth2 in remote write specs.

[cmwylie19]

@jan--f tested it and the RemoteWrite works very well with oauth2. Thank you.

@redmikhail , do you want to add anything? I know we have talked about a separate crd (operant) for handling remote writes that does all of the plumbing?

[cmwylie19]

I am going to close this issue for now as we are using oauth2 for bearer token refreshing. Thanks!