Real-time communication R&D and POC

[binarylogic]

Before we embark on #3211, we need to research communication strategies between a running Vector instance and the Vector CLI. This strategy should also be proven with a simple standalone POC. This will help to validate the viability of our chosen strategy before we attempt to integrate it into Vector.

Prior Art

• https://github.com/aheart/hearth
• https://github.com/netdata/netdata

Requirements

• Determine the best protocol for the New top subcommand v1 #3211 and the upcoming web UI.
• The protocol should allow for streaming.
• Ensure that the ecosystem and tooling for this protocol are sufficient and will enable productive development.
• Implement a small standalone POC demonstrating communication from a Rust backend (AKA Vector) and an interface (CLI or Web UI). The Web UI is the end goal, so whatever protocol we choose should work with our upcoming admin UI.

Future concerns

1. This protocol will be used to transmit metrics data initially. You can see our internal metrics event here, docs on it here, and proto definition here.
2. New top subcommand v1 #3211 is only concerned with Vector's internal metrics, which are derived from internal events. An example is here. So the shapes and types of metrics will be limited and predictable.
3. Our hope is that we can use this protocol for future endeavors which will include communicating with multiple remote Vector instances.

[binarylogic]

Closing this since we've landed on a protocol and are now shifting work into Vector itself.