Add support for dynamically updating instance config

[kkirchoff]

Hello. While many of our applications are container based with fixed topologies, we have several use cases that would greatly benefit from dynamic updates. We spoke with @ jonpjenkins from the Google Team and agreed to enter a feature request.

One such use case is a auditing system that needs to scan all of our databases for PII. This system will need to connect to many systems, and the list will need to be updated as new databases are added or switched. The scanners run in GKE, but for some bewildering reason or another, they are persistent and do not run on a 1:1 scanner to job model.

The most basic and effective model that we see is to:

• Define the configuration as a bucket, GSM or other object that is hosted on a central object service.
• Provide for a customer to define the object address at startup so that the appropriate configuration flexibility is maintained.
• Retrieve the configuraion every X seconds or minutes and perform a hot start if it changes.
• Limit hot starts so as to not cause a restart loop

Many popular servers such as TCP proxies allow for hot restarting by receiving a signal and then:
(a) running a lint check on the new configuration
(b) terminating the listener. Existing connections should continue to persist, using the old configuration and state, but the old daemon cannot receive new connections
(c) starting a new listener and worker pool that will receive new connections with the new configuration.

Apache, Sendmail, squid, Envoy and other processes use this basic reload feature for non-disruptive uploads.

Most of these examples provide easy to replicate hot restarts. I can give you the envoy hot restarted code if you're looking for an example of that. Their documentation is horrendous, but we're running it as a MySQL proxy on prem and we non-disruptively update the entire configuration this way out of the box!

Oh, one more thing:
For containers, it would be great if the configuration checker would kick off the hot restart. Same for GCE, but if you could allow a user to update the local config and manually kick off a hot restart, that would be extra great.

[enocom]

Just to summarize our conversation elsewhere, right now the closest thing that matches this is instances_metadata. However, this key assumes a Unix socket connection, which doesn't work for clients using TCP connections.

Like the other examples you've note, Go allows users to close a net.Listener without affecting any open connections. So we could be starting listeners on startup. When the configuration changed, we'd close the listener and open new listeners per the configuration.

For containers, it would be great if the configuration checker would kick off the hot restart.

Would you mind elaborating on this? Do you mean that you'd start the proxy with a pointer to the configuration (a GCS object, a Secret Manager secret, etc) and the proxy would set itself up entirely from this configuration? Something like:

$ cloud_sql_proxy --dynamic-config gs://mybucket/config.json

Same for GCE, but if you could allow a user to update the local config and manually kick off a hot restart, that would be extra great.

How would this be different from the example above?

[enocom]

Adding some additional information here from @davidcollom:

We don't want to specifically name the instances as we dynamically scale up/down read-replicas along with instances as we scale applications within that region.

Additionally we'd like to not restart the proxy when we do this as our applications as this could impact multiple applications at a time and/or additional upstream failures (queries in flight etc)

In terms of V2... It would be nice overall to allow project + labels lookup to give finer grained connections in a dynamic environment.

[enocom]

Currently, we're thinking about making it possible to dynamically configure instance connection names using:

• Secret Manager secrets
• Pub/Sub topics connected to a secret
• (possibly) Compute Engine Metadata.

The first two are the priority at the moment.

[Goodsmileduck]

Hi, is there any update for that feature ?

[enocom]

Not yet. It remains pretty high up on the list and will likely get some attention later this year.

[enocom]

In the meantime, we have an example that shows how to connect the proxy to secret manager, such that it restarts when a new instance connection name shows up. See https://github.com/GoogleCloudPlatform/cloud-sql-proxy/tree/main/examples/disaster-recovery.

[enocom]

This might be a good solution for people who want to automatically cut over to newly promoted read replicas.