HDDS-6856. HA aware reads from Snapshots

[ivandika3]

What changes were proposed in this pull request?

This adds a support for Ozone client to be able to do read-only snapshot operations (GetSnapshotInfo, ListSnapshots, SnapshotDiff, ListSnapshotDiffJobs) from non-leader OM.

For example, currently Snapdiff request will always go to the current OM leader. The leader will start a Snapdiff job if it does not exist yet and once it's finished, the user can rerun the snapdiff command again to receive the snapshot diffs returned by the Snapdiff job. However, if there is a OM leader change and another snapshot diff is called, the new OM leader needs to start the Snapdiff job again and the client will receive that the snapdiff is not ready yet, although it was ready at the previous leader. Furthermore, additional snapdiff job also adds some overhead.

This adds the following features

• Single (no-failover) proxy implementation to pick a specific OM node to get snapshot from SingleOMFailoverProxyProviderBase
• Supports for non-leader OM node to serve snapshot-related reads
  • This is supported by adding a new omNodeId proto field which if set means that the OM request is intended for the particular OM node
  • The client request will fail immediately for any NotLeaderException since it implies that the OM node does not accept the request.
• Ozone snapshot shell supports to read snapshots from followers

A possible usage would be to implement asynchronous bucket replication using bucket snapshot and snapdiffs similar to Cloudera Replication Manager (https://docs.cloudera.com/replication-manager/1.5.4/replication-policies/topics/rm-pvce-understand-ozone-replication-policy.html) mentioned in https://issues.apache.org/jira/browse/HDDS-12307. Since asynchronous replication does not require the most up-to-date key information, we can use OM listener (#7262) to generate snapdiff jobs required for the incremental snapshot. This way, the OM leader does not need to handle the replication related requests.

In the future, we can address https://issues.apache.org/jira/browse/HDDS-9666 to support reading keys in a snapshot from non-leader OM.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6856

How was this patch tested?

Unit test (proxy provider implementation), integration test (Ozone client), and acceptance test (Ozone shell, I moved the snapshot shell tests to be under ozone-ha environment).

Clean CI: https://github.com/ivandika3/ozone/actions/runs/13590426911

[peterxcli]

Left a few comments, feel free to take a look and see if they make sense. Thanks!