Use k8s volcano replicas to shrink job manifest size

[clumsy]

Description

Using replicas for repetitive pod configuration in kubernetes_scheduler has been removed in f6907e8

The rationale is here

Unfortunately for a large setup we can easily breach default limits, 1.5Mb: etcdserver: request is too large
It's not always possible to bump max-request-bytes, e.g. for AWS EKS.

Currently both job-specific and even TorchX own environment variables are contributing to breaching this limit.

We would like to find a way to make replicas work to minimize job manifest size.

Motivation/Background

Increase the maximum cluster size we can support with k8s

Detailed Proposal

E.g. using ConfigMap with per node/role config or Downward API. Make use of the fact we have roles with many replicas that share a huge chunk of their configuration.

Alternatives

Don't use environment variables and long names anywhere in the configuration, still the limit will be significantly smaller than when using replicas on average.

Additional context/links

• f6907e8
• https://github.com/pytorch/torchx/blame/41be1d8e97825151482323faabf5cdfcdd00f973/torchx/schedulers/kubernetes_scheduler.py#L378-L380
• https://kubernetes.io/docs/concepts/configuration/configmap
• https://kubernetes.io/docs/concepts/workloads/pods/downward-api/

[clumsy]

For your consideration, @kiukchung, @tonykao8080, @andywag, @d4l3k

Willing to contribute as always if there are no objections.

Thanks!

[andywag]

I don't have objections @kiukchung do you have any thoughts?