[DSIP-88][WORER-SERVER] Optimize the connection between Kubernetes task types and Kubernetes clusters.

[wangxj3]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

The optimization primarily focuses on Kubernetes task types, improving the linkage for task submission, log retrieval, and status monitoring. It enhances the original connection creation and destruction mechanism by introducing a connection pool for the Kubernetes cluster, where each worker instance maintains its own set of Kubernetes connections.

Design Detail

The current connection between Kubernetes tasks and the Kubernetes cluster may reduce the interaction efficiency between the tasks and the cluster when the task concurrency is high, and could even lead to task submission or status retrieval failures.

Compatibility, Deprecation, and Migration Plan

The optimization primarily focuses on Kubernetes task types, improving the linkage for task submission, log retrieval, and status monitoring.

Test Plan

It enhances the original connection creation and destruction mechanism by introducing a connection pool for the Kubernetes cluster, where each worker instance maintains its own set of Kubernetes connections.

Code of Conduct

• I agree to follow this project's Code of Conduct

[wangxj3]

I will submit a pr to fix it

[SbloodyS]

Please provide more details of this feature since you mark it as an DSIP. @wangxj3

[wangxj3]

Please provide more details of this feature since you mark it as an DSIP. @wangxj3

fix it