Implement TrainingRuntimes finalizer mechanism

[tenzen-y]

What you would like to be added?

We would like to implement the Finalizer trainer.kubeflow.org/resource-in-use mechanism against TrainingRuntime and ClusterTrainingRuntime.

• TrainingRuntime: Implement trainer.kubeflow.org/resource-in-use finalizer mechanism to TrainingRuntime #2608
• ClusterTrainingRuntime: WIP

Why is this needed?

The Finalizer prevents accidentally removing the TrainingRuntime when the referenced TrainJob has already been created.
If we do not have such a mechanism, the TrainingRuntime will be accidentally removed, and TrainJob will be disrupted since the actual Jobs (JobSet) will be built by TrainingRuntime and TrainJob.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[tenzen-y]

/assign

[tenzen-y]

/remove-label lifecycle/needs-triage