diff --git a/docs/design_proposals/configurable-kubecontext.md b/docs/design_proposals/configurable-kubecontext.md new file mode 100644 index 00000000000..e58699a9169 --- /dev/null +++ b/docs/design_proposals/configurable-kubecontext.md @@ -0,0 +1,202 @@ +# Configurable kubecontext + +* Author(s): Cornelius Weig (@corneliusweig) +* Design Shepherd: Balint Pato (@balopat) +* Date: 29 June 2019 +* Status: Under implementation + +## Background + +So far, Skaffold always uses the currently active kubecontext when interacting with a kubernetes cluster. +This is problematic when users want to deploy multiple projects with different kubecontexts, because the user needs to manually switch the context before starting Skaffold. +In particular when working on multiple such projects in parallel, the current behavior is limiting. + +Open issues concerning this problem are + +- Allow option to specify the kubectl context ([#511](https://github.com/GoogleContainerTools/skaffold/issues/511)) +- Support kube.config and kube.context for specifying alternative Kubernetes config file or context ([#2325](https://github.com/GoogleContainerTools/skaffold/issues/2325)) +- Feature: Support regex in profile activation via kubeContext ([#1677](https://github.com/GoogleContainerTools/skaffold/issues/1677)) +- Skaffold.yaml files are not portable ([#480](https://github.com/GoogleContainerTools/skaffold/issues/480)) +- Support forcing a context and a namespace for a profile/command ([#2426](https://github.com/GoogleContainerTools/skaffold/issues/2426)) + +There also was an attempt to add a configuration option to `skaffold.yaml` (Support for overriding kubectl context during deployment [#1540](https://github.com/GoogleContainerTools/skaffold/pull/1540)). + +The goal of this document is to create an agreement on what options should be supported and identify edge cases. + +### Recommended use-cases + +##### As Skaffold user, I want to define the kubecontext for a single skaffold run. +Use CLI flag or environment variable. + +##### As enterprise user, I want to define a default kubecontext for a project to be used across different machines. +Use the kubecontext configuration in `skaffold.yaml`. +Think twice before using this approach in open source projects, as the setting will not be portable. + +##### As individual user, I want to define a default kubecontext for a project. +Use kubecontext setting in the global Skaffold config (via `skaffold config set ...`). + +##### As Skaffold user with multiple profiles, I want to use different kubecontexts for different profiles. +Use the kubecontext configuration in `skaffold.yaml`. + + +## Design + +There are four places where kubecontext activation can be added: + + + + + + + + + + + + + + + + + + + + + + + + + + +
Precedence Kind Details
1. (highest)CLI option + The kubernetes standard to set the kubecontext is --context. + However, in Skaffold this term is so overloaded that it should more precisely be named --kube-context. + This flag is necessary for IDE integration. +
2.Env variable + SKAFFOLD_KUBE_CONTEXT, similar to other Skaffold flags. +
3.skaffold.yaml + Json-path deploy.kubeContext. + This option is shareable, and requires some error handling for profile activation by kubecontext (see below). +
4. (lowest)Global Skaffold config + This should give users the possibility to define a default context globally or per project. + This variant is not shareable. +
+ +--- + +Beside the kubecontext, also the namespace needs to be specified. +Ideally, the namespace should also offer the same override variants as kubecontext. +This is out of scope for this design proposal. +As long as this is not implemented, there is always the workaround, to duplicate a kubecontext and set the default namespace for this kubecontext to the desired value. +Then this kubecontext/namespace pair can be activated with the kubecontext activation machinery detailed in this design proposal. + +### Detailed discussion +#### Option in `skaffold.yaml` +A configuration option in `skaffold.yaml` has the advantage of being most discoverable: +it is in the place where users configure all aspects of their pipeline. +In addition, it allows to define the kubecontext per Skaffold profile. + +A natural place for the config in `skaffold.yaml` is in `latest.DeployConfig`, resulting in a json path `deploy.kubeContext`. + +Profiles have a double role, because they may override the kubecontext to a different value as before, but they may also be activated by a kubecontext. +To catch unexpected behavior, the profile activation will perform the following logic: + +1. All profiles are checked if they become active with the _original_ kubecontext. +2. If any profile was activated by the current kubecontext, the effective kubecontext may not change. + In other words, the effective kubecontext _after_ profile activation must match the kubecontext _before_ profile activation. + - If the context has changed, return an error. + - If the context has not changed, continue. + +It is therefore possible that subsequent profiles repeatedly switch the kubecontext, in which case the last one wins. + +For example, the following sequence will result in an error: + +- The current context is `minikube` and activates some profile. +- Some profile also overrides the kubecontext and deploys to kubecontext `gke_abc_123`. +- Thus the `minikube`-specific profile would be deployed to `gke_abc_123`, and this will be forbidden. + +**Note**: `skaffold.yaml` is meant to be shared, but kubecontext names vary across users. +Sharing therefore makes only sense in a corporate setting where context names are the same across different machines. + +#### Option in global Skaffold config +Specifying a default kubecontext globally is straightforward. For example, via new config option +```yaml +global: + default-context: docker-for-desktop +``` + +On the other hand, building a relation between projects and kubecontext needs to solve two questions: + +1. How to identify projects +2. How to save the project/kubecontext relation + +##### How to identify projects + +There are at least three possibilities: + +- Identify projects by their absolute host path. + This is guaranteed to be unique, but may break if a user moves his project to another location. +- Identify projects by a new `metadata.name` entry in `skaffold.yaml` (see also [#2200](https://github.com/GoogleContainerTools/skaffold/issues/2200)). + This has the drawback of being potentially not unique, so that users accidentally pin the kubecontext for more projects than intended. + On the other hand, this is the standard approach taken by kubernetes resources. +- Identify project by their initial commit. + This variant is stable against relocations. + It is also unique unless a user forks a project and wants to define different kubecontexts for each fork. + Compared to the other alternatives, it is rather slow. + +**\** + +Resolution: We will go ahead with the `metadata.name` approach. As the name may not be unique, this requires special documentation. + + +##### How to save project/kubecontext relations + +Currently, the Skaffold config uses the kubecontext as identifier. + +There are two possibilities to add the relation: + +1. Reverse the mapping project/kubecontext and save as list under `kube-context` entries: + ```yaml + kubecontexts: + - kube-context: my-context + skaffoldConfigs: + - config-name + ``` + The drawback here is that the data structure does not forbid conflicting entries, such as: + ```yaml + kubecontexts: + - kube-context: context1 + skaffoldConfigs: + - my-project + - kube-context: context2 + skaffoldConfigs: + - my-project + ``` +2. Add a new top-level entry in Skaffold config: + ```yaml + global: {} + kubecontexts: [] + skaffoldConfigs: + my-project: my-context + my-other-project: my-other-context + '*': default-context + ``` + This option will be more complex to implement wrt `skaffold config`. + +### Open Issues/Questions + +**\** + +Resolution: The top-level entry (option 2) overall has the better trade-offs. + +## Implementation plan +1. Implement the CLI flag and env var variant first. This should also be the most important for the IDE integration. [#2447](https://github.com/GoogleContainerTools/skaffold/pull/2447) +2. Implement `skaffold.yaml` variant. [#2510](https://github.com/GoogleContainerTools/skaffold/pull/2510) +3. Implement the global Skaffold config variant to override a kubecontext for a skaffold config (`skaffoldConfigs`). [#2558](https://github.com/GoogleContainerTools/skaffold/pull/2558) +4. Implement the global Skaffold config variant to set a default kubecontext (`skaffoldConfigs['*']`). [#2558](https://github.com/GoogleContainerTools/skaffold/pull/2558) +5. ~~Implement the namespace functionality.~~ (out of scope) + +## Integration test plan + +A single test covers the overall kubecontext override functionality sufficiently (part of [#2447](https://github.com/GoogleContainerTools/skaffold/pull/2447)). +Other test-cases such as precedence of the different variants and error cases should be covered by unit tests.