Skip to content

Commit 211d08a

Browse files
nkubalankubala
authored
Merge pull request #2384 from corneliusweig/w/context
Design proposal for configurable kubecontext
2 parents d079d77 + c6096fe commit 211d08a

File tree

1 file changed

+202
-0
lines changed

1 file changed

+202
-0
lines changed

Diff for: ‎docs/design_proposals/configurable-kubecontext.md

+202
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
# Configurable kubecontext
2+
3+
* Author(s): Cornelius Weig (@corneliusweig)
4+
* Design Shepherd: Balint Pato (@balopat)
5+
* Date: 29 June 2019
6+
* Status: Under implementation
7+
8+
## Background
9+
10+
So far, Skaffold always uses the currently active kubecontext when interacting with a kubernetes cluster.
11+
This is problematic when users want to deploy multiple projects with different kubecontexts, because the user needs to manually switch the context before starting Skaffold.
12+
In particular when working on multiple such projects in parallel, the current behavior is limiting.
13+
14+
Open issues concerning this problem are
15+
16+
- Allow option to specify the kubectl context ([#511](https://github.com/GoogleContainerTools/skaffold/issues/511))
17+
- Support kube.config and kube.context for specifying alternative Kubernetes config file or context ([#2325](https://github.com/GoogleContainerTools/skaffold/issues/2325))
18+
- Feature: Support regex in profile activation via kubeContext ([#1677](https://github.com/GoogleContainerTools/skaffold/issues/1677))
19+
- Skaffold.yaml files are not portable ([#480](https://github.com/GoogleContainerTools/skaffold/issues/480))
20+
- Support forcing a context and a namespace for a profile/command ([#2426](https://github.com/GoogleContainerTools/skaffold/issues/2426))
21+
22+
There also was an attempt to add a configuration option to `skaffold.yaml` (Support for overriding kubectl context during deployment [#1540](https://github.com/GoogleContainerTools/skaffold/pull/1540)).
23+
24+
The goal of this document is to create an agreement on what options should be supported and identify edge cases.
25+
26+
### Recommended use-cases
27+
28+
##### As Skaffold user, I want to define the kubecontext for a single skaffold run.
29+
Use CLI flag or environment variable.
30+
31+
##### As enterprise user, I want to define a default kubecontext for a project to be used across different machines.
32+
Use the kubecontext configuration in `skaffold.yaml`.
33+
Think twice before using this approach in open source projects, as the setting will not be portable.
34+
35+
##### As individual user, I want to define a default kubecontext for a project.
36+
Use kubecontext setting in the global Skaffold config (via `skaffold config set ...`).
37+
38+
##### As Skaffold user with multiple profiles, I want to use different kubecontexts for different profiles.
39+
Use the kubecontext configuration in `skaffold.yaml`.
40+
41+
42+
## Design
43+
44+
There are four places where kubecontext activation can be added:
45+
<table>
46+
<thead>
47+
<th>Precedence</th> <th>Kind</th> <th>Details</th>
48+
</thead>
49+
<tbody>
50+
<tr>
51+
<td>1. (highest)</td>
52+
<td>CLI option</td>
53+
<td>
54+
The kubernetes standard to set the kubecontext is <code>--context</code>.
55+
However, in Skaffold this term is so overloaded that it should more precisely be named <code>--kube-context</code>.
56+
This flag is necessary for IDE integration.
57+
</td>
58+
</tr>
59+
<tr>
60+
<td>2.</td>
61+
<td>Env variable</td>
62+
<td>
63+
<code>SKAFFOLD_KUBE_CONTEXT</code>, similar to other Skaffold flags.
64+
</td>
65+
</tr>
66+
<tr>
67+
<td>3.</td>
68+
<td><code>skaffold.yaml</code></td>
69+
<td>
70+
Json-path <code>deploy.kubeContext</code>.
71+
This option is shareable, and requires some error handling for profile activation by kubecontext (see below).
72+
</td>
73+
</tr>
74+
<tr>
75+
<td>4. (lowest)</td>
76+
<td>Global Skaffold config</td>
77+
<td>
78+
This should give users the possibility to define a default context globally or per project.
79+
This variant is not shareable.
80+
</td>
81+
</tr>
82+
</tbody>
83+
</table>
84+
85+
---
86+
87+
Beside the kubecontext, also the namespace needs to be specified.
88+
Ideally, the namespace should also offer the same override variants as kubecontext.
89+
This is out of scope for this design proposal.
90+
As long as this is not implemented, there is always the workaround, to duplicate a kubecontext and set the default namespace for this kubecontext to the desired value.
91+
Then this kubecontext/namespace pair can be activated with the kubecontext activation machinery detailed in this design proposal.
92+
93+
### Detailed discussion
94+
#### Option in `skaffold.yaml`
95+
A configuration option in `skaffold.yaml` has the advantage of being most discoverable:
96+
it is in the place where users configure all aspects of their pipeline.
97+
In addition, it allows to define the kubecontext per Skaffold profile.
98+
99+
A natural place for the config in `skaffold.yaml` is in `latest.DeployConfig`, resulting in a json path `deploy.kubeContext`.
100+
101+
Profiles have a double role, because they may override the kubecontext to a different value as before, but they may also be activated by a kubecontext.
102+
To catch unexpected behavior, the profile activation will perform the following logic:
103+
104+
1. All profiles are checked if they become active with the _original_ kubecontext.
105+
2. If any profile was activated by the current kubecontext, the effective kubecontext may not change.
106+
In other words, the effective kubecontext _after_ profile activation must match the kubecontext _before_ profile activation.
107+
- If the context has changed, return an error.
108+
- If the context has not changed, continue.
109+
110+
It is therefore possible that subsequent profiles repeatedly switch the kubecontext, in which case the last one wins.
111+
112+
For example, the following sequence will result in an error:
113+
114+
- The current context is `minikube` and activates some profile.
115+
- Some profile also overrides the kubecontext and deploys to kubecontext `gke_abc_123`.
116+
- Thus the `minikube`-specific profile would be deployed to `gke_abc_123`, and this will be forbidden.
117+
118+
**Note**: `skaffold.yaml` is meant to be shared, but kubecontext names vary across users.
119+
Sharing therefore makes only sense in a corporate setting where context names are the same across different machines.
120+
121+
#### Option in global Skaffold config
122+
Specifying a default kubecontext globally is straightforward. For example, via new config option
123+
```yaml
124+
global:
125+
default-context: docker-for-desktop
126+
```
127+
128+
On the other hand, building a relation between projects and kubecontext needs to solve two questions:
129+
130+
1. How to identify projects
131+
2. How to save the project/kubecontext relation
132+
133+
##### How to identify projects
134+
135+
There are at least three possibilities:
136+
137+
- Identify projects by their absolute host path.
138+
This is guaranteed to be unique, but may break if a user moves his project to another location.
139+
- Identify projects by a new `metadata.name` entry in `skaffold.yaml` (see also [#2200](https://github.com/GoogleContainerTools/skaffold/issues/2200)).
140+
This has the drawback of being potentially not unique, so that users accidentally pin the kubecontext for more projects than intended.
141+
On the other hand, this is the standard approach taken by kubernetes resources.
142+
- Identify project by their initial commit.
143+
This variant is stable against relocations.
144+
It is also unique unless a user forks a project and wants to define different kubecontexts for each fork.
145+
Compared to the other alternatives, it is rather slow.
146+
147+
**\<What option has the best tradeoffs?\>**
148+
149+
Resolution: We will go ahead with the `metadata.name` approach. As the name may not be unique, this requires special documentation.
150+
151+
152+
##### How to save project/kubecontext relations
153+
154+
Currently, the Skaffold config uses the kubecontext as identifier.
155+
156+
There are two possibilities to add the relation:
157+
158+
1. Reverse the mapping project/kubecontext and save as list under `kube-context` entries:
159+
```yaml
160+
kubecontexts:
161+
- kube-context: my-context
162+
skaffoldConfigs:
163+
- config-name
164+
```
165+
The drawback here is that the data structure does not forbid conflicting entries, such as:
166+
```yaml
167+
kubecontexts:
168+
- kube-context: context1
169+
skaffoldConfigs:
170+
- my-project
171+
- kube-context: context2
172+
skaffoldConfigs:
173+
- my-project
174+
```
175+
2. Add a new top-level entry in Skaffold config:
176+
```yaml
177+
global: {}
178+
kubecontexts: []
179+
skaffoldConfigs:
180+
my-project: my-context
181+
my-other-project: my-other-context
182+
'*': default-context
183+
```
184+
This option will be more complex to implement wrt `skaffold config`.
185+
186+
### Open Issues/Questions
187+
188+
**\<What Skaffold config structure has the best tradeoffs?\>**
189+
190+
Resolution: The top-level entry (option 2) overall has the better trade-offs.
191+
192+
## Implementation plan
193+
1. Implement the CLI flag and env var variant first. This should also be the most important for the IDE integration. [#2447](https://github.com/GoogleContainerTools/skaffold/pull/2447)
194+
2. Implement `skaffold.yaml` variant. [#2510](https://github.com/GoogleContainerTools/skaffold/pull/2510)
195+
3. Implement the global Skaffold config variant to override a kubecontext for a skaffold config (`skaffoldConfigs`). [#2558](https://github.com/GoogleContainerTools/skaffold/pull/2558)
196+
4. Implement the global Skaffold config variant to set a default kubecontext (`skaffoldConfigs['*']`). [#2558](https://github.com/GoogleContainerTools/skaffold/pull/2558)
197+
5. ~~Implement the namespace functionality.~~ (out of scope)
198+
199+
## Integration test plan
200+
201+
A single test covers the overall kubecontext override functionality sufficiently (part of [#2447](https://github.com/GoogleContainerTools/skaffold/pull/2447)).
202+
Other test-cases such as precedence of the different variants and error cases should be covered by unit tests.

0 commit comments

Comments
 (0)