Implementation of parent and plugin overriding

[davidfestal]

What does this PR do?

This PR implements the parent or plugin overriding.

The overriding of parent devfile or devfile plugins is implemented using the Strategic merge patch K8S library.

Implementation entry points;

The implementation is available for import in the pkg/utils/overriding package.

Available methods are:

• OverrideDevWorkspaceTemplateSpec() to work with GO structs directly
• OverrideDevWorkspaceTemplateSpecBytes to work with json or yaml content.

In both case the result is a DevWorkspaceTemplateSpecContent Go struct that corresponds to the code content of a devfile 2.0 (without the top-level apiVersion and metadata fields).

Note that one can only override an existing element of a parent or a plugin.

If a new element is added, it should be added in the main (child) devfile body. And then simple merging of top-level lists is done between the main devfile body, the flattened parent and the flattened plugins. If the key (name of ID) of a element is found several times in the merge, an error is thrown.

The corresponding methods are:

• MergeDevWorkspaceTemplateSpec() to work with GO structs directly
• MergeDevWorkspaceTemplateSpecBytes to work with json or yaml content.

In both case the result is a DevWorkspaceTemplateSpecContent Go struct that corresponds to the code content of a devfile 2.0 (without the top-level apiVersion and metadata fields).

Examples of use:

• An example of use with GO structs can be found in the following test: https://github.com/devfile/kubernetes-api/blob/overriding-implementation/pkg/utils/overriding/overriding_test.go#L160
• An example of use with YAML files structs can be found in the following test: https://github.com/devfile/kubernetes-api/blob/overriding-implementation/pkg/utils/overriding/overriding_test.go#L168
• A similar example exists for the merging part.

YAML examples:

YAML example files are provided and used in GO tests.

Overriding

Overriding examples can be found in the following test-fixtures/patches folder: each sub-folder contains 3 Yaml files:

• original.yaml: the original Yaml that should be overridden
• patch.yaml: the Yaml patch that should be used to override the original file
• result.yaml: the expected result of the overriding, which is the result of applying strategic merge patch on the original file.

Merging

Merging examples can be found in the following test-fixtures/merges folder: each sub-folder contains up to 4 Yaml files:

• main.yaml: the Yaml of the main (child) devfile body content
• parent.yaml: the Yaml of a flattened parent referenced by the main devfile
• plugin.yaml: the Yaml a flattened plugin referenced by the main devfile
• result.yaml: the expected result of the merging of all those devfile contents.

When opened in VSCode or Hosted Che, all these example files are validated with appropriate and up-to-date schemas. So they benefit from YAML language services such as completion and doc hovers.

Contributions are welcomed in the future to further complete the tests (and at the same time documentation) of overriding rules.

Noteworthy (and possibly breaking) changes

• Events:
  • Detailed change: Instead of directly referencing the Events structure, the DevWorkspaceTemplateSpecContent struct contains a pointer to an Events struct.
  • Reason: This make testing easier, since it avoids a number of corner cases around serialization, when an empty Events object is created although no events field is present in the parsed json or yaml.
  • Impact: it will require a change in depending code, and a nil-check after json or yaml parsing.

Implementation details

Keys

Currently in the existing model, for some top-level lists (components, commands), the keys of elements are not directly in the objects stored in the list, but one level lower:

commands:
  - exec:
      id: "commandId"  <------ the key of the given command in the list of commands 

instead of

commands:
  - id: "commandId"
    exec:
      ....

Furthermore for some components as plugins, the key of a plugin in the list of components may be calculated, and may depend on a number of fields (id and registryUrl for example).
Thus a Keyed interface is defined and implemented for commands, components and component overrides, and it is then used to setup fixed keys in the commands and components lists of both the original object and the patch object during overriding.

Union normalizing

In the GO API, some model types are defined as Kubernetes unions with discriminators.

The API Union interface defines helpers to manage correctly manage those unions, especially maintaining the fact that values in a union are mutually-exclusive.

This union management logic is then used in the overriding implementation, but is also available in a dedicated pkg/utils/unions package that allows managing all the unions embedded in a wider object.

What issues does this PR fix or reference?

This PR references issue #91

Is your PR tested? Consider putting some instruction how to test your changes

The overriding logic implemented in this PR has associated tests

[kadel]

Going over this PR I noticed something that for some reason I didn't notice before.

The current design looks quite weird, and it is not how Strategic Merge Patch works in Kubernetes.

More in my comment at #25 (comment)

[davidfestal]

Going over this PR I noticed something that for some reason I didn't notice before.

The current design looks quite weird, and it is not how Strategic Merge Patch works in Kubernetes.

More in my comment at #25 (comment)

@kadel I implemented the agreement we found (summarized here ) in the PR, and updated the PR description to add details about the merging side of things (addition of elements that are not present in parents or plugins).

[davidfestal]

cc @ranakan19

[amisevsk]

Generally LGTM, with a few minor comments/nitpicks. I'll add a commit to resolve those for simplicity.

[amisevsk]

I had to rebase on master as there was a conflict. Commit 13fbbc9 resolves the minor comments from my review.

[davidfestal]

Thanks @amisevsk for this deep review and very-valuable fixes.