Improving CI speed with image caching

[shaneutt]

Problem Statement

A large portion of the time our CI runs take today are image pulls that happen inside ContainerD inside of KIND clusters.

Kind does not currently natively support loading image caches into ContainerD and so the effect is that images are pulled repeatedly (and in the case of docker registry images rate-limits can be quickly reached).

We should implement a mechanism in our CI to cache these images between builds in order to improve CI execution times.

Proposed Solution

• pre-save images into a local CI cache for each image that needs to be used in testing
• implement a KTF feature to enable kind load functionality to load the image archives on build

OR

• implement a mechanism in upstream kind for mounting the ContainerD image directories on the host system
• add KTF functionality for the image dir mount
• add the mounted directory to actions/cache on build

OR

• use an existing caching option like https://github.com/docker/build-push-action/ to build a cached image registry
• add tooling to KTF to configure cluster.Clusters with the CI registry

Acceptance Criteria

• as a contributor the CI for my PR's runs quicker because less competitive container image downloads need to finish in order to start my tests.