WIP: add-ons design proposal

[errordeveloper]

No description provided.

[christopherhein]

Overall 💯 agree with this feature. Added some changes and my thoughts.

What could really help this would be talking about a couple example projects or addons that would scope into this. For example --addons=helm (or whatever implementation looks like)

When we pass helm as an addon eksctl uses the compiled in helm libraries to call helm init from the go SDK. How do we tell it to use RBAC? Did it auto create the SA and ClusterRoleBinding, etc? When you call the sdk does it automatically write the files to the persons home directory?

Or for another --addons=dashboard when this is triggered it uses the referenced github manifest links to apply to the cluster.

Last one, kube2iam when you deploy this is will use the helm chart with params (do we have a dependency graph that we now have to manage?) then it will deploy the IAM Assume Role Cloudformation template which referenced there node group to get the instance role ARN? it is auto-compiled into the nodegroup CFN, if so how would we handle removing it.

Hopefully that will help guide this a little more from the experience perspective. For example if I as a customer still need to manage my own upgrades of these addons I believe that would be acceptable as long as the initial heavy lifting was done. But maybe we strive for the implementation to be idempotent and if you deploy the dashboard and we upgrade the cli with a newer version you could run eksctl apply addon dashboard and it would apply the manifests in place just updating the existing. For helm this could get complicated cause now we need a way to persist the state of the individual addon names that they were deployed as.

Again super excited about this feature, happy to see we're starting with a proposal!

[stefanprodan]

Maybe the proposal could include a sample config file for the addons.

apiVersion: eksctl.io/v1alpha1
kind: Config
metadata:
  name: eks-dev
spec:
  addons:
    - name: cert-manager
      namespace: kube-system
      chart:
        repository: https://kubernetes-charts.storage.googleapis.com
        name: cert-manager
        version: v0.5.0
      values:
        createCustomResource: true
        rbac:
          create: true
    - name: kured
      namespace: adm
      chart:
        repository: https://kubernetes-charts.storage.googleapis.com
        name: kured
        version: 0.1.0
      values:
        updateStrategy: OnDelete
        rbac:
          create: true

dismissing feedback over conversation.

[errordeveloper]

I'll probably get back to this only after I'm back from my vocation, and that'd be in December. Just FYI.

[errordeveloper]

TIL: kops doesn't have a way to easily upgrade only network add-ons (or at least it's not documented); it happens to be part of cluster upgrade... we should verify this, of course :)

[yhvh]

I vote against this feature, too much complexity for such a small gain.
I would prefer the project maintained a tighter focus.

[mumoshu]

I think I have the same sentiment as @yhvh. The specs looks awesome, but the gain seems so small.

As long as eksctl is able to create correct IAM role/permissions for well-known addons, I don't mind if it doesn't actually install k8s resource objects.

I'd just write some bash/pulumi/helmfile/etc to install them.

[cdenneen]

@yhvh @mumoshu Can you elaborate on the small gain here? The result I believe will allow you to create a cluster (new, replacement, standby, etc) and have many of the addons automatically install, configure in a repeatable/consistent fashion. This married with GitOps proposal will allow clusters to be built with completeness. I think this proposal is very useful and has tremendous potential gains.

To build a cluster (minimal) THEN apply pieces to enhance it (cluster autoscaler, hpa, helm, kube2iam, weave-flux, etc) and THEN to deploy your services to it (jenkins, gitlab, artifactory, etc) much less your own helm charts for app1, app2, app3.

With having the addons + GitOps implemented you'll have the ability to build a cluster that has weave networking, IAM policy changes, cluster autoscaler, flux, etc as well as jenkins, gitlab, artifactory and app1/app2 deployed... especially in a disaster scenario you are "back in business". In a non-disaster scenario you stand up replacement cluster and then cut over. If you implement additional pieces like @istio and px-motion (@portworx) you could do so much more.

[prageethw]

my view also addons gonna be an extra burden and quickly can go out of date or will be a too much hassle maintaining them, a great example is kops addons, I found they are out of date whereas helm moved on a long time ago with new patches, versions etc.

[cdenneen]

Addons would be helm based and if your using GitOps then any updates you do should be in git and therefore should be whatever version you specified. If you manually install something like cluster auto scaler via helm you would have to update the helm release manually same as via a config for eksctl. The addons aren’t eksctl specific they are using helm charts.

[prageethw]

Addons would be helm based and if your using GitOps then any updates you do should be in git and therefore should be whatever version you specified. If you manually install something like cluster auto scaler via helm you would have to update the helm release manually same as via a config for eksctl. The addons aren’t eksctl specific they are using helm charts.

I agree on one thing that if addons introduced to eksctl it has to be based on helm, but what puzzles me where the real value we see on addons if that's the case. To me, kops add-on is an old concept they adopted before helm became mainstream.

One reason I like to keep plain old k8s separate from all other extra such as metrics server, ca, etc because at any time they add-ons can go broke against particular k8s version, kops or even eks. This is something we face in prod every day we never upgrade helm charts until we are highly confident it will work after some extensive testing, so it is often required whether a particular version of the addon is compatible with specific k8s or eks,kops or whatever minor differences on them.

so even we have add-ons we ultimately will end up in helm to manage such situations hence my argument it is not an ideal concept, now if add-ons would be part of EKS itself i find it makes more sense as it will be tested before a major release, (still I found giving full control to individual user what add-ons they wanna install more measured approach)

[mumoshu]

@cdenneen Hey! Thanks for the response.

Indeed, the concept of add-ons is great, a tool that declaratively manages a cluster and its apps as a whole is awesome, GitOps is the way to go, I believe too. I think we are on the same page.

It is just that doing it in the scope of eksctl, especially as the form of "add-ons", is just an implementation detail to me. And "add-ons" seemed not a good implementation to me.

Add-ons are great if and only if it is well maintained. And it turned out to be not easy to maintain them. Honestly saying even a big project, like kops, who should have so many users and contributors, seems to have failed maintaining it well. Why repeat the same thing that failed before, without major improvements(I suppose. Sry if I missed that it is covered in this proposal) over the last trial?

Also, add-ons is not a pre-requisite for GitOps. For example, I'm currently trying to apply GitOps to both clusters and apps(helm releases). I'd just use eksctl to declaratively manage clusters, and helmfile to declaratively manage apps.

I'm developing an another high-level command that accepts a config file that covers both clusters and apps, calls eksctl and helmfile. My GitOps pipeline implemented by atlantis/codefresh/brigade/etc the command. Implementation-wise, you can use whatever tools that match your use-case. It is true that the market of such high-level commands is still small, but it doesn't mean eksctl should be one.

So, you do need a tool to declaratively manage everything including clusters and apps, and probably a CD solution(whether it is GitOps flavored or not, it doesn't matter) that supports it. But the tool should not necessarily be eksctl.

[cdenneen]

100% agree. I never said eksctl was the only solution but coming from keeping infra (Cloudformation, Terraform, etc) and config (Chef, Puppet, Ansible, etc) separate; maybe even data (Hiera, data bags, zookeeper, etc) things tend to get messy and “lost”. In the world of Kubernetes where everything is fully declarative and explicit I think having this under one umbrella truly helps keeps things easier to maintain. The eksctl config can specify version of k8s (1.10.3) and version of helm charts that work with 1.10. Then new cluster is created with 1.11 and all the same addons with newer helm chart versions to see if compatible. You could spin up brand new cluster, ALL your config and apps, run full test suite and determine if everything works as a whole. There is someone about running a command and 30 minutes later having an entire cluster with every service, application (nodejs, dotnetcore, php, mongodb, vault, elasticsearch, etc, etc) fully installed, configured and running. Without that you spin up new dev cluster with 1.11 and then wire up each of your pipelines to deploy to it one by one. Trust me I can see pitfalls in both scenarios but I think there is huge value of having a “complete” cluster from single config or repo! The term “addons” I think seems to be the “crux” of this. Should it be called “legos”? ;-) I mean if you can build me a foundation repeatedly that’s pretty good but if you can build me my entire house, with all the same detail down to the exact same picture on the mantel, now that’s something impressive.

[mumoshu]

Thanks again for sharing your idea!

I think having this under one umbrella truly helps keeps things easier to maintain

I completely agree with you :)

You could spin up brand new cluster, ALL your config and apps, run full test suite and determine if everything works as a whole

Yep, this is exactly the same U/X I want to achieve via my own high-level command and GitOps pipelines.

I'd also like to:

• Deploy my own apps along with eksctl add-ons, so that I don't need to script anything to be run after eksctl
• Regularly update only my apps, not eksctl add-ons, after the initial cluster bootstrap
• In case one of add-ons is out-dated, I'd like to update it out of eksctl
• Include terraform/cloudformation variables in my declarative "clusterr" config to manage cloud-provider resources along with my cluster (on top of the cloud-provider resources) and apps(on top of clusters).
• Have a reusable GitOps pipeline or its framework implemented as a Kubernetes operator, or something similar that runs on a serverless platform like Lambda or Fargate. Then we don't need to reimplement the whole GitOps pipelines on top of various CI/CD services, platforms, etc.

I wish we could come up with a generic solution someday!

[whereisaaron]

My add-ons design proposal it don’t do it 😄 Every project you ‘add-on’ you might as well volunteer to maintain those projects too. If eksctl installs them, eksctl on the hook to patch and upgrade them for every release of those projects too. Every if just firing off helm charts you are still tracking versions and breaking changes to chart values settings.

If you think a project is utterly essential cluster bootstrap material, and can’t be installed and maintained separately by other tools, like maybe EKS AMI, CNI plug-in, core-dns. But anything that can be installed (and upgraded) by your choice(s) of helm/brigade/kubectl/octopus/cfn/terraform/etc. after eksctl bootstrap should IMHO not be part of eksctl.

I could see maybekiam/kube2iam could be integrated in to eksctl bootstrapping, given the AWS IAM policy integration required, which is a legit eksctl bootstrapping role; It would be nice if eksctl handles everything up to point that users could do the rest with their choice of k8s native tools, but no further.

I have used other k8s cluster creation systems that have ‘add ons’, like GKE and kube-aws and every time I have used the add ons I’ve regretted it later due some maintenance drama upgrading or reconfiguring the add-on later.

Also, everyone’s stack of in-cluster add-ons is different and integrated and secured in different ways. Currently supporting everyone’s node/ASG/VPC bootstrapping desires is challenge enough for eksctl without trying to be the kitchen sink of all k8s add-on services combinations.

Totally not my decision but that my personal opinion. I think it somewhat aligns with what @mumoshu is saying, but he is just way more polite 😉

[mumoshu]

@errordeveloper @christopherhein @stefanprodan @whereisaaron FYI, there's a relevant discussion in SIG Cluster Lifecycle to form a shared interface for addon installation and overall lifecycle kubernetes-sigs/cluster-addons#10

[errordeveloper]

there's a relevant discussion in SIG Cluster Lifecycle to form a shared interface for addon installation and overall lifecycle kubernetes-sigs/cluster-addons#10

Yes, indeed. My original intention was to develop something under eksctl umbrella first, and them propose it upstream. However, the upstream conversation has started before we started to work on anything.

I've been meaning to share an update here and close the PR. Myself and other colleagues of mine will continue to take active part in upstream discussions, and once a clear definition of a spec emerges, we are going to look at how it can be implemented in eksctl.