kubelet maxPods value getting wrongly set from Karpenter

[Uthpal-p]

Description

Observed Behavior:
On setting spec.kubelet.maxPods = "x" on the ec2NodeClass manifest, the nodes that get created using this ec2NodeClass manifest have a different value "y" getting passed to the max-pods arg of kubelet.

This issue is seen only on old ec2NodeClasses that were migrated from v0.37.5 to v1.0.6. The maxPods spec was added to the ec2NodeClass manifest after the migration to v1.0.6. The maxPods value is correctly getting passed to the kubelet on ec2NodeClasses that are newly created after the karpenter version upgrade.

Expected Behavior:
On setting spec.kubelet.maxPods = "x" on the ec2NodeClass manifest, the nodes that get created using this ec2NodeClass manifest should have the same value "x" getting passed to the max-pods arg of "/usr/bin/kubelet" cmd.

Reproduction Steps (Please include YAML):

1. Create a ec2cn and nodepool pair in v0.37.5 karpenter version
2. Upgrade the Karpenter Version to v1.0.6
3. Add max-pods spec under spec.kubelet.maxPods
4. The new nodes created using the above spec will have randomly generated maxPods value passed to kubelet.

ec2nc and nodepool YAML:
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
annotations:
karpenter.sh/stored-version-migrated: "true"
finalizers:

• karpenter.k8s.aws/termination
  name: temp-nodeclass
  spec:
  amiSelectorTerms:
• alias: al2@latest
  blockDeviceMappings:
• deviceName: /dev/xvda
  ebs:
  deleteOnTermination: true
  volumeSize: 500Gi
  volumeType: gp3
  kubelet:
  clusterDNS:
  • 10.42.0.10
    maxPods: 30
    metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 1
    httpTokens: required
    role: eks-role

---

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
annotations:
compatibility.karpenter.sh/v1beta1-kubelet-conversion: '{"clusterDNS":["10.42.0.10"]}'
compatibility.karpenter.sh/v1beta1-nodeclass-reference: '{"kind":"EC2NodeClass","name":"temp-nodeclass","apiVersion":"karpenter.k8s.aws/v1beta1"}'
karpenter.sh/nodepool-hash-version: v3
karpenter.sh/stored-version-migrated: "true"
name: temp-nodepool
spec:
disruption:
budgets:
- nodes: 10%
consolidateAfter: 300s
consolidationPolicy: WhenEmpty
template:
metadata:
labels:
node_type: al_node
spec:
expireAfter: Never
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: temp-nodeclass
requirements:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- key: kubernetes.io/os
operator: In
values:
- linux
- key: karpenter.sh/capacity-type
operator: In
values:
- on-demand
- key: karpenter.k8s.aws/instance-category
operator: In
values:
- m
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values:
- "5"

Versions:

• Chart Version:
  v1.0.6
• Kubernetes Version (kubectl version):
  EKS Kubernetes v1.30

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

As part of the migration, Karpenter adds a compatibility.karpenter.sh/v1beta1-kubelet-conversion annotation to the NodePool to maintain compatibility between conversions between v1beta1 and v1. It's mentioned in the v1 upgrade guide that in order for Karpenter to respect the actual values that are set in the EC2NodeClass over that old annotation, you have to drop that annotation from your NodePool.

So, I suspect if you remove the old annotation from the NodePool, the new maxPods should be respected and everything should start working. Note that when you do this, you will most likely see drift in your NodeClaims because you are changing the configuration that Karpenter is using to evaluate for drift.