/etc/eks/bootstrap.sh devops3-vlocity-kube --kubelet-extra-args "--node-labels=jenkins=true --register-with-taints=dedicated=jenkins:NoSchedule"

[Sam123ben]

What happened:
I dont see the taints happening but can find the labels got modified on the nodes whjich is good. But I dont see the nodes got tainted as I was expecting.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• AWS Region:
• Instance Type(s):
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion):
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version):
• AMI Version:
• Kernel (e.g. uname -a):
• Release information (run cat /tmp/release on a node):

[Sam123ben]

I used the below command with the double quotes and again tried with the single quotes as well.
/etc/eks/bootstrap.sh devops3-vlocity-kube --kubelet-extra-args "--node-labels=jenkins=true --register-with-taints=dedicated=jenkins:NoSchedule"

[gacopl]

Hello mate, it's because you are not escaping stuff properly. Such user-data works for me without problem

/etc/eks/bootstrap.sh plab --kubelet-extra-args "--node-labels=role=elastic-node --eviction-hard=\"memory.available<500Mi,nodefs.available<2Gi,nodefs.inodesFree<5%\" --register-with-taints=\"role=elastic-node:NoSchedule\" --system-reserved=memory=1Gi --kube-reserved=memory=100Mi"

[micahhausler]

As @gacopl noted, you need to properly escape quotes. Does that fix the issue for you?

[Bwvolleyball]

Does anyone happen to know how those escapes change if the script is being called via terraform?

Here's how we set up the call to the script in terraform:

Editing this to what works for us:

locals {
  node-data = <<DATA
#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh --kubelet-extra-args '--register-with-taints="environment=dev:NoSchedule"  --node-labels=com.site/environment=dev' terraform-eks
/opt/aws/bin/cfn-signal --exit-code $? \
	 --stack  terraform-eks \
	 --resource NodeGroup  \
	 --region us-west-2  \
DATA
}

[JessieAMorris]

This is what I'm using:

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh --apiserver-endpoint '${aws_eks_cluster.eks.endpoint}' --b64-cluster-ca '${aws_eks_cluster.eks.certificate_authority.0.data}' --kubelet-extra-args '--register-with-taints="ip=rotating:NoExecute" --node-labels="ip=rotating"' '${var.cluster_name}'
USERDATA

[christiangda]

As most of you mentioned, the problems are the quotes, I'm using cfn template to deploy my nodes and after spent some time checking it, I created a custom userData script:

        UserData:
          Fn::Base64: !Sub |

            #!/bin/bash
            set -e
            set -o xtrace

            CMD_EXTRA_ARG=""
            FIXED_NODES_LABELS="aws.eks.cluster=${ClusterName},aws.eks.nodegroup=${NodeGroupName}"
            FIXED_NODES_TAINTS=""

            # Add more arguments from parameter
            if [ ! -z "${AditionalExtraArguments}" ]
            then
              CMD_EXTRA_ARG=$CMD_EXTRA_ARG"${AditionalExtraArguments}"
            fi

            # Add more labels from parameter
            if [ ! -z "${CustomNodeLabels}" ]
            then
              if [ ! -z "$FIXED_NODES_LABELS" ]
              then
                NODES_LABELS=$FIXED_NODES_LABELS,${CustomNodeLabels}
              else
                NODES_LABELS=${CustomNodeLabels}
              fi
              CMD_EXTRA_ARG=$CMD_EXTRA_ARG" --node-labels="$NODES_LABELS
            fi

            # Add more taints from parameter
            if [ ! -z "${CustomNodeTaints}" ]
            then
              if [ ! -z "$FIXED_NODES_TAINTS" ]
              then
                NODES_TAINTS=$FIXED_NODES_TAINTS,${CustomNodeTaints}
              else
                NODES_TAINTS="\"${CustomNodeTaints}\""
              fi
              CMD_EXTRA_ARG="$CMD_EXTRA_ARG --register-with-taints=$NODES_TAINTS"
            fi

            if [ ! -z "$CMD_EXTRA_ARG" ]
            then
              /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments} --kubelet-extra-args "$CMD_EXTRA_ARG"
            else
              /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
            fi

            /opt/aws/bin/cfn-signal --exit-code $? \
                    --stack ${AWS::StackName} \
                    --resource NodeGroup  \
                    --region ${AWS::Region}

My cfn parameters template are

  CustomNodeLabels:
    Type: String
    Default: ""
    Description: |
      String in format key1=value1,key2=value2,... see: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
      Example: custom.workload.type=monitoring,custom.company=SomeCompany

  CustomNodeTaints:
    Type: String
    Default: ""
    Description: |
      String in format key1=value1:NoSchedule,key2=value2:NoExecute,... see: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
      Example: dedicated=monitoring:NoSchedule

  BootstrapArguments:
    Type: String
    Default: ""
    Description: |
      Arguments to pass to the bootstrap script. Do not include '--kubelet-extra-args', see files/bootstrap.sh in https://github.com/awslabs/amazon-eks-ami
      see: https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh
      Example: --use-max-pods false --enable-docker-bridge true

  AditionalExtraArguments:
    Type: String
    Default: ""
    Description: |
      Aditional --kubelet-extra-args arguments, do not include labels or taints
      see: https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh
      Example: --system-reserved=memory=1Gi --kube-reserved=memory=256Mi

I've checked the file 30-kubelet-extra-args.conf, and all the parameters are there

[root@ip-10-60-16-95 log]# cat /etc/systemd/system/kubelet.service.d/30-kubelet-extra-args.conf
[Service]
Environment='KUBELET_EXTRA_ARGS=--system-reserved=memory=1Gi --kube-reserved=memory=256Mi --node-labels=aws.eks.cluster=staging-01,aws.eks.nodegroup=sad-01,custom.workload.type=monitoring,custom.company=SomeCompany --register-with-taints="dedicated=monitoring:NoSchedule"'

This is not the best way to solve it, but maybe work for you

[Erokos]

Hi,
I also use Terraform and like to use the template resource:

data "template_file" "eks_node_userdata" {
  template = "${file("${path.module}/user-data.sh")}"

  vars {
      kubeconfig_cert_auth_data = "fill in the blanks"
      cluster_endpoint          = "fill in the blanks"
      cluster_name              = "${var.eks_cluster_name}"
      node_label                = "${lookup(var.worker_launch_config_lst[count.index], "kubelet_extra_args", local.worker_lt_defaults["kubelet_extra_args"])}"
      node_taint                = "${lookup(var.worker_launch_config_lst[count.index], "node_taint", "")}"
      ami_id                    = "${lookup(var.worker_launch_config_lst[count.index], "eks_ami_id", local.worker_lt_defaults["eks_ami_id"])}"
      instance_type             = "${lookup(var.worker_launch_config_lst[count.index], "instance_type", "")}"
      docker_config_json        = "${jsonencode(lookup(var.worker_launch_config_lst[count.index], "docker_config_json", ""))}"
  }
}

#!/bin/bash -xe

/etc/eks/bootstrap.sh --use-max-pods true --b64-cluster-ca ${kubeconfig_cert_auth_data} \
--apiserver-endpoint ${cluster_endpoint} ${cluster_name} \
--kubelet-extra-args '--register-with-taints="${node_taint}" --node-labels="${node_label},ami_id=${ami_id},instance_type=${instance_type}"' --docker-config-json ${docker_config_json}

This works for me.
Hope that helps.

[cartermckinnon]

This appears to be resolved.