tags: initial implementation of tags #149

nicolehanjing · 2020-10-28T22:19:12Z

first pass of tags implementation

What type of PR is this?
/kind feature

What this PR does / why we need it:
We have two formats of the cluster tags in the legacy implementation:

// TagNameKubernetesClusterPrefix is the tag name we use to differentiate multiple
// logically independent clusters running in the same AZ.
// The tag key = TagNameKubernetesClusterPrefix + clusterID
// The tag value is an ownership value
const TagNameKubernetesClusterPrefix = "kubernetes.io/cluster/"

// TagNameKubernetesClusterLegacy is the legacy tag name we use to differentiate multiple
// logically independent clusters running in the same AZ.  The problem with it was that it
// did not allow shared resources.
const TagNameKubernetesClusterLegacy = "KubernetesCluster"

As @nckturner mentioned before, the current format ("kubernetes.io/cluster/<clusterID>": <ownership>") causes issues for customers trying to use tags to organize billing.

After discussion, we'd like to see tags with formats:

1) kubernetes.io/cluster/ = <clusterID>
2) kubernetes.io/cluster/<clusterID> = ""

the second format is only used for shared resources e.g. subnets

and once we have other workarounds, will make corresponding updates.
Which issue(s) this PR fixes:

Part of #125

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

pkg/providers/v2/cloud.go

pkg/providers/v2/tags.go

nckturner · 2020-11-13T18:58:17Z

So just to summarize our conversation from this morning, we need to keep the kubernetes.io/cluster/<clusterID> = shared|owned for discovery, but we will allow passing in additionalTags (or something similar) via the config to aid in billing and other tag use cases, like where users automatically delete all resources from an account that aren't tagged a certain way. It also makes sense to use "tag-on-create" where possible because we believe it is better supported (although I am verifying that). There have been some changes to the APIs since V1 was written, so we should take advantage of those.

nckturner · 2020-11-13T19:38:34Z

So actually as of now I think the resource tagging API is fully supported in all regions, so I don't think we need to prefer tag-on-create for that reason. Tag-on-create might be nice for other reasons, like being eaiser to implement and fewer total number of api calls.

nicolehanjing · 2020-11-13T21:58:03Z

So just to summarize our conversation from this morning, we need to keep the kubernetes.io/cluster/<clusterID> = shared|owned for discovery, but we will allow passing in additionalTags (or something similar) via the config to aid in billing and other tag use cases, like where users automatically delete all resources from an account that aren't tagged a certain way. It also makes sense to use "tag-on-create" where possible because we believe it is better supported (although I am verifying that). There have been some changes to the APIs since V1 was written, so we should take advantage of those.

Thanks @nckturner for the summary
for my current implementation, I only used the first format which is kubernetes.io/cluster/ = <clusterID>, but as we are keeping kubernetes.io/cluster/<clusterID> = shared|owned, I will need to keep both of them in this initial implementation, right?

andrewsykim

Sorry for the late review @nicolehanjing

andrewsykim · 2020-12-08T15:08:12Z

pkg/providers/v2/cloud.go

@@ -61,6 +77,30 @@ type EC2Metadata interface {
 	GetMetadata(path string) (string, error)
 }

+func readAWSCloudConfig(config io.Reader) (*v1alpha1.AWSCloudConfig, error) {
+	if config == nil {
+		return nil, fmt.Errorf("no AWS cloud provider config file given")


nit: use errors.New instead since there's no args

andrewsykim · 2020-12-08T15:08:39Z

pkg/providers/v2/cloud.go

+		return nil, err
+	}
+	if cfg.Kind != "AWSCloudConfig" {
+		return nil, fmt.Errorf("invalid cloud configuration object %q", cfg.Kind)


"invalid Kind for cloud config: %q"

andrewsykim · 2020-12-08T15:09:13Z

pkg/providers/v2/cloud.go

@@ -82,7 +122,7 @@ func azToRegion(az string) (string, error) {
 }

 // newCloud creates a new instance of AWSCloud.
-func newCloud() (cloudprovider.Interface, error) {
+func newCloud(cfg v1alpha1.AWSCloudConfig) (cloudprovider.Interface, error) {


nit: pass in the pointer

andrewsykim · 2020-12-08T15:12:32Z

pkg/providers/v2/tags.go

+	return clusterName, nil
+}
+
+func (t *awsTagging) init(clusterName string) error {


In general I tend to avoid naming functions init because init is a reserved function name in Go

Recommend creating a constructor function for awsTagging like NewAWSTags and have it receive clusterName as one of the parameters

andrewsykim · 2020-12-08T15:14:45Z

pkg/providers/v2/cloud.go

@@ -53,6 +61,14 @@ type cloud struct {
 	region    string
 	ec2       EC2
 	metadata  EC2Metadata
+	tagging   awsTagging


nit: just name this tags?

pkg/providers/v2/tags.go

nicolehanjing · 2020-12-08T19:16:40Z

@andrewsykim updated! PTAL :)

nckturner · 2020-12-08T23:21:09Z

pkg/providers/v2/tags.go

+}
+
+func (t *awsTagging) hasClusterTag(tags []*ec2.Tag) bool {
+	// if the clusterName is not configured -- we consider all instances.


Nit: wondering if this logic can be refactored slightly. IMO returning true from a function called hasClusterTag when there isn't a tag to look for is bit confusing (for the sole purpose of causing some side effect behavior in the caller), wondering if its worth separating return values to have distinct values for "is cluster tag configured" and "does cluster tag exist". We could return a specific error (or bool value) that the caller can check for if cluster tag is not configured.

good point, personally I prefer to return a specific error with some info for the caller to check, will update!

pkg/providers/v2/tags.go

ayberk · 2020-12-09T01:52:13Z

pkg/providers/v2/cloud.go

+		// we got an error where the decode wasn't related to a missing type
+		return nil, err
+	}
+	if cfg.Kind != "AWSCloudConfig" {


Are there any other required fields we have to verify here?

Good call, I think for this PR we don't technically have to validate anything since ClusterName is optional (I think) but there will be a need to validate field values at some point. Here's one example of how to do it:

https://github.com/kubernetes/kubernetes/blob/5344afd4fbfc3cb2ed6785d89b6f760886f100b5/pkg/credentialprovider/plugin/config.go#L71-L128

pkg/providers/v2/tags.go

ayberk · 2020-12-09T02:42:13Z

pkg/providers/v2/tags.go

+
+	// no clusterName is a sign of misconfigured cluster, but we can't be tagging the resources with empty
+	// strings
+	if len(t.ClusterName) == 0 {


If this is a sign of misconfigured cluster, we should log a warning here.

Does this mean clusterName is actually required? If so we should validate it when we read the config and return an error if it's empty. I thought clusterName was optional and it can be discovered through instances but I might be mis-remembering the details.

I think requiring clusterName is reasonable, it's an explicit step from the user requiring them to understand how/why the cluster name is needed.

If there's a way to infer the cluster name, I think that'd be ideal. "We will add the cluster name automatically, but you can override it via config" sounds reasonable to me. Although I'm not sure if there's a case where the user would want to override it.

yeah I agree, will figure out if we can discover the cluster name through instances
for now I leave a TODO note here and a warning log in https://github.com/kubernetes/cloud-provider-aws/pull/149/files#diff-78811b2c178b96d8cf3e546ccbc24defbe15e8ef228da31b36d17ecfe29174ceR183

pkg/providers/v2/tags.go

pkg/providers/v2/cloud.go

andrewsykim · 2020-12-09T20:10:57Z

pkg/providers/v2/cloud_test.go

+	for _, testcase := range testcases {
+		t.Run(testcase.name, func(t *testing.T) {
+			var buffer bytes.Buffer
+			_, err := buffer.WriteString(testcase.configData)


This can be simplieifed to:

bytes.NewBufferString(testcase.configData)

andrewsykim

We also need to update the Instances implementation to use the clusterName tags for API calls.

nicolehanjing · 2020-12-10T02:25:26Z

pkg/providers/v2/cloud.go

+			return nil, fmt.Errorf("failed to read AWS cloud provider config file: %v", err)
+		}
+
+		errs := validateAWSCloudConfig(cfg)


@andrewsykim updated! PTAL :)

added validation function & unit tests

update the Instances implementation to use the clusterName tags for API calls

andrewsykim · 2020-12-14T21:47:18Z

pkg/providers/v2/cloud.go

 	cloudprovider "k8s.io/cloud-provider"
+	"k8s.io/cloud-provider-aws/pkg/apis/config/v1alpha1"


import alias this to awsconfigv1alpha1

andrewsykim · 2020-12-14T21:48:10Z

pkg/providers/v2/cloud.go

+		allErrs = append(allErrs, field.Invalid(field.NewPath("Kind"), "invalid Kind for cloud config: %q", config.Kind))
+	}
+
+	if config.APIVersion != "config.aws.io/v1alpha1" {


use awsconfigv1alpha1.SchemeGroupVersion.String() instead of config.aws.io/v1alpha1

andrewsykim · 2020-12-14T21:48:32Z

pkg/providers/v2/cloud.go

+	allErrs := field.ErrorList{}
+
+	if config.Kind != "AWSCloudConfig" {
+		allErrs = append(allErrs, field.Invalid(field.NewPath("Kind"), "invalid Kind for cloud config: %q", config.Kind))


field value should be the json serialized version field.NewPath("kind")

andrewsykim · 2020-12-14T21:48:41Z

pkg/providers/v2/cloud.go

+	}
+
+	if config.APIVersion != "config.aws.io/v1alpha1" {
+		allErrs = append(allErrs, field.Invalid(field.NewPath("APIVersion"), "invalid APIVersion for cloud config: %q", config.APIVersion))


APIVersion -> apiVersion

andrewsykim · 2020-12-14T21:48:52Z

pkg/providers/v2/cloud.go

+		allErrs = append(allErrs, field.Invalid(field.NewPath("APIVersion"), "invalid APIVersion for cloud config: %q", config.APIVersion))
+	}
+
+	fieldPath := field.NewPath("Config")


Config -> config

andrewsykim · 2020-12-14T21:49:16Z

pkg/providers/v2/cloud.go

+	}
+
+	fieldPath := field.NewPath("Config")
+	if config.Config == (v1alpha1.AWSConfig{}) {


I don't think this check is required, it would fail in field-specific validation anyways

andrewsykim · 2020-12-14T21:49:25Z

pkg/providers/v2/cloud.go

+	}
+
+	if len(config.Config.ClusterName) == 0 {
+		allErrs = append(allErrs, field.Required(fieldPath.Child("ClusterName"), "cluster name cannot be empty"))


ClusterName -> clusterName

nicolehanjing · 2021-01-12T20:03:21Z

@andrewsykim @nckturner @ayberk PTAL :)

andrewsykim

/approve
/lgtm

There's a few TODOs in the comments and the code but I think we can address those with #156

k8s-ci-robot · 2021-02-02T16:06:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, nicolehanjing

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andrewsykim]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from andrewsykim and nckturner October 28, 2020 22:19

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 28, 2020

nicolehanjing force-pushed the nicoleh-tags branch 6 times, most recently from 8139e29 to 5b1e59d Compare November 4, 2020 19:01

nicolehanjing force-pushed the nicoleh-tags branch 3 times, most recently from 0f2fe6d to 891e3e0 Compare November 12, 2020 00:41

nicolehanjing changed the title ~~[WIP] tags: initial implementation of tags~~ tags: initial implementation of tags Nov 12, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 12, 2020

andrewsykim reviewed Nov 12, 2020

View reviewed changes

pkg/providers/v2/cloud.go Outdated Show resolved Hide resolved

pkg/providers/v2/tags.go Show resolved Hide resolved

nicolehanjing force-pushed the nicoleh-tags branch from 891e3e0 to 3d37978 Compare November 12, 2020 06:03

andrewsykim reviewed Dec 8, 2020

View reviewed changes

nicolehanjing force-pushed the nicoleh-tags branch from 53c8739 to 6e7dfe7 Compare December 8, 2020 19:15

nckturner reviewed Dec 8, 2020

View reviewed changes

nicolehanjing force-pushed the nicoleh-tags branch from 6e7dfe7 to 42c3beb Compare December 8, 2020 23:55

ayberk reviewed Dec 9, 2020

View reviewed changes

pkg/providers/v2/tags.go Outdated Show resolved Hide resolved

ayberk reviewed Dec 9, 2020

View reviewed changes

pkg/providers/v2/tags.go Outdated Show resolved Hide resolved

ayberk reviewed Dec 9, 2020

View reviewed changes

pkg/providers/v2/tags.go Show resolved Hide resolved

nicolehanjing force-pushed the nicoleh-tags branch from 42c3beb to 6c30210 Compare December 9, 2020 06:13

andrewsykim reviewed Dec 9, 2020

View reviewed changes

pkg/providers/v2/tags.go Show resolved Hide resolved

pkg/providers/v2/cloud.go Outdated Show resolved Hide resolved

nicolehanjing force-pushed the nicoleh-tags branch from 6c30210 to b5229bd Compare December 9, 2020 19:38

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 9, 2020

andrewsykim reviewed Dec 9, 2020

View reviewed changes

nicolehanjing force-pushed the nicoleh-tags branch from b5229bd to d31cbae Compare December 9, 2020 23:48

nicolehanjing commented Dec 10, 2020

View reviewed changes

tags: initial implementation of tags

7b5dbaf

nicolehanjing force-pushed the nicoleh-tags branch from d31cbae to 253ca08 Compare December 11, 2020 23:01

andrewsykim reviewed Dec 14, 2020

View reviewed changes

tags: initial implementation of tags

fb56cd9

nicolehanjing force-pushed the nicoleh-tags branch from 253ca08 to fb56cd9 Compare December 15, 2020 07:20

andrewsykim reviewed Feb 2, 2021

View reviewed changes

k8s-ci-robot assigned andrewsykim Feb 2, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 2, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 2, 2021

k8s-ci-robot merged commit d8f0022 into kubernetes:master Feb 2, 2021

ayberk mentioned this pull request Feb 2, 2021

REQUEST: New membership for ayberk kubernetes/org#2481

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tags: initial implementation of tags #149

tags: initial implementation of tags #149

nicolehanjing commented Oct 28, 2020 •

edited

Loading

nckturner commented Nov 13, 2020

nckturner commented Nov 13, 2020

nicolehanjing commented Nov 13, 2020 •

edited

Loading

andrewsykim left a comment

andrewsykim Dec 8, 2020

nicolehanjing Dec 8, 2020

andrewsykim Dec 8, 2020

andrewsykim Dec 8, 2020

andrewsykim Dec 8, 2020

andrewsykim Dec 8, 2020

andrewsykim Dec 8, 2020

nicolehanjing commented Dec 8, 2020

nckturner Dec 8, 2020

nicolehanjing Dec 8, 2020

ayberk Dec 9, 2020

andrewsykim Dec 9, 2020

ayberk Dec 9, 2020

andrewsykim Dec 9, 2020

andrewsykim Dec 9, 2020

ayberk Dec 9, 2020

nicolehanjing Dec 9, 2020

andrewsykim Dec 9, 2020

andrewsykim left a comment

nicolehanjing Dec 10, 2020

andrewsykim Dec 14, 2020

andrewsykim Dec 14, 2020

andrewsykim Dec 14, 2020

andrewsykim Dec 14, 2020

andrewsykim Dec 14, 2020

andrewsykim Dec 14, 2020

andrewsykim Dec 14, 2020

nicolehanjing commented Jan 12, 2021

andrewsykim left a comment

k8s-ci-robot commented Feb 2, 2021

		cloudprovider "k8s.io/cloud-provider"
		"k8s.io/cloud-provider-aws/pkg/apis/config/v1alpha1"

tags: initial implementation of tags #149

tags: initial implementation of tags #149

Conversation

nicolehanjing commented Oct 28, 2020 • edited Loading

nckturner commented Nov 13, 2020

nckturner commented Nov 13, 2020

nicolehanjing commented Nov 13, 2020 • edited Loading

andrewsykim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolehanjing commented Dec 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewsykim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolehanjing commented Jan 12, 2021

andrewsykim left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 2, 2021

nicolehanjing commented Oct 28, 2020 •

edited

Loading

nicolehanjing commented Nov 13, 2020 •

edited

Loading