Add ASG lifecycle management Lambda function #392

joshbeard · 2021-10-15T00:20:57Z

Description

This introduces an Auto Scaling Group instance termination lifecycle
hook using Lambda and related resources. The Lambda function is a Python
script that is triggered when the persistent runner instance in the ASG
is terminated. The function receives the instance ID of the "parent"
runner and queries for spawned instances that it launched to terminate.
Additionally, it will check for other "orphaned" instances that have a
gitlab-runner-parent-id tag that doesn't match an existing instance. This
resolves the issue where spawned instances could be orphaned when their
parent runner is terminated.

This feature is disabled by default.

The user data script is updated to provide the 'parent' instance ID as a
tag named 'gitlab-runner-parent-id' on spawned instances.

A new sub-module is provided called "terminate-workers". It is optional
to use this feature, and the input variable
asg_terminate_lifecycle_hook_create can be toggled true or false
for this behavior.

This partially addresses concerns discussed in issue #214

Migrations required

NO

Verification

I've tested existing standard configuration with this addition disabled and enabled with expected results.

On a default setup without enabling this feature, there are 2 changes - the user data is updated to provide the gitlab-runner and gitlab-runner-parent tags to spawned worker instances.
```
Plan: 0 to add, 2 to change, 0 to destroy.
```
When enabled with its default configuration, there are 10 resource additions and the two changes mentioned previously.
```
Plan: 10 to add, 2 to change, 0 to destroy.
```

Documentation

We use pre-commit to update the Terraform inputs and outputs in the documentation via terraform-docs. Ensure you have installed those components.

Documentation was updated, tflint and pre-commit hooks ran.

kayman-mk · 2021-10-16T09:33:34Z

What a nice feature! I need this too.

I deployed a new runner into my development environment and noticed the following:

asg_terminate_lifecycle_hook_create was not set:

the name of my docker+machine instance was changed. It got "gitlab-runner" appended. No idea where this is from. The AWS Console showed "Gitlab Runner eu-central-1b TESTgitlab-runner". I am using name_docker_machine_runners = "Gitlab Runner ${each.value.availability_zone} TEST" to set the name.
my docker+machine instance got a new tag with key = true and value = gitlab-runner-parent. This looks quite strange as the key of the tag is not meaningful.

asg_terminate_lifecycle_hook_create = true:

a new dependency was introduced. Now I need python installed on my machine. Can we get rid of it?

Error: can't find external program "python.exe"
│
│   with module.gitlab_runner.module.gitlab_runner_test["subnet-0bec5588fdad57ecd"].module.terminate_workers_lifecycle_function[0].module.terminate_runner_workers_lambda.data.external.archive_prepare[0],
│   on .terraform\modules\gitlab_runner.gitlab_runner_test.terminate_workers_lifecycle_function.terminate_runner_workers_lambda\package.tf line 7, in data "external" "archive_prepare":
│    7: data "external" "archive_prepare" {

after installing Python I got some strange error messages. The script builds a zip file and then fails with

 zip: adding content of directory: modules/gitlab_runner/gitlab_runner_local/modules/terminate-workers/lambda/
│ zip: adding: lambda_function.py
│ zip: adding: requirements.txt
│ Traceback (most recent call last):
│   File "/mnt/sales-team/platform/gitlab-ci-runner/.terraform/modules/gitlab_runner.gitlab_runner_test.terminate_workers_lifecycle_function.terminate_runner_workers_lambda/package.py", line 1270, in <module>
│     main()
│   File "/mnt/sales-team/platform/gitlab-ci-runner/.terraform/modules/gitlab_runner.gitlab_runner_test.terminate_workers_lifecycle_function.terminate_runner_workers_lambda/package.py", line 1266, in main
│     exit(args.command(args))
│   File "/mnt/sales-team/platform/gitlab-ci-runner/.terraform/modules/gitlab_runner.gitlab_runner_test.terminate_workers_lifecycle_function.terminate_runner_workers_lambda/package.py", line 1158, in build_command
│     bpm.execute(build_plan, zs, query)
│   File "/mnt/sales-team/platform/gitlab-ci-runner/.terraform/modules/gitlab_runner.gitlab_runner_test.terminate_workers_lifecycle_function.terminate_runner_workers_lambda/package.py", line 318, in __exit__
│     self.close()
│   File "/mnt/sales-team/platform/gitlab-ci-runner/.terraform/modules/gitlab_runner.gitlab_runner_test.terminate_workers_lifecycle_function.terminate_runner_workers_lambda/package.py", line 308, in close
│     os.replace(self._tmp_filename, self.filename)
│ FileNotFoundError: [Errno 2] No such file or directory: 'builds/de2e9dd0e52807f975c7bdb95a410fb17ad8cb3648a76229913b8e154420a753.zip.tmp' -> 'builds/de2e9dd0e52807f975c7bdb95a410fb17ad8cb3648a76229913b8e154420a753.zip'

There is a "build" directory but it does not contain the "zip.tmp" but the "zip" file instead. Seems that it has been moved.

kayman-mk · 2021-10-16T09:45:27Z

Ok, got it running with asg_terminate_lifecycle_hook_create = true.

Found the same naming problems as decribed above: Name of instance and tags.
The lifecycle hook is present now. But killing the Agent does not kill my docker+machine runner. I waited for some minutes but it is still there. As my Agent is automatically restartet via the ASG, could it be an issue with the Heartbeat Timeout of the hook? I didn't touch it and expected to work out of the box. It shows 90 seconds.

kayman-mk · 2021-10-16T09:48:31Z

I expect one additional parameter only: terminate_orphaned_spot_instances = true. Everything else should be handled internally.

modules/terminate-workers/lambda/requirements.txt

variables.tf

template/gitlab-runner.tpl

joshbeard · 2021-10-16T16:10:33Z

Thanks for the review, @kayman-mk! I'll work on fixes and adjustments from your feedback. The Python requirement comes from the use of an external module (https://registry.terraform.io/modules/terraform-aws-modules/lambda/) for the Lambda function and is mentioned as a prerequisite when using this feature in the documentation updates in this pull request. However, I think we can get rid of this and handle this simple use case in this module. I'll do some refactoring for this.

On the issue of it not actually doing anything - I'm curious if you can check the CloudWatch event rule to see if it triggered an invocation and if it failed? There's also a CloudWatch log stream for the function itself. On the worker instance, do you see the gitlab-runner-parent tag with a value of the parent instance ID?

kayman-mk · 2021-10-16T17:20:15Z

Nope, I do not see the tag. I guess it's the missing comma in thePARENT_TAG

joshbeard · 2021-10-19T02:34:40Z

I've made adjustments from @kayman-mk's review, including removing the use of an external module and thus eliminating the 'python/pip' dependencies.

npalm · 2021-10-19T21:36:03Z

@joshbeard will check asap, please can you rebase?

joshbeard · 2021-10-21T00:24:53Z

@npalm rebased

npalm

@joshbeard thanks for your contribution. Made some small remakrs, please can youhave a look. I also need todo a bit more testing.

README.md

modules/terminate-workers/README.md

modules/terminate-workers/variables.tf

modules/terminate-workers/cloudwatch.tf

modules/terminate-workers/main.tf

modules/terminate-workers/variables.tf

joshbeard · 2021-10-25T14:13:00Z

@npalm Thank you so much for the thorough review and consideration! I will resolve the issues you pointed out, rebase, and push within the next couple of days.

joshbeard · 2021-11-08T23:52:47Z

@npalm Thank you so much for the thorough review and consideration! I will resolve the issues you pointed out, rebase, and push within the next couple of days.

Couple of days = couple of weeks ;) I've been away for the past couple of weeks.

main.tf

modules/terminate-workers/variables.tf

modules/terminate-workers/iam.tf

modules/terminate-workers/lambda/lambda_function.py

modules/terminate-workers/iam.tf

modules/terminate-workers/versions.tf

joshbeard · 2021-12-02T06:24:02Z

@npalm @kayman-mk I've finally gotten back around to this and have made updates based on your most recent feedback in commit a290efa.

I noticed the README is updated with some unrelated changes from the pre-commit hook. Not sure why. I left it as a separate commit for now to make it easier to adjust if you want me to remove the unrelated changes (which still seem to be accurate?) That's visible specifically in ab8a2b7

I'll rebase this after follow-up reviews and potential tweaks.

npalm · 2021-12-09T20:55:58Z

Great work, added to my backlog.

npalm · 2021-12-16T22:22:39Z

Will check in the weekend

kayman-mk · 2022-01-08T19:47:54Z

@joshbeard @npalm I did a quick check in my environment.

Deploying the module: Success
Killing the runner manually: The executor was killed by the Lambda function. Success
Didn't find any suspicious log entries in the Lambda log.
Executors have a new tag: gitlab-runner-parent-id

What I noticed:

Lambda timeout is 3s. This might be too short as the duration of the call was 2s.
I have to enable the new function via asg_terminate_lifecycle_hook_create. This doesn't feel handy. I think it is a base functionality which should always be enabled to be sure not to waste money.
Log messages are JSON formatted and have a structured field "Level". Really appreciated. Makes log analysis easy.

npalm · 2022-01-13T22:02:18Z

@kayman-mk I also will do a check. Would suggest to make the lambda memory as well the time out configurable with a reasonable default

joshbeard · 2022-01-13T22:09:04Z

@kayman-mk I also will do a check. Would suggest to make the lambda memory as well the time out configurable with a reasonable default

@npalm I've made the lambda memory size and the timeout configurable: https://github.com/npalm/terraform-aws-gitlab-runner/pull/392/files#diff-05b5a57c136b6ff596500bcbfdcff145ef6cddea2a0e86d184d9daa9a65a288e

joshbeard · 2022-01-15T16:44:12Z

Rebased against "develop"

GertVil · 2022-01-30T10:19:38Z

Really looking forward to having this merged, because this was one of the "issues" I documented before we could look into using this module at work. Do you have any idea when this can be merged @npalm? (Thank you for the awesome work!)

npalm · 2022-02-02T07:39:29Z

Really looking forward to having this merged, because this was one of the "issues" I documented before we could look into using this module at work. Do you have any idea when this can be merged @npalm? (Thank you for the awesome work!)

@GertVil I will try to check the PR tonight. Can you provide a short direction how to test the lambda?

GertVil · 2022-02-02T12:12:03Z

Really looking forward to having this merged, because this was one of the "issues" I documented before we could look into using this module at work. Do you have any idea when this can be merged @npalm? (Thank you for the awesome work!)

@GertVil I will try to check the PR tonight. Can you provide a short direction how to test the lambda?

I'm afraid that @joshbeard will probably be in a better position to provide you with that information since he created all this so he probably has a scenario that he has used before.

npalm

@joshbeard great job! works like a charm! Only one small change request, see inline. Can you also run a rebase please.

variables.tf

modules/terminate-instances/cloudwatch.tf

joshbeard · 2022-02-07T18:11:37Z

@joshbeard great job! works like a charm! Only one small change request, see inline. Can you also run a rebase please.

@npalm Adjustments made and rebased against 'develop'

npalm · 2022-02-24T23:27:17Z

Sorry, missed the update. Will merge the PR tomorrow. If you have time for a rebase, please. Othewrise I will fix it locally!

This introduces an Auto Scaling Group instance termination lifecycle hook using Lambda and related resources. The Lambda function is a Python script that is triggered when the persistent runner instance in the ASG is terminated. The function receives the instance ID of the "parent" runner and queries for spawned instances that it launched to terminate. Additionally, it will check for other "orphaned" instances that have a `gitlab-runner-parent-id` tag that doesn't match an existing instance. This resolves the issue where spawned instances could be orphaned when their parent runner is terminated. This feature is disabled by default. The user data script is updated to provide the 'parent' instance ID as a tag named 'gitlab-runner-parent-id' on spawned instances. A new sub-module is provided called "terminate-workers". It is optional to use this feature, and the input variable `asg_terminate_lifecycle_hook_create` can be toggled `true` or `false` for this behavior.

joshbeard · 2022-02-25T00:19:24Z

Rebased again. The README changed a bit since my last rebase, as I'm sure you're aware. I added the resources to the "resources" section, but that may get overwritten with the current pre-commit config.

npalm · 2022-02-25T11:48:20Z

@joshbeard thx, checking. Just a qucik tip for a next time. Fore reviewing it helps if you are not squashing, only rebase. Now I have to go over the full PR again. No worries we get this merged!

npalm

@joshbeard All good, thanks!!!

joshbeard · 2022-02-25T13:42:07Z

@joshbeard thx, checking. Just a qucik tip for a next time. Fore reviewing it helps if you are not squashing, only rebase. Now I have to go over the full PR again. No worries we get this merged!

I'm sorry about that. Thanks for the review.

joshbeard · 2022-02-25T13:43:25Z

@joshbeard All good, thanks!!!

Great! Thanks for your work on this, the review, feedback, and including this in the project!

## [4.40.0](4.39.1...4.40.0) (2022-02-25) ### Features * Add ASG lifecycle management Lambda function ([#392](#392)) ([5beb9d7](5beb9d7)) * Skip runner download and install if it's already done ([#446](#446)) ([54c10f3](54c10f3))

semantic-releaser · 2022-02-25T14:13:09Z

🎉 This PR is included in version 4.40.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

This introduces an Auto Scaling Group instance termination lifecycle hook using Lambda and related resources. The Lambda function is a Python script that is triggered when the persistent runner instance in the ASG is terminated. The function receives the instance ID of the "parent" runner and queries for spawned instances that it launched to terminate. Additionally, it will check for other "orphaned" instances that have a `gitlab-runner-parent-id` tag that doesn't match an existing instance. This resolves the issue where spawned instances could be orphaned when their parent runner is terminated. This feature is disabled by default. The user data script is updated to provide the 'parent' instance ID as a tag named 'gitlab-runner-parent-id' on spawned instances. A new sub-module is provided called "terminate-workers". It is optional to use this feature, and the input variable `asg_terminate_lifecycle_hook_create` can be toggled `true` or `false` for this behavior.

joshbeard mentioned this pull request Oct 15, 2021

Old runners not cleaned up after ASG update #214

Closed

kayman-mk reviewed Oct 16, 2021

View reviewed changes

modules/terminate-workers/lambda/requirements.txt Outdated Show resolved Hide resolved

kayman-mk reviewed Oct 16, 2021

View reviewed changes

variables.tf Outdated Show resolved Hide resolved

kayman-mk reviewed Oct 16, 2021

View reviewed changes

template/gitlab-runner.tpl Outdated Show resolved Hide resolved

joshbeard marked this pull request as draft October 16, 2021 16:11

joshbeard marked this pull request as ready for review October 19, 2021 02:34

npalm requested changes Oct 24, 2021

View reviewed changes

kayman-mk reviewed Nov 13, 2021

View reviewed changes

kayman-mk mentioned this pull request Nov 13, 2021

Occasionally EC2s are left in a Stopped state #398

Closed

npalm requested changes Feb 2, 2022

View reviewed changes

variables.tf Outdated Show resolved Hide resolved

modules/terminate-instances/cloudwatch.tf Outdated Show resolved Hide resolved

npalm approved these changes Feb 25, 2022

View reviewed changes

npalm merged commit 5beb9d7 into cattle-ops:develop Feb 25, 2022

semantic-releaser bot added the released label Feb 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ASG lifecycle management Lambda function #392

Add ASG lifecycle management Lambda function #392

joshbeard commented Oct 15, 2021 •

edited

Loading

kayman-mk commented Oct 16, 2021

kayman-mk commented Oct 16, 2021

kayman-mk commented Oct 16, 2021

joshbeard commented Oct 16, 2021

kayman-mk commented Oct 16, 2021

joshbeard commented Oct 19, 2021

npalm commented Oct 19, 2021

joshbeard commented Oct 21, 2021

npalm left a comment

joshbeard commented Oct 25, 2021

joshbeard commented Nov 8, 2021

joshbeard commented Dec 2, 2021 •

edited

Loading

npalm commented Dec 9, 2021

npalm commented Dec 16, 2021

kayman-mk commented Jan 8, 2022

npalm commented Jan 13, 2022

joshbeard commented Jan 13, 2022

joshbeard commented Jan 15, 2022

GertVil commented Jan 30, 2022

npalm commented Feb 2, 2022

GertVil commented Feb 2, 2022

npalm left a comment •

edited

Loading

joshbeard commented Feb 7, 2022

npalm commented Feb 24, 2022

joshbeard commented Feb 25, 2022

npalm commented Feb 25, 2022

npalm left a comment

joshbeard commented Feb 25, 2022

joshbeard commented Feb 25, 2022

semantic-releaser bot commented Feb 25, 2022

Add ASG lifecycle management Lambda function #392

Add ASG lifecycle management Lambda function #392

Conversation

joshbeard commented Oct 15, 2021 • edited Loading

Description

Migrations required

Verification

Documentation

kayman-mk commented Oct 16, 2021

kayman-mk commented Oct 16, 2021

kayman-mk commented Oct 16, 2021

joshbeard commented Oct 16, 2021

kayman-mk commented Oct 16, 2021

joshbeard commented Oct 19, 2021

npalm commented Oct 19, 2021

joshbeard commented Oct 21, 2021

npalm left a comment

Choose a reason for hiding this comment

joshbeard commented Oct 25, 2021

joshbeard commented Nov 8, 2021

joshbeard commented Dec 2, 2021 • edited Loading

npalm commented Dec 9, 2021

npalm commented Dec 16, 2021

kayman-mk commented Jan 8, 2022

npalm commented Jan 13, 2022

joshbeard commented Jan 13, 2022

joshbeard commented Jan 15, 2022

GertVil commented Jan 30, 2022

npalm commented Feb 2, 2022

GertVil commented Feb 2, 2022

npalm left a comment • edited Loading

Choose a reason for hiding this comment

joshbeard commented Feb 7, 2022

npalm commented Feb 24, 2022

joshbeard commented Feb 25, 2022

npalm commented Feb 25, 2022

npalm left a comment

Choose a reason for hiding this comment

joshbeard commented Feb 25, 2022

joshbeard commented Feb 25, 2022

semantic-releaser bot commented Feb 25, 2022

joshbeard commented Oct 15, 2021 •

edited

Loading

joshbeard commented Dec 2, 2021 •

edited

Loading

npalm left a comment •

edited

Loading