Skip to content

Commit 69ff087

Browse files
committed
Fix race condition if a workflow job gets canceled before the runner VM accepted the job leading to an idling VM, always default to latest GitHub runner release if 'github_runner_download_url' is left empty
1 parent 5001a86 commit 69ff087

File tree

3 files changed

+19
-9
lines changed

3 files changed

+19
-9
lines changed

README.md

+4-3
Original file line numberDiff line numberDiff line change
@@ -149,12 +149,13 @@ Error applying IAM policy for cloudrun service "v1/projects/github-spot-runner/l
149149

150150
2. Solution: Override the Organization Policy "Domain Restricted Sharing" in the project, by setting it to "Allow all".
151151

152-
#### The VM Instance immediately stops after it was created without processing a workflow job
152+
#### The VM instance stops shortly after it was created without processing a workflow task
153153

154-
The VM will shoutdown itself if the registration at the GitHub runner group fails. This can be caused by:
155-
* An invalid jit-config.
154+
The VM will stop itself if the registration at the GitHub runner group fails. This can be caused by:
156155
* A typo in the GitHub Enterprise, Organization, Repository name. Check the Terraform variables `github_enterprise`, `github_organization`, `github_repositories` for typos.
157156
* A not existing GitHub runner group within the Enterprise/Organization. Check the Terraform variable `github_runner_group` for typos.
157+
* The GitHub runner version is [deprecated](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners#controlling-runner-software-updates-on-self-hosted-runners). The GitHub runner won't accept any Workflow job. Check the Terraform variable `github_runner_download_url` and update to latest GitHub runner version.
158+
* An invalid jit-config.
158159

159160
You can observer the runner registration process by connecting to the VM instance via SSH and running:
160161
```

compute.tf

+13-4
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,13 @@ apt-get update && apt-get -y install docker.io docker-buildx curl jq ${local.git
8686
useradd -d /home/agent -u ${var.github_runner_uid} agent
8787
usermod -aG docker agent
8888
newgrp docker
89-
curl -s -o /tmp/agent.tar.gz -L '${var.github_runner_download_url}'
89+
RUNNER_DOWNLOAD_URL='${var.github_runner_download_url}'
90+
if [ -z "$${RUNNER_DOWNLOAD_URL}" ]; then
91+
RUNNER_VERSION=$(curl -s "https://github.com/actions/runner/tags/" | grep -Eo "$Version v[0-9]+.[0-9]+.[0-9]+" | sort -r | head -n1 | tr -d ' ' | tr -d 'v')
92+
echo "Downloading latest runner v$${RUNNER_VERSION}"
93+
RUNNER_DOWNLOAD_URL="https://github.com/actions/runner/releases/download/v$${RUNNER_VERSION}/actions-runner-linux-x64-$${RUNNER_VERSION}.tar.gz"
94+
fi
95+
curl -s -o /tmp/agent.tar.gz -L $${RUNNER_DOWNLOAD_URL}
9096
mkdir -p /home/agent
9197
chown -R agent:agent /home/agent
9298
pushd /home/agent
@@ -95,14 +101,17 @@ encoded_jit_config=$1
95101
echo -n $encoded_jit_config | base64 -d | jq '.".runner"' -r | base64 -d > .runner
96102
echo -n $encoded_jit_config | base64 -d | jq '.".credentials"' -r | base64 -d > .credentials
97103
echo -n $encoded_jit_config | base64 -d | jq '.".credentials_rsaparams"' -r | base64 -d > .credentials_rsaparams
98-
sed -i 's/{{SvcNameVar}}/actions.runner.${var.github_organization}.$${agent_name}.service/g' bin/systemd.svc.sh.template
99-
sed -i 's/{{SvcDescription}}/GitHub Actions Runner (${var.github_organization}.$${agent_name})/g' bin/systemd.svc.sh.template
104+
sed -i 's/{{SvcNameVar}}/actions.runner.service/g' bin/systemd.svc.sh.template
105+
sed -i 's/{{SvcDescription}}/GitHub Actions Runner/g' bin/systemd.svc.sh.template
100106
cp bin/systemd.svc.sh.template ./svc.sh && chmod +x ./svc.sh
101107
./bin/installdependencies.sh || shutdown now
102108
./svc.sh install agent || shutdown now
103109
./svc.sh start || shutdown now
104110
popd
105111
rm /tmp/agent.tar.gz
106-
echo "Setup finished"
112+
echo "Setup finished - waiting for Workflow Job"
113+
sleep 60
114+
journalctl -u actions.runner.service -o json --no-pager | jq -e '.|.MESSAGE|match("Running job:")' || shutdown now
115+
echo "Accepted Workflow Job - processing"
107116
EOT
108117
}

variables.tf

+2-2
Original file line numberDiff line numberDiff line change
@@ -94,8 +94,8 @@ variable "github_runner_prefix" {
9494

9595
variable "github_runner_download_url" {
9696
type = string
97-
description = "The download link pointing to the gitlab runner package"
98-
default = "https://github.com/actions/runner/releases/download/v2.321.0/actions-runner-linux-x64-2.321.0.tar.gz"
97+
description = "A download link pointing to the gitlab runner package (WARNING: deprecated runner versions won't process jobs). If this variable is empty (by default), the latest runner release will be downloaded."
98+
default = ""
9999
}
100100

101101
variable "github_runner_uid" {

0 commit comments

Comments
 (0)