☁️ Terraform AWS Spacelift Workerpool On EC2

Terraform module deploying a Spacelift worker pool on AWS EC2 using an autoscaling group.

This module can optionally deploy a Lambda function to auto-scale the worker pool. The function adds or removes workers depending on the worker pool queue length.

✨ Usage

SaaS

The most important is that you should provide SPACELIFT_TOKEN and SPACELIFT_POOL_PRIVATE_KEY environmental variables in the secure_env_vars variable to the module. More information can be found in the docs.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

module "my_workerpool" {
  source = "github.com/spacelift-io/terraform-aws-spacelift-workerpool-on-ec2?ref=v2.13.0"
  
  secure_env_vars = {
    SPACELIFT_TOKEN            = var.worker_pool_config
    SPACELIFT_POOL_PRIVATE_KEY = var.worker_pool_private_key
  }

  min_size          = 1
  max_size          = 5
  worker_pool_id    = var.worker_pool_id
  security_groups   = var.worker_pool_security_groups
  vpc_subnets       = var.worker_pool_subnets
}

Note: Previous versions of this module placed the token and private key directly into the configuration variable. This is still supported, but it is recommended to use the secure_env_vars variable instead as this will store the values in secrets manager instead of in the userdata which is insecure.

You can also set the optional secure_env_vars_kms_key_id to a kms key id to use for encrypting the secure strings in secrets manager. This defaults to the default kms key that AWS uses.

You also need to add the required values for spacelift_api_key_endpoint, spacelift_api_key_id, spacelift_api_key_secret and worker_pool_id to the module block for the Lambda Autoscaler function to set the required SPACELIFT_API_KEY_ENDPOINT, SPACELIFT_API_KEY_ID, SPACELIFT_API_KEY_SECRET_NAME and SPACELIFT_WORKER_POOL_ID parameters.

You can also set the optional autoscaling_max_create and autoscaling_max_terminate values in the module block for Lambda Autoscaler function to set the optional AUTOSCALING_MAX_CREATE and AUTOSCALING_MAX_KILL parameters. These parameters default to 1. These parameters set the maximum number of instances the utility is allowed to create or terminate in a single run.

Self-hosted

For self-hosted, other than the aforementioned SPACELIFT_TOKEN and SPACELIFT_POOL_PRIVATE_KEY variables, you also need to provide the selfhosted_configuration variable. In selfhosted_configuration, the only mandatory field is s3_uri which should point to the location of the launcher binary in S3:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

module "my_workerpool" {
  source = "github.com/spacelift-io/terraform-aws-spacelift-workerpool-on-ec2?ref=v2.13.0"

  secure_env_vars = {
    SPACELIFT_TOKEN = var.worker_pool_config
    SPACELIFT_POOL_PRIVATE_KEY = var.worker_pool_private_key
  }

  min_size                 = 1
  max_size                 = 5
  worker_pool_id           = var.worker_pool_id
  security_groups          = var.worker_pool_security_groups
  vpc_subnets              = var.worker_pool_subnets
  enable_autoscaling       = false
  selfhosted_configuration = {
    s3_uri = "s3://spacelift-binaries-123ab/spacelift-launcher"
  }
}

Note: the module will parse the s3_uri and set s3:GetObject IAM permission accordingly. However, if the S3 bucket is KMS encrypted, it will fail. In that case, you can create a custom instance profile for yourself and provide it via the custom_iam_role_name variable.

Default AMI

The default AMI used by this module comes from the spacelift-worker-image repository. You can find the full list of AMIs on the releases page.

ARM-based AMI

You can use an ARM-based AMI by setting the ami_id variable to an arm64 AMI, and ec2_instance_type to an ARM-based instance type (e.g. t4g.micro).

We recommend using Spacelift AMIs because they come with every required tool preinstalled.

You can find an example of ARM-based workerpool in the examples directory.

❗️ If you use custom runner images, make sure they support ARM. The default Spacelift images do support it.

📚 How to generate docs

The generated documentation is between BEGIN_TF_DOCS and END_TF_DOCS comments in the README.md file. Use the following command to update the docs:

$ make docs

Requirements

Name	Version
aws	>= 5.55.0

Providers

Name	Version
aws	>= 5.55.0
local	n/a
null	n/a

Modules

Name	Source	Version
asg	terraform-aws-modules/autoscaling/aws	~> 8.0

Resources

Name	Type
aws_cloudwatch_event_rule.scheduling	resource
aws_cloudwatch_event_target.scheduling	resource
aws_cloudwatch_log_group.log_group	resource
aws_iam_instance_profile.this	resource
aws_iam_policy.secure_env_vars	resource
aws_iam_role.autoscaler	resource
aws_iam_role.this	resource
aws_iam_role_policy.autoscaler	resource
aws_iam_role_policy.s3	resource
aws_iam_role_policy_attachment.secure_env_vars	resource
aws_iam_role_policy_attachment.this	resource
aws_lambda_function.autoscaler	resource
aws_lambda_permission.allow_cloudwatch_to_call_lambda	resource
aws_secretsmanager_secret.this	resource
aws_secretsmanager_secret_version.this	resource
aws_ssm_parameter.spacelift_api_key_secret	resource
null_resource.download	resource
null_resource.token_check	resource
aws_ami.this	data source
aws_iam_policy_document.autoscaler	data source
aws_iam_policy_document.secure_env_vars	data source
aws_partition.current	data source
aws_region.this	data source
local_file.autoscaler_zip	data source

Inputs

Name	Description	Type	Default	Required
additional_tags	Additional tags to set on the resources	map(string)	{}	no
ami_id	ID of the Spacelift AMI. If left empty, the latest Spacelift AMI will be used.	string	""	no
autoscaler_architecture	Instruction set architecture of the autoscaler to use	string	"amd64"	no
autoscaler_s3_package	Configuration to retrieve autoscaler lambda package from s3 bucket	object({ bucket = string key = string object_version = optional(string) })	null	no
autoscaler_version	Version of the autoscaler to deploy	string	"latest"	no
autoscaling_max_create	The maximum number of instances the utility is allowed to create in a single run	number	1	no
autoscaling_max_terminate	The maximum number of instances the utility is allowed to terminate in a single run	number	1	no
autoscaling_timeout	Timeout (in seconds) for a single autoscaling run. The more instances you have, the higher this should be.	number	30	no
base_name	Base name for resources. If unset, it defaults to sp5ft-${var.worker_pool_id}.	string	null	no
configuration	User configuration. This allows you to decide how you want to pass your token and private key to the environment if you dont wish to use secrets manager and the secret strings functionality - be that directly, or using SSM Parameter Store, Vault etc. NOTE: One of var.configuration or var.secure_env_vars is required.	string	""	no
create_iam_role	Determines whether an IAM role is created or to use an existing IAM role	bool	true	no
custom_iam_role_name	Name of an existing IAM to use. Used when create_iam_role = false	string	""	no
disable_container_credentials	If true, the run container will not be able to access the instance profile credentials by talking to the EC2 metadata endpoint. This is done by setting the number of hops in IMDSv2 to 1. Since the Docker container goes through an extra NAT step, this still allows the launcher to talk to the endpoint, but prevents the container from doing so.	bool	true	no
domain_name	Top-level domain name to use for pulling the launcher binary	string	"spacelift.io"	no
ec2_instance_type	EC2 instance type for the workers. If an arm64-based AMI is used, this must be an arm64-based instance type.	string	"t3.micro"	no
enable_autoscaling	Determines whether to create the Lambda Autoscaler function and dependent resources or not	bool	true	no
enable_monitoring	Enables/disables detailed monitoring	bool	true	no
enabled_metrics	List of CloudWatch metrics enabled on the ASG	list(string)	[ "GroupDesiredCapacity", "GroupInServiceInstances", "GroupMaxSize", "GroupMinSize", "GroupPendingInstances", "GroupStandbyInstances", "GroupTerminatingInstances", "GroupTotalInstances" ]	no
iam_permissions_boundary	ARN of the policy that is used to set the permissions boundary for any IAM roles.	string	null	no
instance_market_options	The market (purchasing) option for the instance	any	{}	no
instance_refresh	If this block is configured, start an Instance Refresh when this Auto Scaling Group is updated based on instance refresh configration.	any	{}	no
launch_template_default_version	Default Version of the launch template	string	null	no
launch_template_update_default_version	Whether to update Default Version each update. Conflicts with default_version	bool	null	no
launch_template_version	Launch template version. Can be version number, $Latest, or $Default	string	null	no
max_size	Maximum number of workers to spin up	number	10	no
min_size	Minimum numbers of workers to spin up	number	0	no
poweroff_delay	Number of seconds to wait before powering the EC2 instance off after the Spacelift launcher stopped	number	15	no
schedule_expression	Autoscaler scheduling expression	string	"rate(1 minute)"	no
secure_env_vars	Secure env vars to be stored in Secrets Manager, their values will be exported at run time as export {key}={value}. This allows you pass the token, private key, or any values securely. NOTE: One of var.configuration or var.secure_env_vars is required.	map(string)	{}	no
secure_env_vars_kms_key_id	KMS key ID to use for encrypting the secure strings, default is the default KMS key	string	null	no
security_groups	List of security groups to use	list(string)	n/a	yes
selfhosted_configuration	Configuration for selfhosted launcher	object({ s3_uri = string # If provided, the launcher binary will be downloaded from that URI. Mandatory for selfhosted. Format: s3:///. For example: s3://spacelift-binaries-123ab/spacelift-launcher run_launcher_as_spacelift_user = optional(bool) # Whether to run the launcher process as the spacelift user with UID 1983, or to run as root. http_proxy_config = optional(string) # The value of the HTTP_PROXY environment variable to pass to the launcher, worker containers, and Docker daemon. https_proxy_config = optional(string) # The value of the HTTPS_PROXY environment variable to pass to the launcher, worker containers, and Docker daemon. no_proxy_config = optional(string) # The value of the NO_PROXY environment variable to pass to the launcher, worker containers, and Docker daemon. ca_certificates = optional(list(string)) # List of additional root CAs to install on the instance. Example: ["-----BEGIN CERTIFICATE-----abc123-----END CERTIFICATE-----"]. power_off_on_error = optional(bool) # Indicates whether the instance should poweroff when the launcher process exits. This allows the machine to be automatically be replaced by the ASG after error conditions. If an instance is crashing during startup, it can be useful to temporarily set this to false to allow you to connect to the instance and investigate. })	{ "ca_certificates": [], "http_proxy_config": "", "https_proxy_config": "", "no_proxy_config": "", "power_off_on_error": true, "run_launcher_as_spacelift_user": true, "s3_uri": "" }	no
spacelift_api_key_endpoint	Full URL of the Spacelift API endpoint to use, eg. https://demo.app.spacelift.io	string	null	no
spacelift_api_key_id	ID of the Spacelift API key to use	string	null	no
spacelift_api_key_secret	Secret corresponding to the Spacelift API key to use	string	null	no
tag_specifications	Tag specifications to set on the launch template, which will apply to the instances at launch	list(object({ resource_type = string tags = optional(map(string), {}) }))	[]	no
volume_encryption	Whether to encrypt the EBS volume	bool	false	no
volume_encryption_kms_key_id	KMS key ID to use for encrypting the EBS volume	string	null	no
volume_size	Size of instance EBS volume	number	40	no
vpc_subnets	List of VPC subnets to use	list(string)	n/a	yes
worker_pool_id	ID (ULID) of the the worker pool.	string	n/a	yes

Outputs

Name	Description
autoscaling_group_arn	ARN of the auto scaling group
autoscaling_group_name	Name of the auto scaling group
instances_role_arn	ARN of the IAM role of the EC2 instances. Will only be populated if the IAM role is created by this module
instances_role_name	Name of the IAM role of the EC2 instances. Will only be populated if the IAM role is created by this module
launch_template_id	ID of the launch template