Skip to content

Latest commit

 

History

History
147 lines (102 loc) · 6.23 KB

README.md

File metadata and controls

147 lines (102 loc) · 6.23 KB
Raw

Ecstatic

An AWS Lambda-ready script for updating and monitoring ECS agents.

How It Works

The script does the following when the lambda handler is called or it's run manually from the command-line:

  1. Loops through the ECS clusters in your account
  2. Loops through the container instances in the current cluster
  3. If an ECS agent is in a disconnect state, or if a previous attempt to update the agent failed:
    • A warning is logged and the cluster is not updated
  4. If all ECS agents in a cluster are healthy:
  5. If an agent is out-of-date and the request to UpdateContainerAgent is accepted:
    • Further updates in the current cluster are delayed until later runs
    • This is to affect an incremental roll-out of agent updates across a cluster

Also, if an agent update request is accepted or if an agents is in an error state, a Slack-style message will be sent to ECSTATIC_WEBHOOK_URL, if set.

Installing with Terraform

Add and edit the following module block to your AWS provider enabled Terraform configs:

module "ecstatic" {
  source             = "git@github.com:firstlookmedia/ecstatic//terraform?ref=v0.0.1"
  subnet_ids         = [ "${aws_subnet.vpc_subnet_1.id}", "${aws_subnet.vpc_subnet_2.id}" ]
  security_group_ids = [ "${aws_security_group.vpc_allow_all.id}" ]
  webhook_url        = "https://hooks.slack.com/services/JGKDLKTJDKG/FJSKFJGJKSKG/GJDKSKGJ"
}

The lambda is configured to run in a VPC, therefore, you'll need to edit the following variables in the module block:

  1. subnet_ids – at least one VPC subnet must be specified
  2. security_group_ids – at least one security group specified must allow network access to the target ECS clusters
  3. webhook_url – (optional) should be set to enable – or removed to disable – webhook messages

Once the module block is added and edited, run:

terraform init --upgrade
terraform apply --target=module.ecstatic

This will create the following resources in your AWS account:

  1. aws_lambda_function named ecstatic
  2. aws_iam_role named ecstatic_lambda
  3. aws_iam_role_policy named ecstatic_lambda
  4. aws_iam_role_policy_attachment
    • Attaches the lambda IAM role to service-role/AWSLambdaVPCAccessExecutionRole
  5. aws_cloudwatch_event_rule named ecstatic_update
  6. aws_cloudwatch_event_target named ecstatic_update
  7. aws_lambda_permission named ecstatic_update

The initial Terraform apply will pull the most recent released version of ecstatic from S3:

The schedule expression for the CloudWatch rule that is created to trigger updates defaults to rate( 1 hour ). If you would like to trigger this less frequently, you can override the value by setting update_schedule_expression in the module block, e.g.

module "ecstatic" {
  source = "git@github.com:firstlookmedia/ecstatic//terraform?ref=v0.0.1"
  ...
  update_schedule_expression = "rate( 4 hours )"
}

Supported Terraform module variables are defined in terraform/variables.tf.

Running from the Command-Line

Ecstatic can also be run locally and manually from the command-line. The steps to do this are:

1. Install Python and create a virtual environment

We use Homebrew, PyEnv, and pyenv-virtualenv. The latter two tools are great for managing multiple installed versions of Python and needed modules.

$ brew install pyenv pyenv-virtualenv
$ pyenv install 3.7.1
$ pyenv virtualenv 3.7.1 ecstatic

2. Get the code, activate the virtual environment, and install requirements

$ git clone git@github.com:firstlookmedia/ecstatic.git
$ cd ./ecstatic
$ pyenv local ecstatic
$ pip install -r requirements.txt

3. Run the script

$ AWS_PROFILE=ecs-admin ./ecstatic.py

You can learn how to configure your AWS credentials to work with Boto3 here: Credentials.

aws-profile-gpg

We recommend using aws-profile-gpg, a tool that generates role-specific IAM access tokens while safely storing your secret access keys in a GPG encrypted file.

# run ecstatic using aws-profile-gpg
$ AWS_PROFILE=ecs-admin aws-profile-gpg ./ecstatic.py

Environment Variables

The following environment variables are used by ecstatic:

Future Enhancements

We have a few ideas for enhancements, including

  • Checking for available Docker updates
  • Comparing the actual and target AMI versions of container instances in Auto Scaling Groups
  • Including or skipping container instances based on a Tags

If you have ideas or feedback, feel free to send feedback via GitHub Issues.

Resources

Further Reading