DynamoDB Step Worker

Example project for AWS Step Functions and State Machines to process data objects stored in a DynamoDB table with parallel Scans using Segments.

The CloudFormation Stack creates two State Machines: Puppeteer and Worker. You can configure the Puppeteer to fan out the desired number of Worker executions to process the DynamoDB objects with the following configuration:

{
  "Comment": "Run State Machine Workers",
  "Configuration": {
    "TableName": "step-worker-url-stable",
    "TableHash": "url",
    "Workers": 4,
    "BatchSize": 250
  }
}

Usage

Dependencies

# Install Go dependencies

$ > make install

Deployment

# Compile Lambda w/ Go and deploy CloudFormation Stack

$ > make build package deploy

Execute State Machine

# Get ARN of State Machine 

$ > make outputs

[
  {
    "OutputKey": "Puppeteer",
    "OutputValue": "arn:aws:states:eu-west-1:1234567890:stateMachine:step-worker-puppeteer-stable",
    "Description": "ARN for State Machine"
  }
]

# Execute State Machine and configure BatchSize

$ > aws stepfunctions start-execution \
    --state-machine-arn arn:aws:states:eu-west-1:1234567890:stateMachine:step-worker-puppeteer-stable \
    --input '
    {
      "Comment": "Run State Machine with BatchSize 25",
      "Configuration": {
        "TableName": "step-worker-url-stable",
        "TableHash": "url",
        "Workers": 4,
        "BatchSize": 250
      }
    }'

License

Feel free to use the code, it's released using the MIT license.

Contribution

You are welcome to contribute to this project! 😘

To make sure you have a pleasant experience, please read the code of conduct. It outlines core values and beliefs and will make working together a happier experience.