AWS Step Functions to AWS Lambda MicroVMs with a task token callback

Client → AWS Step Functions → AWS Lambda MicroVMs → Amazon DynamoDB

Run a job on an AWS Lambda MicroVM from AWS Step Functions, get its result with a task token callback, and always terminate the MicroVM.

AWS Lambda MicroVMs run long or stateful work, for up to eight hours, in a Firecracker-isolated environment built from your own image. A MicroVM isn't invoked and doesn't return a response: it runs, and bills, until it's terminated or reaches its maximum duration. This pattern uses an AWS Step Functions workflow to get the job's result back and to make sure the MicroVM always stops, with no AWS Lambda function in between.
The workflow starts the MicroVM with RunMicrovm through the AWS SDK integration and passes the execution input as the job in the run hook payload, with the execution ARN as the idempotency token. The next step stores its task token in an Amazon DynamoDB table under the MicroVM ID and waits with .waitForTaskToken. The worker reads the token, sends SendTaskHeartbeat every 20 seconds, and finishes with SendTaskSuccess or SendTaskFailure.
The workflow calls TerminateMicrovm on every path: success, a reported failure, missed heartbeats or the maximum duration. The maximum duration is also set on the MicroVM, so the platform stops it if someone stops the execution by hand. Replace the example job, which counts prime numbers, with your own batch job, data processing step or agent run.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/lambda-microvms-sfn-callback/python/sam

Deploy

1. Set your Region and a name for a new Amazon S3 bucket that holds the worker code: export AWS_REGION=us-east-1 and export ARTIFACT_BUCKET=sfn-microvms-callback-artifacts-$(aws sts get-caller-identity --query Account --output text)-$AWS_REGION2. Create the bucket, then zip and upload the worker: aws s3 mb s3://$ARTIFACT_BUCKET --region $AWS_REGION, (cd src/worker && zip -j ../../worker.zip Dockerfile app.py), then aws s3 cp worker.zip s3://$ARTIFACT_BUCKET/worker.zip --region $AWS_REGION3. Deploy the stack and enter the artifact bucket name for ArtifactBucket when prompted: sam deploy --guided


Testing

Start executions with a successful job, a failing job and a hung job, then confirm each MicroVM is terminated. See the README for detailed instructions.

Cleanup

1. Delete the stack: sam delete --stack-name STACK_NAME --region $AWS_REGION. Enter y at each confirmation prompt.
2. Delete the artifact bucket and the local zip: aws s3 rb s3://$ARTIFACT_BUCKET --force --region $AWS_REGION, then rm worker.zip.
3. Confirm the stack has been deleted: aws cloudformation list-stacks --region $AWS_REGION --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus"

Created by:

Manish S

Manish S

AWS Support Engineer, trying to build things