Bulk object tagging with Amazon S3 Batch Operations and AWS Step Functions

AWS Step Functions Job workflow → Amazon SNS Job result → Amazon S3 Batch Operations → Amazon S3 Source objects

AWS Step Functions creates an Amazon S3 Batch Operations job with a generated manifest, polls it to the end, and publishes the result to Amazon SNS.

An AWS Step Functions workflow calls Amazon S3 Control CreateJob through the AWS SDK integration. Amazon S3 generates the object list from a key prefix, so there is no manifest file to build, and the job tags every matching object.
The workflow waits, calls DescribeJob, and loops until the job reaches a final status. The loop is bounded and the state machine has a timeout.
A Choice state branches on the failed-task count. A job with zero failed tasks publishes a success message to Amazon SNS. Any other end publishes a message with the job ID and counts, and the execution fails.
The job role can tag only objects under one prefix, so a run with another prefix exercises the failure branch.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/s3-sfn-batch-operations/typescript/cdk

Deploy

npm installnpm run buildIf the account and Region are not bootstrapped for the AWS CDK yet: npx cdk bootstrapnpx cdk deploy


Testing

See the GitHub repo for detailed testing instructions.

Cleanup

npx cdk destroy

Created by:

Nithin Chandran R

Nithin Chandran R

Technical Account Manager at AWS