Read and Transform S3 objects using a distributed map state
This distributed map state reads all objects in an S3 bucket.
Each Batch of files is treated by a NESTED Distributed map state, the nested distributed map breaks the workload into smaller batches.
For processing each small batch will is treated by an INLINE map to get and publish object body as results.
< Back to all workflows
Clone repo
git clone https://github.com/aws-samples/step-functions-workflows-collection/tree/main/s3-bucket-nested-distributed-map/cd step-functions-workflows-collection/s3-bucket-nested-distributed-map
Deploy
1. Bootstrap CDK, if needed: <code>cdk bootstrap aws://{your-aws-account-number}/{your-aws-region}</code>2. Deploy the stack: <code>npm run cdk:deploy</code>
Testing
See the GitHub repo for detailed testing instructions.
Cleanup
1. Delete the stack:
npm run cdk:destroy.