File copy between Amazon S3 buckets via Amazon Eventbridge Pipes & AWS Step Functions

Amazon S3 → Amazon SQS → EventBridge Pipes → Step Functions → Amazon S3

Copy and manipulate an object between Amazon S3 buckets using Amazon SQS, Amazon EventBridge Pipes and AWS Step Functions

This sample project demonstrates how to copy an object with JSON encoded content between two S3 buckets and manipulate it in transit.
When an object is created in the source S3 bucket, an S3 Event Notification is fires and enqueued onto an SQS Queue.
The EventBridge service polls the SQS Queue and invokes an EventBridge Pipe.
The EventBridge Pipe then executes an AWS Step Functions Express Workflow state machine.
This copies the file from the source S3 Bucket to the destination S3 Bucket.
In a real world scenario the state machine would be modified to manipulate the JSON object as desired

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/s3-sqs-eventbridge-pipe-sfn-s3

Deploy

sam deploy


Testing

See the GitHub repo for detailed testing instructions.

Cleanup

Delete the stack: sam delete.

Created by:

Dan Straw

Dan Straw

Senior Solutions Architect at AWS

Follow on LinkedIn