Exponential Backoff in Amazon SQS Processing with AWS Step Functions and Amazon EventBridge Pipes

Amazon SQS → Pipes → AWS Step Functions → AWS Lambda

This pattern demonstrates a seamless integration between an SQS queue and AWS Step Functions using Amazon EventBridge Pipes.

This pattern enables direct routing of messages from an SQS queue into a Step Functions state machine, which orchestrates the processing flow and manages AWS Lambda invocation errors. When Lambda encounters an error, the erroneous message is directed to an SQS Dead Letter Queue (DLQ) for further inspection or reprocessing.
The state machine manages message processing and invokes a Lambda function, which is intentionally set with a reserved concurrency of 0 to simulate throttling.
When errors, such as throttling, occur in the Lambda function, the state machine employs an exponential backoff retry strategy that starts with a 5-second delay, doubling this delay after each subsequent retry, for up to 6 maximum retry attempts. If all retry attempts are exhausted, the message is directed to a Dead Letter Queue (DLQ) for further analysis.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/sqs-stepfunctions-pipes-pattern

Deploy

sam deploy --guided


Testing

Retrieve the Queue URL from the outputs of the SAM deployment and send a test message using AWS CLI. Monitor the Step Functions state machine to observe the execution. If the Lambda function fails, check the DLQ for the erroneous message.

Cleanup

To delete the deployed resources, use: sam delete

Created by:

Mohammed Atiq

Mohammed Atiq

Solutions Architect at AWS and Serverless enthusiast.

Follow on LinkedIn