Amazon Data Firehose Data transformation with AWS Lambda

Amazon Data Firehose → AWS Lambda → Amazon Data Firehose

Transform incoming source data and deliver the transformed data to destinations.

The purpose of this pattern is to deploy the infrastructure necessary to enable Amazon Data Firehose data transformation.
Amazon Data Firehose can invoke a Lambda function to transform incoming source data and deliver the transformed data to destinations. In this architecture, Amazon Data Firehose then invokes the specified Lambda function asynchronously with each buffered batch using the AWS Lambda synchronous invocation mode. The transformed data is sent from Lambda to Amazon Data Firehose. Amazon Data Firehose then sends it to the destination S3 bucket when the specified destination buffering size or buffering interval is reached, whichever happens first.

< Back to all patterns

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/firehose-transformation-sam-java

Deploy

sam deploy --guided


Testing

See the GitHub repo for detailed testing instructions.

Cleanup

Delete objects in S3: aws s3 rm s3://BUCKET_NAME --recursive
Delete the stack: sam delete.

Created by: