Amazon Data Firehose data transformation with AWS Lambda

Amazon Kinesis Firehose → AWS Lambda → Amazon S3

Transform incoming source data and deliver the transformed data to S3.

This pattern demonstrates how to transform streaming data received by Amazon Data Firehose using AWS Lambda before delivering the transformed data to Amazon S3.
To transform incoming source data, Kinesis Data Firehose invokes the specified Lambda function asynchronously with each buffered batch using the AWS Lambda synchronous invocation mode. The transformed data is sent from Lambda to Kinesis Data Firehose. Kinesis Data Firehose then sends it to the destination S3 bucket when the specified destination buffering size or buffering interval is reached, whichever happens first.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/firehose-transformation-cdk-typescript

Deploy

cdk deploy


Testing

See the GitHub repo for detailed testing instructions.

Cleanup

cdk delete

Created by:

Edward Schaefer

Edward Schaefer

Solutions Architect @ Amazon Web Services

Follow on LinkedIn