Amazon Data Firehose Data Transformation with AWS Lambda

Amazon Kinesis Firehose → AWS Lambda → Amazon S3

Transform incoming source data and deliver the transformed data to destinations.

The purpose of this pattern is to deploy the infrastructure necessary to enable Amazon Data Firehose data transformation.
Kinesis Data Firehose can invoke a AWS Lambda function to transform incoming source data and deliver the transformed data to destinations. In this architecture, Kinesis Data Firehose invokes the specified Lambda function asynchronously with each buffered batch using the AWS Lambda synchronous invocation mode. The transformed data is sent from Lambda to Kinesis Data Firehose. Kinesis Data Firehose then sends it to the destination Amazon S3 bucket when the specified destination buffering size or buffering interval is reached, whichever happens first.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/firehose-transformation-terraform

Deploy

terraform initterraform planterraform apply


Testing

See the README in the GitHub repo for detailed testing instructions.

Cleanup

terraform destroy
terraform show

Created by:

Naresh Rajaram

Naresh Rajaram

Cloud Infrastructure Architect, AWS

Follow on LinkedIn