Amazon CloudWatch Metric Streams to Amazon Data Firehose

Amazon CloudWatch → Amazon Kinesis Firehose → Amazon S3

Create CloudWatch Metric stream using Amazon Data Firehose and save them in Amazon S3

This pattern sets up Amazon CloudWatch Metric stream and associates that with Amazon Data Firehose. Through this setup you can continuously stream metrics to a destination of choice with near-real-time delivery and low latency. There are various destinations supported, which include Amazon Simple Storage Service (S3) and several third party provider destinations like Datadog, NewRelic, Splunk and Sumo Logic, but in this pattern we use S3. This setup also provides capability to stream all CloudWatch metrics, or use filters to stream only specified metrics. Each of the metric streams can include up to 1000 filters that can either include or exclude namespaces or specific metrics. Another limitation for a single metric stream is it can either include or exclude the metrics, but not both. If any new metrics are added matching the filters in place, an existing metric stream will automatically include them.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/cloudwatch-metric-streams-firehose-terraform

Deploy

terraform initterraform planterraform apply


Testing

In the same account and region, launch an EC2 instance. You should be able to see metrics arrive n S3 bucket in few minutes.

Cleanup

terraform destroy

Created by:

Kiran Ramamurthy

I am a Senior Partner Solutions Architect for Enterprise Transformation. I work predominantly with partners and specialize in migrations and modernization.

Follow on LinkedIn