Amazon EventBridge to Amazon OpenSearch Service via Amazon Data Firehose

Amazon EventBridge Classic bus → Amazon Data Firehose → AWS Lambda → Amazon S3 → Amazon OpenSearch Service

Stream every event on an EventBridge bus to OpenSearch through Amazon Data Firehose for near real-time search, with payload flattening, daily index rotation and an S3 backup.

This pattern gives you full-text search over every event on an Amazon EventBridge bus, usually within about 60 seconds of the event being emitted. CloudWatch metrics tell you how many events flowed; this tells you what was in them.
A catch-all EventBridge rule matches every event on a custom bus and sends it to an Amazon Data Firehose delivery stream. The rule pattern matches on a source prefix of an empty string, because every EventBridge event carries a source and an entirely empty pattern is rejected.
Firehose buffers for 60 seconds or 1 MB, whichever comes first. That buffer is the floor Firehose allows and it sets the end-to-end latency of the pattern.
Before indexing, a Lambda transform flattens the EventBridge envelope: detail-type is renamed to detail_type so it needs no escaping in queries, and the fields inside detail are promoted to the top level so a dashboard can filter on claimId rather than detail.claimId. Envelope fields win on collision, so a business payload carrying its own source key is indexed as detail_source instead of masking the real event source.
Documents land in a daily-rotated index, events-YYYY-MM-DD, and every document is also written to an S3 bucket, which makes that bucket an audit trail rather than only a dead letter destination.
Firehose authenticates to OpenSearch with SigV4 using its delivery role. A managed domain authorizes each request against its own access policy, so the role is granted access on both sides: an identity policy on the role and a domain access policy naming that role as principal.
EventBridge delivery is at-least-once, so an event can occasionally be indexed twice; each document keeps the EventBridge event ID in its id field for deduplication at query time. The pattern adds no dead-letter queue on the rule target and no alarms. The README lists what to add for production.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/eventbridge-firehose-opensearch/typescript/cdk

Deploy

cd cdknpm installcdk deploy -c dashboardAccessIp=YOUR_IP/32


Testing

See the GitHub repo README.md for detailed testing instructions.
Read the bus name from the stack outputs: BUS=$(aws cloudformation describe-stacks --stack-name EventMonitorStack --query "Stacks[0].Outputs[?OutputKey=='EventBusName'].OutputValue" --output text)
Emit a test event onto the bus: aws events put-events --entries '[{"Source":"demo.test","DetailType":"TestEvent","Detail":"{\"message\":\"Hello OpenSearch\",\"claimId\":\"CLM-001\"}","EventBusName":"'"$BUS"'"}]'
Wait 60 to 90 seconds for the Firehose buffer to flush, then open the DashboardsUrl from the stack outputs and create an index pattern of events-* with time as the time field.

Cleanup

cd cdk
cdk destroy

Created by:

Antoine Boucherie

Antoine Boucherie

Principal Solutions Architect, AWS Global Financial Services