Amazon S3 to Amazon OpenSearch Serverless with OpenSearch Ingestion

Amazon S3 Source bucket → Amazon SQS Queue → Amazon OpenSearch Ingestion → Amazon OpenSearch Serverless

Ingest JSON objects from Amazon S3 into Amazon OpenSearch Serverless using an Amazon OpenSearch Ingestion (OSIS) pipeline.

This pattern indexes JSON Lines files from Amazon S3 into an Amazon OpenSearch Serverless collection with an Amazon OpenSearch Ingestion pipeline. The source bucket sends s3:ObjectCreated notifications to an Amazon SQS queue. The pipeline polls the queue, reads each new object, parses every line into a document with parse_json, and writes it to the s3-logs index. Documents that fail to index go to a separate dead-letter S3 bucket, so they don't trigger the pipeline again. The template needs no application code.
Cost: the pipeline bills for at least 1 Ingestion OCU per hour while it runs, so stop it with aws osis stop-pipeline when you're not using it. The collection is in a collection group with a minimum of 0 OCUs, so it scales to zero and stops billing for compute after 10 minutes with no requests. The network policy allows public access for the demo; use a VPC endpoint for production.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/s3-opensearch-serverless-osis/sam

Deploy

sam deploy --capabilities CAPABILITY_NAMED_IAM --guided


Testing

See the GitHub repo for detailed testing instructions.

Cleanup

Empty both S3 buckets, then delete the stack: sam delete.

Created by:

Sudhansu Mohapatra

Sudhansu Mohapatra

Delivery Consultant at AWS