Automate ingestion of new documents into an Amazon S3 Vectors knowledge base using AWS Step Functions Distributed Map with validation
When new documents land in an S3 bucket, a Step Functions workflow fans out via Distributed Map to process each document in parallel.
For each document, a Lambda function reads the content, generates vector embeddings using Amazon Bedrock, and stores them with PutVectors in the S3 Vectors vector bucket.
After ingestion completes, a validation step uses QueryVectors to confirm the new content is searchable. A Choice state either confirms success or rolls back by deleting the newly added vectors if validation fails.
< Back to all patterns
Clone repo
git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/sfn-s3vectors-rag-refresh-cdkDeploy
npm installcd lambda && npm install && cd ..cdk deployTesting
See the GitHub repo for detailed testing instructions.
Cleanup
Delete the stack:
cdk destroy.Created by:
