Aggregate data from multiple third-party locations using distributed processing with 3-tier architecture
This workflow demonstrates large-scale data aggregation from multiple third-party locations using AWS Step Functions' distributed processing capabilities with a 3-tier architecture.
The main workflow orchestrates the entire process by querying DynamoDB for client locations, then uses distributed map to process multiple locations in parallel. Each location is processed by a standard execution child workflow that handles data extraction and pagination.
A second express execution child workflow performs the actual API calls to third-party endpoints with query parameters and pagination support. Data is temporarily stored in S3 as JSON files organized by task ID.
Finally, an AWS Glue job consolidates all partial files into a single output file uploaded to the destination S3 bucket, with status updates tracked in DynamoDB.
< Back to all workflows
Clone repo
git clone https://github.com/aws-samples/step-functions-workflows-collection/tree/main/distributed-data-stream-aggregator/cd step-functions-workflows-collection/distributed-data-stream-aggregator
Deploy
Follow the step-by-step deployment instructions in the README.md to create DynamoDB tables, S3 buckets, Glue job, EventBridge connections, and deploy the state machines.
Testing
1. Populate the locations DynamoDB table with test data containing task_id and location information.
2. Execute the state machine using the AWS CLI with task_id and task_sort_key as input.
3. Monitor execution progress in the Step Functions console and verify data consolidation in S3.
Cleanup
1. Delete the state machines using AWS CLI:
aws stepfunctions delete-state-machine2. Delete DynamoDB tables:
aws dynamodb delete-table3. Delete S3 buckets:
aws s3 rb --force4. Delete Glue job and EventBridge connection
Additional resources
Created by:
