Orchestrate continuous running Glue Workflow

This workflow demonstrates how to orchestrate a continuous running Glue Workflow.

1. Triggering Glue Workflow: Initially, the Step Function triggers a Glue workflow to begin its execution
2. Polling for Completion: After triggering the Glue workflow, the Step Function enters a polling loop. It periodically checks the status of the Glue workflow, every 60 seconds.
3. Completion Check: Once the Glue workflow completes, the Step Function checks if there are any failed actions within the workflow.
4. Decision Making: If there are no failed actions detected, the Step Function initiates a new execution of itself, thereby starting the process all over again. This effectively creates a continuous loop of triggering Glue workflows
5. Decision Making: If there are failed actions within the Glue workflow, the Step Function stops its execution. This ensures that the process halts when errors or failures occur during the Glue workflow execution

< Back to all workflows

GitHub icon View this workflow on GitHub


Clone repo

git clone https://github.com/aws-samples/step-functions-workflows-collection/tree/main/sfn-continous-running-glue-workflow-tf/cd step-functions-workflows-collection/

Deploy

terraform initterraform apply


Testing

See the GitHub repo for detailed testing instructions.

Cleanup

1. Delete the stack: terraform destroy.

Created by:

Swapnil Kesarkar

Swapnil Kesarkar is a Mission Critical Engineer at Schuberg Philis, based out of Netherlands

Follow on LinkedIn