Select your cookie preferences

We use cookies and similar tools to enhance your experience, provide our services, deliver relevant advertising, and make improvements. Approved third parties also use these tools to help us deliver advertising and provide certain site features.

CodeCommit to CodePipeline to Step Functions

Created with SnapCodeCommitCode PipelineStep Functions

Create a CodePipeline that invokes Steps Functions to run Glue queries

CDK CodePipeline spins up against AWS CodeCommit repository, which triggers AWS StepFunctions to call Amazon Athena NamedQueries to create Views and Tables.
The branch Name and CodeCommit repository details must be provided in the configuration file src/config.ts.

#!/usr/bin/env node
import 'source-map-support/register';

import { App, Stack } from 'aws-cdk-lib';
import { StepFunctionInvokeAction } from 'aws-cdk-lib/aws-codepipeline-actions';
import { PolicyStatement, Effect } from 'aws-cdk-lib/aws-iam';

import { DemoPipeline } from './src/stacks/stack_pipeline';
import { S3GlueStack } from './src/stacks/stack_s3buckets';
import { TestGlueStack } from './src/stacks/stack_glue';
import { AthenaStack } from './src/stacks/stack_athena';

const app = new App();

// make sure account and region provided here should match with one provided at file 'src/config.ts'.

const s3BucketsStack = new S3GlueStack(app, 's3GlueBucketsStack', {
    //env: { account: account, region: region }
});

const glueStack = new TestGlueStack(app, 'glueTablesStack', {
    //env: { account: account, region: region }
    s3EmpMaster: s3BucketsStack.s3EmpMaster,
});
glueStack.addDependency(s3BucketsStack);

const athenaStack = new AthenaStack(app, 'athenaStack', {
    gDB: glueStack.gDB,
    s3aQuery: s3BucketsStack.s3aQuery
})
athenaStack.addDependency(glueStack);

const demoPipeline = new DemoPipeline(app, 'demoPipeline', {
    //env: { account: account, region: region }
});

demoPipeline.addDependency(athenaStack);

const viewStep = new StepFunctionInvokeAction({
    actionName: 'UpdateViewsStage',
    stateMachine: athenaStack.sViewsMachine
});

demoPipeline.pipeline.pipeline.addStage({
    stageName: 'DeployViews',
    actions: [viewStep]
});

app.synth();

< Back to all patterns


GitHub icon Visit the GitHub repo for this pattern.

Download

git clone https://github.com/aws-samples/serverless-patterns/ cd serverless-patterns/cdk-codecommit-codepipeline-sfn-athena-glue

Deploy

npx cdk bootstrap aws://accountnumber/regionnpx cdk synthnpx cdk deploy


Testing

See the GitHub repo for detailed testing instructions.

Cleanup

1. Delete the stack: npx cdk destroy.
2. Confirm the stack has been deleted: aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus".
3. You see a message confirming DELETE_COMPLETE.

Atul Sharma

Presented by Atul Sharma

Atul is Sr. Cloud Application Architect at AWS Pro-serve India.

Feroz Kumar

Presented by Feroz Kumar

Feroz is Cloud Application Consultant at AWS Pro-serve India.

Jayesh A Shinde

Presented by Jayesh A Shinde

Jayesh is Cloud Application Architect at AWS Pro-serve India.