How to Migrate Data to DynamoDB?

Once you have the infrastructure ready and your application coded, if you are performing a migration, it's time to start converting and migrating your data. The following section will provide you with a set of tips to help you build a migration plan.

AWS Database Migration Service (AWS DMS)

The recommended option for migrating your data is to use AWS Database Migration Service (DMS). This service is designed to facilitate database migrations and can be a valuable tool for your migration process.

However, if you are using the single table design approach, AWS DMS might not be the best fit, as it creates separate tables for each table in the source database. If you are migrating from MongoDB and still want to explore using AWS DMS, I have included a detailed blog post in the resources that explains how to perform a live migration from a MongoDB cluster to Amazon DynamoDB.

Custom Script

Due to the unique structure of your application table using the single table design, you will need to create your own customizable script for migration.

Your script will consist of two parts: retrieving the data from the original database and saving the items into the DynamoDB table.

Retrieving the data from the source

Most databases allow you to export data in formats such as JSON or CSV. If that is the case, you can store those exported items in Amazon S3 and then upload them to DynamoDB.

For databases with custom export formats, you may need to build a script that performs the data extraction for you.

The result of this step is having one file per entity in Amazon S3, and potentially some additional files for relationships if you are migrating from a relational database.

Importing the data to DynamoDB

You have a couple of options for importing the data:

1. Create a table file in Amazon S3 and import it into DynamoDB.
2. Process the items one by one and upload them to DynamoDB.

DynamoDB provides a feature that allows you to import data from Amazon S3 into a table, handling capacity management and data import for you. With this feature, you can import a file stored in Amazon S3, formatted like the DynamoDB table, into DynamoDB. However, note that this feature requires creating a new table; you cannot import data into an existing table.

To build the table file for direct import into DynamoDB or to iterate over the dump files from the database, you will need to create a process to handle that. You can leverage AWS Step Functions distributed map, which is a state type in Step Functions that iterates over items in a JSON or CSV file and executes a set of states. The maximum parallelization with Distributed map is 10,000.