AWS CDK custom resources for DynamoDB inserts

In the past fifteen years, I found that numerous projects rely on some predefined database values. Think about lists of postal codes, countries ,etc. If you are into infrastructure as code like me, AWS CDK on Typescript offers a decent development experience. However sometimes you will end-up out of luck by missing Cloudformation (and therefore CDK) support. This article will show how you can leverage CDK to insert into DynamoDB on a CDK deploy. It will display two ways of writing, using AwsSdkCall for singular rows, or batch writes for multiple.

AwsCustomResource with AwsSdkCall workings

In this code example we will use AwsSdkCall together with the CDK construct AwsCustomResource. This is a very nifty construct that introduces a lambda runtime wrapper for your desired runtime environment. During packaging your desired AwsSdkCall will be parsed to Cloudformation yml. On execution of your Cloudformation stack the AwsSdkCall will be passed to the lambda as an event parameter, after which the Lambda runtime will execute your desired call(s).

As you can see AwsSdkCall is a really powerful concept that basically has the same power as your AWS CLI. This ultimately will give you everything you need to automate maybe half of your custom use-cases. You can even return values to your Cloudformation execution and use these values later on in your CDK.

Using singular Dynamo inserts AwsSdkCall

This option is valid if you need to insert one or some rows. Each call will create a CDK resource. If you need to do dozens of records, the batch-write option might be a more appropriate pick.

This code fragment displays a custom CDK construct that is able to insert a record based an AwsCustomResource using a AwsSdkCall.

export interface CdkCallCustomResourceConstructProps {
  tableName: string
  tableArn: string
}

export class CdkCallCustomResourceConstruct extends Construct {

  constructor(scope: Construct, id: string, props: CdkCallCustomResourceConstructProps) {
    super(scope, id);
    this.insertRecord(props.tableName,props.tableArn,{
      id: {S:'ID_1'},
      userName: {S:'Tim'}
    })
  }

  private insertRecord( tableName: string,tableArn: string, item: any) {
    const awsSdkCall: AwsSdkCall = {
      service: 'DynamoDB',
      action: 'putItem',
      physicalResourceId: PhysicalResourceId.of(tableName + '_insert'),
      parameters: {
        TableName: tableName,
        Item: item
      }
    }
    const customResource: AwsCustomResource = new AwsCustomResource(this, tableName+"_custom_resource", {
          onCreate: awsSdkCall,
          onUpdate: awsSdkCall,
          logRetention: RetentionDays.ONE_WEEK,
          policy: AwsCustomResourcePolicy.fromStatements([
            new PolicyStatement({
              sid: 'DynamoWriteAccess',
              effect: Effect.ALLOW,
              actions: ['dynamodb:PutItem'],
              resources: [tableArn],
            })
          ]),
          timeout: Duration.minutes(5)
        }
    );
  }
}

Important parts of this fragment worth noting are that you need to insert using the DynamoJSON compatible format: like “{id: {S:’ID_1'}}. This functionality is leveraging AwsSdkCall. In the AwsSdkCall block a service can be any service mentioned in the service list. Actions are declared per service like for DynamoDB the full list can be viewed here. The physicalResourceId is a unique ID in Cloudformation that represents this customResource’s insert. Parameters represent a map of keys and values passed to the service’s action.

Now let’s look at the part that initialises and executes the AwsSdkCall called AwsCustomResource. AwsCustomResource is a CDK construct that has three life cycle phases which you can leverage: onCreate, onUpdate, and onDelete. You can assign different calls to different life cycles phases, and should minimally declare one of them.

Using batch writes to insert multiple Dynamo records at once

If you need to declare dozens of rows, it might be slow and messy to create multiple entries like this. Dynamo’s batch-write API can be useful to insert bigger records sets.

export interface CdkCallCustomResourceConstructProps {
  tableName: string
  tableArn: string
}

interface RequestItem {
  [key: string]: any[]
}

interface DynamoInsert {
  RequestItems: RequestItem
}

export class BatchInsertCustomResourceConstruct extends Construct {

  constructor(scope: Construct, id: string, props: CdkCallCustomResourceConstructProps) {
    super(scope, id);
    this.insertMultipleRecord(props.tableName,props.tableArn,[{
      id: {S:'ID_2'},
      userName: {S:'Pete'}
    },
      {
        id: {S:'ID_3'},
        userName: {S:'Dina'}
      }])
  }

  private insertMultipleRecord( tableName: string,tableArn: string, items: any[]) {
    const records = this.constructBatchInsertObject(items, tableName);

    const awsSdkCall: AwsSdkCall = {
      service: 'DynamoDB',
      action: 'batchWriteItem',
      physicalResourceId: PhysicalResourceId.of(tableName + 'insert'),
      parameters: records
    }

    const customResource: AwsCustomResource = new AwsCustomResource(this, tableName+"_custom_resource", {
          onCreate: awsSdkCall,
          onUpdate: awsSdkCall,
          logRetention: RetentionDays.ONE_WEEK,
          policy: AwsCustomResourcePolicy.fromStatements([
            new PolicyStatement({
              sid: 'DynamoWriteAccess',
              effect: Effect.ALLOW,
              actions: ['dynamodb:BatchWriteItem'],
              resources: [tableArn],
            })
          ]),
          timeout: Duration.minutes(5)
        }
    );
  }

  private constructBatchInsertObject(items: any[], tableName: string) {
    const itemsAsDynamoPutRequest: any[] = [];
    items.forEach(item => itemsAsDynamoPutRequest.push({
      PutRequest: {
        Item: item
      }
    }));
    const records: DynamoInsert =
        {
          RequestItems: {}
        };
    records.RequestItems[tableName] = itemsAsDynamoPutRequest;
    return records;
  }
}

In this snippet the action is changed to batchWriteItem. It now supports taking multiple rows at once. However it might make sense to limit your writes to batches of writes to let your DynamoDB keep up. This approach allows for inserting data like lists of countries or postal codes in your stack.

However there is a downside to keep in mind. All the data you will put into this batch write will be made part of your Cloudformation template. Currently this has a 10Mb limit to its template.

Conclusions

CDK custom resources offer an escape hatch for whenever a service is to new, or its api is just poorly supported. This opens up basically every AWS API to your CDK project!

Examples on Github

As good examples can be hard to come by I made a repo just for this example. It shows the single and multiple write example. You can deploy it using the command showed in the readme. Visit my Github repo.