Using AWS CloudFormation Custom Resources to populate an S3 bucket

Like many people using AWS, I am drawn to AWS CloudFormation as a quick way to build infrastructure  and even full end-to-end application stacks in a programmable, repeatable way.

CloudFormation makes use of Template YAML or JSON files as a Declarative Language. That means that you define what AWS resources you want to build, and it is up to CloudFormation to determine how to build these resources, and in what order. 

In many cases, resources could be built in parallel, and that is the approach that CloudFormation typically uses. If there are references in one Resource block to another Resource block (such as an EC2 resource block defines a '!Ref' to a VPC Subnet block), then CloudFormation is able to identify these dependencies and commence the building of one resource block once the dependent block has completed. 

However, if you have used CloudFormation for any reasonable length of time, you have probably come across the "circular reference gotcha" ! 

For example, suppose you define a CloudFormation Resource such as a CloudFront Distribution, referencing an S3 Bucket. Then, you need to define an Access Origin Control Setting to enable access to this Bucket. And finally, you need to update the S3 Bucket resource with a Bucket Policy to enable access using this Access Origin Control Setting. In the first case, you have created an implicit dependency; the S3 Bucket must be created first in order to associate the CloudFront Distribution to it. But on the other hand, you need to update the S3 Bucket policy after the CloudFront Distribution has been created. 

Take another use-case: I want to create an S3 Bucket, and then subsequently upload data into that Bucket - maybe inline, or maybe by referencing a GitHub or CodeCommit Repo. Unfortunately, there is no mechanism inside CloudFormation to update contents to an S3 Bucket.

In both cases, one solution is to make use of AWS CloudFormation Lambda-backed Custom Resources. These are documented here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources-lambda.html 

They provide one mechanism for customising CloudFormation to address the "circular reference gotcha", and the "can't do it here" problem. 

Taking the second simple example - populating an S3 Bucket with a file - the first step is to create a Lambda Resource inside your CloudFormation Template. In this case, I have written some code which will create an 'index.html' file and upload it to the S3 Bucket:

A few things to note about this code:

• This example is written in Python. It uses the Python SDK ('boto3') to make API calls to the Amazon S3 service in order to upload a file which it firstly creates on the /tmp folder of the Lambda runtime environment. However, you could use other languages if you prefer.
• Make sure you return a cfnresponse to CloudFormation. This is easy to forget; in which case it results in your CloudFormation Stack hanging until it times out !
• The event['RequestType'] is used to identify whether CloudFormation is reading the template file in order to do a "Create", an "Update" on the Stack, or rolling back the Stack (a "Delete"). If you are creating a file on an S3 Bucket, don't forget to include code for the "Delete" action. 
• The event block also includes an event['ResourceProperties']. This is where you define which dependent resources you wish for this custom resource. In this case, we have referenced the name of the S3 Bucket in the CloudFormation template.

The code itself is very straightforward. It simply creates the file and uploads it to the bucket name which has been passed as a resource property.

Having created the Lambda function, there are just a couple of other things to add. Firstly, you need to create a Role for the Lambda function to make use of. This is referenced as MakeIndexLambdaExecutionRole in the example above. 

Secondly, you need to tie everything together by defining the Custom resource itself. This is a very straightforward piece of code, since all you need to do is give it a name, and reference to the Lambda function, and the resource properties you wish to pass it. 

Since this Custom Resource will run after the other resources that depend upon it, you can resolve any circular dependencies, by using a Custom Resource to update any existing resource once it has been created. 

Happy AWS Building !

Dennis (Dendad Trainer)

The AWS CloudFormation Documentation set is here: 

https://docs.aws.amazon.com/cloudformation/index.html

My GitHub Repository with example CloudFormation Templates: 

https://github.com/dendad-trainer/simple-aws-demos