Shared state with Terraform

Sharing Terraform state using S3 and DynamoDB.

Terraform relies on a state file to track how its internal identifiers and other metadata that are not directly reified in the cloud are mapped to the concrete instances and resources that are created to realise your configuration. This mapping is needed to work out the minimum set of actions Terraform needs to take to deploy updates.

By default the state file is stored locally, but this doesn't work when using Terraform in a team environment; everyone running Terraform commands needs to be using the same identifier mapping or things will get... messy.

The state file should not be checked in to version control, but Terraform does support multiple different backends for remote state. The most convenient for my purpose is S3.

To make this work we need a few components:

1. An S3 bucket with the correct permissions where we can store the state files.
2. A tiny DynamoDB table to enable locking of the files in S3.
3. Some updates to the Terraform configuration to use the remote state files.

Let's tackle them one at a time.

S3 bucket

There are a few different ways of creating and managing the S3 bucket itself, but it needs to exist before you run terraform init (below) to move the state file into it. In my case I'm dealing with a few different environments (production, staging, etc.) in different regions. I want to be able to completely destroy and rebuild the staging environment, so I don't want it to manage its own Terraform state bucket.

Instead, I'm going to create a single bucket in the production environment that will (hopefully!) never be destroyed and store all the different environment state files in there. Alternatively you could have a single TF file that just specified the state management resources used by all the others.

In the production TF file:

resource "aws_s3_bucket" "terraform-state-bucket" {
    bucket = "s3-tf-state"
    acl = "private"
}

Dynamo DB

Since the whole point of this exercise is to allow different team members to work on Terraform-managed infrastructure together, we need some way of locking the state file while someone's using it in order to avoid conflicts.

Terraform supports using a DynamoDB table for this. Like the S3 bucket above, I'm creating this in my production environment to ensure it's always available as other environments come and go.

resource "aws_dynamodb_table" "terraform-state-bucket-lock" {
    name = "tf-state-lock"
    billing_mode = "PAY_PER_REQUEST"
    hash_key = "LockID"
    attribute {
        name = "LockID"
        type = "S"
    }
    tags = {
        Name = "Terraform state lock"
    }
}

Configure Terraform to use the bucket

With the S3 bucket and the DynamoDB table in place we can configure Terraform to use them as a state storage backend:

terraform {
    backend "s3" {
        encrypt = true
        bucket = "s3-tf-state"
        region = "eu-west-2"
        key = tf/staging.tf
        dynamodb_table = "tf-state-lock"
    }
}

The only part of this which might not be obvious is key, which is the name to use for the state file.

With the configuration in place you'll need to run terraform init again to move stage management into the new backend. And that's it! Mutiple people or teams can now share Terraform configuration.