How to set Terraform backend configuration dynamically

Terraform backend configuration can be a somewhat confusing topic, especially for the uninitiated. In this post, I will run through how we can set backend configuration and do it dynamically without using external tools like Terragrunt.

There are myriad choice of storage options for a Terraform backend, the two most common being local and remote, wherein state is stored locally in a .terraform directory and remotely in TFC/E. Other standard options such as; s3, gcs, azurerm are predominately used by their respective cloud in Terraform Open-source.

Today we do not overly care what the backend is other than it must not be local.

A basic backend configuration looks like this:

terraform {
    backend = "remote" {
        hostname = "app.terraform.io"
        organization = "ministry-of-magic"

        workspaces {
            name = "sorting-hat-api-prod"
        }
    }
}

Using the above in a file entitled backend.tf is perfectly fine when your code is only responsible for a single environment or project. However, suppose your code will create multiple environments or projects, and input variables drive these. In that case, you will want your backend configuration to be just as dynamic.

Terraform gives us two options out of the box for dynamic state setting, which we will go through here.

CLI-arguments#

When your CI/CD tool -or person- executes the Terraform code as part of the init phase, we would pass in a few additional arguments to instruct Terraform which backend to use.

terraform init \
-backend-config="hostname=app.terraform.io" \
-backend-config="organization=ministry-of-magic" \
-backend-config="workspace=sorting-hat-api-prod"

It is important to note, however, that even in this scenario, you must still declare an empty backend configuration block, such as:

terraform {
    backend "remote" {}
}

Configuration file#

As with the above, you must still configure an empty backend configuration block. In this instance however, you can utilise a backend.hcl file and pass that into the Terraform init command.

hostname     = "app.terraform.io"
organization = "ministry-of-magic"
workspaces { Name = "sorting-hat-api-prod" }

The following command consumes the above configuration:

terraform init -backend-config=backend.hcl

You could go even more dynamic with the configuration file by manipulating the file as part of your pipeline or passing in different files. A callout here is that you could also do with your backend.tf file; however, in the scenario where you are using the same code for multiple environments or projects, it is better to have an empty backend config pass in the configuration.

In summary, there are a couple of options available to you for setting backend configuration dynamically without having to explore external tools.

If you would like to learn a little more about available backend options, have a read of the Backend section of the Terraform documentation.