Migrate to Terraform Cloud: part 2

For benefits of TFC and my assessment, see part 1

Migration walkthrough

Prerequisites

• no state drifts: running terraform plan prior to migrating to TFC should give you a "no changes" output.
• have a valid account and organization on TFC: this walkthrough won’t cover the TFC account setup.
• configure TFC to access your GitHub repos / connected to your GitHub.
• agents/agent pools are already set: you can use terraform to provision the agent pool, but I won’t cover that part in this post.
• Generate a token (one-time setup): this can be done by running terraform login and the token is expected to be stored at ~/.terraform.d/credentials.tfrc.json.

Components

TFC provisioning: workspaces / projects

Prior to migrating to TFC, you can use the same workspace names for different terraform projects(tfstates). For example, you may have a staging workspace in tf_project0 and a staging workspace in tf_project1.

On TFC: as of this is written (08/2023) TFC workspace name is unique per organization.

This means when you migrate multiple terraform projects to TFC, you need to define a TFC workspace naming convention to prevent collision. You can optionally create projects to group / organize TFC workspaces.

My recommendation is to add your terraform project’s name as a prefix to the original terraform workspace name to construct the TFC workspace name. E.g. the staging workspace for your tf_project0 should be named as tf_project0-staging on TFC.

Although this step can be done by terraform CLI (will be covered later in switching backend), I recommend doing this using tfe provider and provision the TFC projects and workspaces using terraform.

Workspace configuration

Important configurations for TFC workspaces:

• tags. tags are what's used for terraform to filter/group workspaces. E.g.
• on a s3 backend, if you specify workspace_key_prefix to be "my_team/my_project/", you'll find all your workspaces (staging, dev, etc) listed under this directory in s3
• on TFC, all workspaces that are tagged with ["my_team", "my_project"] are considered as the workspace for your project. You can query to list all of them via terraform workspace list
• execution mode:
• remote: runs on instances managed by TFC. I didn't go with this approach and I guess you'll have to make some networking level whitelisting changes to grant external access.
• local: still runs on your machine.
• agent: runs on a pool of agents provisioned by you. This is what I find to be the best option but will save the agent provisioing piece for another post.
• apply method: use “auto apply” for non-production environment and “manual apply” for production environment (enforce human review)
• remote state sharing: will cover a bit later.

unpublished private modules

If your terraform uses unpublished private modules, you need to publish these modules to TFC.

TFC can pull GitHub repo based terraform modules as long as the repo name follows the pattern: terraform-<provider>-<module>. terraform module repos are also expected to have release(s) using semver versions.

For example, tf_project0/modules/my_module0 need to be moved to a standalone GitHub repo terraform-my_provider-my_module0 and a release v1.0.0 is published.

After modules are published, change your module reference:

from:

module "my_module0" {
  source  = "../modules/my_module0"
  ...
}

to:

module "my_module0" {
  source  = "app.terraform.io/my_org/my_module0/my_provider"
  version = "1.0.0"
  ...
}

terraform_remote_state

TFC has an equivalent data source for terraform_remote_state: tfe_outputs

This is assuming the remote tfstate has been migrated to TFC. If the remote tfstate is still on a remote backend (e.g. s3), you need to provide access between TFC agent (or equivalent, depending on your execution mode) and your remote backend.

In addition (if the other state is already on TFC), per-workspace configuration needs to be updated: “Remote state sharing" need to include the TFC workspace for the other tfstate.

Using an example:

my_project0 depends on terraform_remote_state from my_project1:

data "terraform_remote_state" "my_project1" {
  backend = "s3"
  config = {
    bucket = "my-terraform-s3"
    key    = "my_org/my_project1/terraform.tfstate"
  }
}

after my_project1 is migrated to TFC, change the above (for my_project0) to:

data "tfe_outputs" "my_project1" {
  organization = "my_org"
  # "staging": my_project1-staging -> my_project0-staging
  workspace    = ["my_team","my_project1","staging"]
}

In addition, remote state sharing needs to be updated. This is another reason why I recommend using tfe to provision TFC workspaces.

tfe_output requires authentication, which is taken care of in the tfe provider specification. will cover a bit later.

workspace rename

Due to TFC limitation on workspace names, you may have to change your workspace name.

This will affect any references of ${terraform.workspace} and need to be updated to avoid unnecessary destroy + recreate operations.

TFC authentication

Both tfe provider and TFC cloud backend requires token authentication. Options are environment variables, variables and token from disk:

• environment variable: TFE_TOKEN for initializing tfe provider and TF_TOKEN_app_terraform_io (or TF_TOKEN_<tfc_host> initializing cloud backend.
• variable: token is accepted for both tfe provider and cloud backend.
• token from disk: ~/.terraform.d/credentials.tfrc.json from your local, or agent token for agents.

My recommendation is to use the 3rd option. This approach can avoid creating duplicated copies of the token per TFC workspace, making token rotation much easier to operate.

point backend to cloud

run terraform plan and make sure "no changes" is in the output: your local .terraform contains a copy of the latest tfstate

Change backend from:

backend "s3" {
  bucket = "my-terraform-s3"
  workspace_key_prefix = "my_org/my_project0/"
  key = "terraform.tfstate"
}

to:

cloud {
  organization = "my_org"
  workspaces {
    tags = ["my_team", "my_project0", "dev"]
  }
}

run terraform init: you'll be asked if you want to migrate, choose yes, and another question for a prefix for your workspace. Use the same TFC naming convention and use <project>-* as the prefix here so your TFC workspace will look like <project>-<original_workspace_name>

Validation

• check the “states” of your TFC workspace: you should see the same tfstate files as your previous remote backend.
• run terraform plan from CLI: it will trigger a new run on your TFC. If all things are taken care of, you should see a passed plan.

Conclusion

If you find this to be helpful, give it a clap and it would mean the world to me. Please share this with whoever needs this, and I’d appreciate it if you want to buy me a coffee