Bootstrap Terraform projects for GCP and Azure

Use object storage services to store the tfstate

Terraform uses a file called the tfstate to store the state of the infrastructure. That file may contain sensitive informations like for instance user credentials for connecting to GCP or Azure managed SQL databases. This is one of the reasons why a Git repository for instance is not suitable to store the tfstate file.

Also you may be working on the Terraform code with other people. For that to be possible, Terraform has to make sure that only one change at a time can be made on the infrastructure. To achieve the preceding, Terraform maintains a .lock file inside the storage backend whenever a change is being applied to the infrastructure. Once the changes are applied, the .lock file is removed and other changes can again be applied by users. The .lock file contains infos about the user making the change and other metadata.

Azure Storage accounts and GCP Buckets object storage services are your best options when working with Azure and GCP. They both natively support tfstate locking and authentication which can be very useful to ease collaboration on Terraform codes and make sure only authorized people have access to the tfstate file. Once the storage services have been created, we need to tell Terraform to use them to store the tfstate file. Here are some tfstate backend configuration examples. Have a look at Azure Tfstate Backend Config and GCP Tfstate Backend Config for details or configuration snippets for other versions of Terraform.

• Azure

terraform {
  backend "azurerm" {
    resource_group_name  = "example-resource-group"
    storage_account_name = "example-storage-account"
    container_name       = "tfstate"
    key                  = "prod.terraform.tfstate"
  }
}

We can tell Terraform to authenticate to the Azure Storage Account using an Access Key or a SAS Token by setting the following environment variables before using Terraform commands :

# Use Storage Account Access Key for authentication
(bash)$ export ARM_ACCESS_KEY=<azure_storage_account_access_key>

# Use Storage Account SAS Token for authentication
(bash)$ export ARM_SAS_TOKEN=<azure_storage_account_sas_token>

• GCP

terraform {
  backend "gcs" {
    bucket  = "example-bucket"
    prefix  = "terraform/state"
  }
}

Before running Terraform commands with this backend configuration, make sure the GCP user or Service Account that is used by Terraform has read and write permissions on the configured bucket.

Use fixed providers versions

Terraform Providers are separate codes maintained either by the Terraform team, the Terraform community or Infrastructure platform companies like Google for Google Cloud Platform, Microsoft for Azure Cloud Platform or Amazon for Amazon Web Services Cloud platform. It defines a set of Terraform resources that can be used for instance to create resources into Cloud Providers platforms by leveraging their REST APIs. Here are examples of Terraform providers configuration for Azure and GCP.

Azure

provider "azurerm" {
  features {}
}

For details and more configuration options see Terraform Azure Provider

GCP

provider "google" {
  project     = "my-project-id"
  region      = "us-central1"
}

For details and more configuration options see Terraform GCP Provider and Terraform GCP Provider Configuration Reference

Once infrastructure resources are created with a specific provider version, it is important to fix that version to make sure the code works the same as before during next runs. Otherwise, Terraform will use the latest available version of the provider each time the code is run and new provider version most of the time means new features, deprecations and changes on associated Terraform resources codes. Here is how you make sure Terraform always uses a specific provider version.

• Use the required providers block

# Azure

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.26.0"
    }
  }
}

# GCP

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "=4.39.0"
    }
  }
}

In addition to version constraints matching the exact version number to use for a provider, there is other forms we could use. See Terraform Version Constraints Syntax for details.

In Terraform versions starting from 0.14, a dependency lock file is automatically created when the terraform init command is run. That file is named .terraform.lock.hcl and contains providers versions selected for use, based on versions contraints defined inside Terraform configuration files.

If there is no version constraints specified inside the required_providers block where we define which providers we want to use, the .terraform.lock.hcl file will contain the latest available provider version. The .terraform.lock.hcl can then be shared alongside the Terraform codes in order to make sure other people using it get the same providers versions.

When there is no version constraints defined inside Terraform configuration files but a .terraform.lock.hcl file already exists, you can ignore providers versions defined inside that dependency file and use the latest available ones thanks to the terraform init -upgrade command.

Configure required Terraform versions

In order to constrain our Terraform codes users to use specific Terraform version(s) for running the codes, we can define version constraints as follows :

terraform {
  required_version = "~> 1.3.6"
}

The ~> operator allows only the righmost version component to increment, which means that we could use Terraform versions 1.3.x with x >= 6 to run the codes. There is other version constraints forms we could use. See Terraform Version Constraints Syntax for details.

Authentication setup options

Authenticating with our own users accounts

This is naturally the authentication method to use when we run the Terraform codes ourselves. The permissions required to update the infrastructure are given to real people, and the information of who made which changes and when can easily be retrieved from the infrastructure platform's audit logs.

Google Cloud Platform (GCP)

• Pre-requisite : Install gcloud
• Then run the following commands :

$ gcloud auth login <gcp_user_account_email>

$ gcloud auth application-default login
(...)
Credentials saved to file: [/home/<user>/.config/gcloud/application_default_credentials.json]
(...)

$ export GOOGLE_APPLICATION_CREDENTIALS=/home/<user>/.config/gcloud/application_default_credentials.json

• After that, we should be able to use Terraform to create ressources we are authorized to on Google Cloud Platform

• The first login command will launch a web browser for authentication. If you prefer or are in an environment where the browser can't be launched you can add the --no-launch-browser option to that command. Run gcloud auth login -h to list all available options

Microsoft Azure Cloud Platform

• Pre-requisite : Install azure cli
• Then run the az login -u <azure_user_account_email> command
• A web browser will be launched for authentication
• After authenticating, we should be able to create Terraform resources we are authorized to on Azure

Authenticating with dedicated service accounts

If we don't want to use our own users accounts to run Terraform codes, for instance when we delegate the execution to CI/CD applications on code changes, it is also possible to do that on GCP using a Google Cloud Service Account and Azure using Managed Identities for instance. Here is the Terraform's Google Cloud and Azure providers documentations about how to use those types of accounts for running Terraform : Terraform Azure Managed Identities and Terraform GCP Service Account