Terraform Remote Backends: Team Collaboration without State Problems

Introduction
#

With Terraform we write infrastructure as code. We run terraform apply in the terminal and the servers are ready. If we work alone, the local state file on the hard drive is enough.

However, when multiple people work on a project, we face problems:

Who has the current state of the infrastructure?
What happens if two people make changes at the same time?
Sensitive data like passwords are saved in plaintext. They must not be saved in Git.

Remote Backends solve these problems. They store the state of the infrastructure in a central place. They lock the state during changes and keep passwords out of Git.

In this article we set up two solutions: The GitLab HTTP backend and Hetzner Object Storage.

Why Local State does not work for Teams
#

The local state file is not suitable for teams:

No Shared Storage: If Developer A creates a server, Developer B does not know about it. If Developer B runs apply, he overwrites the changes of Developer A.
No Locking: If two people make changes at the same time, the state file gets corrupted.
Security Risks: Passwords are saved in plaintext in the state. If you save the file in Git, you expose sensitive data.

A remote backend solves these problems:

flowchart TD
    DevA["Developer A"]
    DevB["Developer B"]
    Lock["State Lock"]
    State["Central State"]

    DevA -->|Step 1: Request lock| Lock
    Lock -->|Step 2: Lock granted| DevA
    DevA -->|Step 3: Write state| State
    DevB -->|Step 4: Request lock| Lock
    Lock -.->|Step 5: Blocked| DevB
    DevA -->|Step 6: Release lock| Lock

    style DevA fill:#1e40af,stroke:#1e3a8a,color:#fff
    style Lock fill:#ef4444,stroke:#b91c1c,color:#fff
    style State fill:#10b981,stroke:#047857,color:#fff

Solution 1: GitLab HTTP Backend
#

If the team already uses GitLab, the HTTP backend is the easiest choice. GitLab has its own interface for Terraform states and supports locking over HTTP.

The Terraform Configuration
#

we keep the backend block in the configuration empty. We pass the credentials later when starting. This avoids passwords in the code.

Let us create the file backend.tf:

terraform {
  backend "http" {
  }
}

Initialization via CLI
#

We need a Personal Access Token from GitLab with API permissions. We initialize Terraform with this command:

terraform init \
  -backend-config=address=https://gitlab.com/api/v4/projects/<PROJECT_ID>/terraform/state/<STATE_NAME> \
  -backend-config=lock_address=https://gitlab.com/api/v4/projects/<PROJECT_ID>/terraform/state/<STATE_NAME>/lock \
  -backend-config=unlock_address=https://gitlab.com/api/v4/projects/<PROJECT_ID>/terraform/state/<STATE_NAME>/lock \
  -backend-config=username=<GITLAB_USERNAME> \
  -backend-config=password=<PERSONAL_ACCESS_TOKEN> \
  -backend-config=lock_method=POST \
  -backend-config=unlock_method=DELETE \
  -backend-config=retry_wait_min=5

Replace the placeholders:

<PROJECT_ID>: The ID of your GitLab project.
<STATE_NAME>: The name for the state.

The state is visible in GitLab under Operate > Terraform states.

Use in GitLab Pipelines
#

In a pipeline we use the token of the runner. We do not need to store private passwords.

Here is a simple example for the file .gitlab-ci.yml:

image: hashicorp/terraform:light

variables:
  TF_STATE_NAME: "default"
  TF_ADDRESS: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}"

before_script:
  - terraform init \
      -backend-config="address=${TF_ADDRESS}" \
      -backend-config="lock_address=${TF_ADDRESS}/lock" \
      -backend-config="unlock_address=${TF_ADDRESS}/lock" \
      -backend-config="username=gitlab-ci-token" \
      -backend-config="password=${CI_JOB_TOKEN}" \
      -backend-config="lock_method=POST" \
      -backend-config="unlock_method=DELETE"

stages:
  - validate
  - plan
  - apply

validate:
  stage: validate
  script:
    - terraform validate

plan:
  stage: plan
  script:
    - terraform plan
  artifacts:
    paths:
      - plan.tfplan

apply:
  stage: apply
  script:
    - terraform apply -auto-approve plan.tfplan
  dependencies:
    - plan
  when: manual

Solution 2: Hetzner Object Storage
#

Hetzner offers cheap and GDPR compliant Object Storage. We use the S3 backend of Terraform for this. Since it is not AWS S3, we must disable some checks.

Limitations with Locking
#

The S3 backend of Terraform normally locks the state via AWS DynamoDB. Since Hetzner Object Storage does not offer this database, there is no automatic locking.

In a team, changes should only run via pipelines. This prevents parallel changes.

The Terraform Configuration for Hetzner S3
#

We create a bucket and credentials in the Hetzner Cloud Console.

The file backend.tf looks like this:

terraform {
  backend "s3" {
    bucket                      = "your-terraform-state-bucket"
    key                         = "env/production/terraform.tfstate"
    region                      = "eu-central-1" # Dummy value for Terraform
    
    endpoints = {
      s3 = "https://nbg1.your-objectstorage.com" # Region of the bucket
    }

    # Disable AWS checks
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    
    use_path_style              = true
  }
}

Initialization and Credentials
#

We store the credentials in environment variables. Terraform reads these variables automatically:

export AWS_ACCESS_KEY_ID="YOUR_HETZNER_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="YOUR_HETZNER_SECRET_KEY"

terraform init

Terraform now stores the state on the servers of Hetzner.

Migrating the Local State
#

If we want to move a local state, we follow these steps:

We add the new backend block.
We run terraform init.
We confirm the prompt with yes to copy the data.

Terraform uploads the state and renames the local file.

Conclusion
#

A remote backend is necessary for working in a team.

GitLab HTTP is easy to set up and offers automatic state locking.
Hetzner Object Storage is cheap and privacy friendly, but needs pipelines for coordination.

Feel free to contact us through the channels below if you have any questions.