10 Best Practices for Infrastructure

as Code with Terraform

Table of Contents
Table of Contents 1

Overview 2

Use a Version Control System (VCS) to Store Configurations 2

Secure the State Files 2

Packer for Image Creation and Terraform for Provisioning 3

Use a Micro-Infrastructure Approach 4

Functional Workspaces 5
Workspace Best Practices 5
Workspace per Infrastructure Unit 6

Integrate with a Secrets Management Solution 7

With the use of a Continuous Integration (CI) Tool 7
From a Workspace with the Terraform Vault Provider 8

Use Continuous Integration (CI) Pipelines 9

Modules and Variables for Code Reuse 10

Avoid Hardcoding - Use Variables 10
Enable Self-Service with Modules 10
Create Standard Modules for Commonly Deployed Infrastructure 11
When to use Modules 11

Policy as Code to Govern Provisioning 12

Testing Terraform Deployments 13

Validating the code itself 13
Validating inputs/variables when before a run 13
Validating the deployed infrastructure 13

Avoid Rolling Your Own Solution 14

Conclusion 15

References 15

1
 Overview

Terraform has become the industry standard for Infrastructure as Code today. In this article, we
present 10 best practices for using Terraform at scale in production. These practices are
opinionated and based on how we’ve seen successful organizations adopt Terraform.

1. Use a Version Control System (VCS) to Store Configurations

The VCS is your single source of truth. All your Terraform configurations need to live in your
VCS. The VCS ensures you have history and can roll back if mistakes occur. VCS should be
used as the default mechanism for introducing changes to infrastructure.

Below is an example of connecting your VCS to Terraform Enterprise (TFE) or Terraform Cloud
(TFC)

2. Secure the State Files

A Terraform state file is a file that stores all the resources that Terraform provisioned in the real
world. It’s a very important file because it allows Terraform to know what’s been deployed in
your infrastructure. This file also contains some secret information such as credentials for your
cloud providers.

It is very important to securely store these state files. Some organizations have standardized on
using an S3 bucket in AWS or blob storage in Azure. However, care needs to be taken to make
sure that these storage locations are private and that the state files are encrypted at rest.

Another alternative is to adopt TFE or TFC for state file secure storage. TFE and TFC securely
manage your state files, which prevents:

2
 ● Leakage of credentials that are embedded in your state files
● Corruption and/or loss of state files

3. Packer for Image Creation and Terraform for Provisioning

When it comes to provisioning VMs, you could choose to use a vanilla image from your favourite
cloud provider or build your own. Packer is a great solution to create VM images for cloud
providers and for vSphere on-premises.

Below are some best practices specifically when using Packer:

● Put into your image any dependencies your app needs to run -- libraries, windows
security updates, etc.
● DO NOT put into your image - things like passwords, or hardcoded configuration that
may dynamically change. Terraform will deal with these.
● Static elements that take a long time to install, go in Packer.
● Dynamic elements that need to be different per-instance, should happen at provisioning
time, use Terraform.
● Use Packer Build Pipelines to build several specialized images

3
 More on Packer Build Pipelines:

● When creating Packer templates, create builds in a step-by-step process. Do not try and
do everything in a single Packer build.
● Instead, create several Packer build.json files and “chain” them together:
● Keep templates as generic as possible and use User Variables.
● For windows images, keep WinRM disabled or blocked by the firewall until the system
has had its final boot after sysprep. Check out this article on best practices with Packer
and Windows.

4. Use a Micro-Infrastructure Approach

Similar to the benefits obtained when using a microservices framework, you should consider
using a micro-infrastructure approach to building your infrastructure resources. The main
benefits are:
● Each team manages their own set of resources which speeds up development and
deployment
● Role based access control (RBAC) can be used to make sure certain resources are not
accessible to certain teams

4
Terraform manages collections of infrastructure resources (networking, infrastructure, app A,
app B, etc…). TFC or TFE manages infrastructure collections with “workspaces”. A workspace
contains everything Terraform needs to manage a given collection of infrastructure.

Functional Workspaces

● “Micro-Infrastructures” are linked to create the complete infrastructure for the

application
● TFC and TFE offer “Remote Backend” which allows workspaces to refer to each other’s
output variables

Below is a diagram showing how an entire organization can use Terraform in a

micro-infrastructure framework.

Workspace Best Practices

● One Workspace Per Environment Per Terraform Configuration

● Name your workspaces with both their component and their environment
● Use per-workspace access controls to delegate ownership of components and regulate
code promotion across environments

5
Workspace per Infrastructure Unit
A single workspace is used to manage a standalone deployable infrastructure unit. Below are
three examples of how to organize workspaces.

6
5. Integrate with a Secrets Management Solution
As organizations move to hybrid cloud environments, they realize the importance of a secrets
management solution. My recommendation here is to use HashiCorp Vault to centralize all your
secrets in one location.

Terraform integrates well with Vault and below are two common examples.

With the use of a Continuous Integration (CI) Tool

If calling terraform as part of your CI pipeline, you can do the following:

● CI tool to call on Vault to generate ephemeral credentials for accessing your provisioning
environment such as Azure in this diagram.
● The CI tool then injects these credentials into the terraform workspace.
● Then the workspace Apply gets triggered.
● In this scenario, the CI tool needs to authenticate itself to Vault, and should have
permissions to request these dynamic credentials
● Also, always make sure your credentials have a short Time-to-live (TTL) to reduce risk
exposure

7
From a Workspace with the Terraform Vault Provider

The Terraform Vault Provider may be used to dynamically retrieve credentials from Vault for the
workspace to execute. In this scenario, Terraform should be able to authenticate to Vault and
have permissions to generate these credentials. The generated credentials are used by the
workspace to authenticate to the cloud provider.

These credentials will appear in the state file, so it is good practice to create them with a short
TTL, as well as restrict user access to the state file.

Notes:
● Steps 1 & 2 happen only when a workspace is on-boarded
● Cloud credentials will be saved in the Terraform state file. To mitigate exposure risk,
restrict access to the state file or/and minimize the credential TTL

8
6. Use Continuous Integration (CI) Pipelines

This is a detailed workflow where we use Jenkins as an example of a CI tool to do the following:
● Grabs configuration files from VCS
● Dynamically creates a workspace (if does not exist)
● Injects the necessary variables into the Terraform workspace before executing the run.
● In this scenario, Terraform will not pull the code directly from VCS (even though
possible), but rather, Jenkins will get the code from VCS and upload it to the Terraform
workspace.

With this, Jenkins is able to promote the same code between different environments.

9
7. Modules and Variables for Code Reuse

Avoid Hardcoding - Use Variables

You can improve reusability of configurations by using variables.

● Variables allow standard infrastructure to be used more flexibly
● Use variables even when unsure but providing sensible defaults

Enable Self-Service with Modules

As the adoption of Terraform at your organization increases, start thinking in terms of a producer
consumer model. This allows the adoption of a self-service framework. Producers create
modules, whereas, consumers don’t need to know the specifics within each module. Consumers
treat modules as black boxes with inputs and outputs. This makes it easier to reuse code and
enables these consumers to serve themselves without the need to submit tickets to a central
team of producers.

10
Create Standard Modules for Commonly Deployed Infrastructure
● Accelerate time to value by offering standard modules for use
● Further improve productivity by designing well test configurations from standard modules
and offering them through self service channels

11
 When to use Modules

It’s not always clear when to create a module and when to put your configs in a config file. The
flowchart above walks through the decision process.

8. Policy as Code to Govern Provisioning

As organizations look for ways to speed up application development, they start to think of
moving away from ticketing systems to more self-service models. It's important to ensure and
enforce some sort of governance when adopting a self-service framework.

There are multiple policy as code solutions out there. The Open Policy Agent (OPA) is a popular
one for generic policy as code. However, specific to Terraform, HashiCorp Sentinel is a policy as
code solution that is shipped with TFE or TFC.

12
 Benefits of Sentinel:
● Reduce mistakes, make tracking easier and reduce coordination with stakeholders of the
provisioning process.
● Leverage Terraform Foundational Policies to accelerate your policy rule development.
● Tightly integrated with all HashiCorp products.

9. Testing Terraform Deployments

Below are the three levels of validation/testing that we usually come across.

1. Validating the code itself

For that we recommend using the validate command.

2. Validating inputs/variables when before a run

For that, we typically use Sentinel policies

3. Validating the deployed infrastructure

For this, we can use the below or any similar tools
- Terratest
- Kitchen Terraform
- Inspec
- Serverspec

Moreover, take a look at the diagram below for an end-to-end view of testing.

13
 Terraform Cloud or Enterprise offers the following checks:
● Syntax - use VS Code Plugin
● Plan - use provider checks
● Sentinel - use governance checks
● Apply - use cloud checks

10. Avoid Rolling Your Own Solution

Consider using Terraform Enterprise or Terraform Cloud from HashiCorp if you need any of
these:

● Multiple Teams to provision infrastructure

● Need to minimize coordination between teams
● Provide guardrails on the provisioning process
● Security around cloud infrastructure provisioning credentials
● Setting up module registries to standardize provisioned infrastructure
● Harden your provisioning process with 2 Factor Authentication (2FA)
● Have a requirement to be supported in production

14
The above diagram shows how to use Terraform Cloud or Terraform Enterprise for your
workflow.

● Developers check in their configs/policies to the Version Control System (VCS)

● A VCS repos is mapped to a workspace on Terraform
● Terraform can monitor these repos and automatically execute newly committed configs
● Alternatively, an orchestrator such as Jenkins may prompt terraform to Plan and Apply a
particular workspace as part of a broader deployment pipeline
● Sentinel policies are checked between the plan and apply phases to enable self-service
and enforce governance
● Following the provisioning, the orchestrator may call external config management tools
to configure the provisioned infrastructure

Conclusion
In this article we reviewed 10 best practices for using Terraform in production environments.
This was an opinionated view based on what we see with successful organizations in their
adoption of Terraform. If you have questions or comments, feel free to message us here.

References
● The Core Terraform Workflow - Guides
● Workspaces - Terraform Cloud and Terraform Enterprise
● Naming - Workspaces - Terraform Cloud and Terraform Enterprise
● Terraform Configurations - Workspaces - Terraform Cloud and Terraform Enterprise

15