Scaling infrastructure deployments with Terraform – From manual deployments to self-service platform

Why Infrastructure as Code?
- Predictable and reproducible deployments:
- Standardization, security and policy compliance:
Terraform basics and core Concepts
Terraform Modules as Building Blocks
Terraform Modules as a Self-Service Capability
- Governance and Guardrails
Conclusion

Infrastructure management has changed significantly over the past decade. What was once managed manually through cloud consoles, scripts, and web UIs is increasingly becoming part of the software development lifecycle. Infrastructure as Code (IaC) is at the heart of this transformation. Instead of configuring infrastructure manually, infrastructure is defined in reusable templates, which are developed, versioned and applied by the platform team.

One of the most widely adopted tools for this approach is Terraform. This article provides an introduction to the basics of Terraform and explores how organizations can move from manual infrastructure provisioning to shared custom modules and ultimately to a self-service approach, enabling non-platform engineers while still guaranteeing compliance with organizational policies.

Why Infrastructure as Code?

Before looking at Terraform itself, it is worth understanding the problem Infrastructure as Code is trying to solve:

Predictable and reproducible deployments:

Traditionally, infrastructure was often created manually. An administrator would log into a cloud console or web UI, create a virtual network, provision a database, configure access rules, and repeat the process for every environment. This approach may work for a small organization or a PoC. As infrastructure grows, however, manual resource management becomes increasingly difficult and error prone:

Different environments may have slightly different configurations. Resources can be forgotten or configured incorrectly. Changes may not be documented properly, and reproducing an environment can become a time-consuming task.

With IaC, the desired infrastructure is described in templates rather than being created manually. Deploying the same infrastructure in different environments thus becomes a matter of applying the correct set of parameters to the templates during the deployment process.

With IaC the before mentioned administrator needs to define how the virtual network, database and access rules should look like in a parametrized template and apply the correct parameter set during the deployment. This approach makes infrastructure deployment more predictable and reproducible, while changes can be tracked through version control.

Standardization, security and policy compliance:

Applying IaC and establishing templates and modules that are used company wide, allows organizations to establish common infrastructure patterns. Naming conventions, tagging, networking configurations as well as security and policy requirements can be met consistently by encapsulating them in custom modules (see chapter 3).

Terraform basics and core Concepts

Declarative configuration:

Terraform is an open source Infrastructure as Code tool provided by Hashicorp that allows infrastructure to be defined and managed through declarative configuration. This means, that instead of scripting every individual step required to create the infrastructure resources in sequence, you describe the desired end state. Terraform then determines how to reach that state: Terraform creates a DAG (Directed Acyclic Graph), defining the order of deployment and thus resolving found dependencies between the defined resources.

Terraform state:

While Terraform determines the needed deployment steps in order to configure the provided cloud environment according to the desired state defined in the template, it does not only compare the current state of the remote resources with their counterparts in the template. It also relies on the Terraform state file. This state file contains a 1:1 mapping of the defined resources in the template to their real counterparts, as well as additional metadata, such as dependencies as of the last deployment. (This is especially important in case of decommission of previously defined resources.)

Per default the state is stored in a file in the local working directory, though, it is recommended to use a remote location to store the state. With a remote state setup, Terraform can use remote locking, thereby preventing multiple users to apply changes to the same project at the same time.

Providers:

In comparison to other IaC tools, Terraform is not relying on one specific cloud provider or service. It uses providers to interact with different cloud services, such as AWS, Azure or Google Cloud, but is not limited to those. While this article focuses on the usage of Terraform with Azure, note that Terraform also has providers for Kubernetes, VMWare, vSphere and various other services and platforms, which can also be run in an on premise environment. Thus, a Terraform provider just serves as a translator between Terraform code and API calls to the underlying platform.

Modules:

Modules allow Terraform resources and configurations to be packaged into reusable building blocks. For example, a module can contain an Azure SQL Server resource, as well as an encryption key resource and a private endpoint resource. Additionally, while certain properties of the SQL Server might still be exposed as input parameters for the module (more on that later), values of other properties, such as public access, can be enforced by hardcoding them in the module. This way, modules can help standardizing infrastructure setups across multiple applications, as well as enforce security and company policies.

Terraform Modules as Building Blocks

While Terraform technically does not rely on a given folder structure or naming convention, a typical setup of a Terraform module will contain the following files:

provider.tf
versions.tf
input.tf
output.tf
main.tf
README.md

In the following subchapters, we will define a module for the previously discussed example of the platform administrator being tasked to deploy a SQL Server. The following Azure resources will need to be deployed by the module:

MS SQL Server
private endpoint to MS SQL Server
Customer Managed Encryption Key (CMK)
Role assignment for the managed identity of the SQL Server in the Keyvault holding the CMK

provider.tf

In the provider.tf file, the needed providers for the module to function are defined. Additionally, the features block enables the developers to define certain resource attributes on provider level.

provider "azurerm" {
  features {}
}

versions.tf

The versions.tf file defines the required provider versions and is needed to prevent version drift between the provider used during module development and actual usage in a productive environment.

terraform {
  required_version = ">= 1.16"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=> 5.9"
    }
  }
}

input.tf

Defines the variables used in the module. The specific values are passed in a module call in a Terraform project and are used to parametrize the resources that are defined in the main.tf file.

variable "sqlServerName" {
    description = "Name of the Azure SQL Server to be deployed."
    type = string
}

# Terraform variables can also contain a validation block in the definition. The value provided during 
# module call will be validated by the provided condition. If it evaluates to ´true´, the module call 
# will continue. If the condition evaluates to ´false´ the terraform plan/apply will fail with the 
#custom error message.
variable "environment" {
    description = "Environment for deployment. Possible values are: dev, uat, sit, prd."
    type = string
    validation {
        condition     = contains(["dev", "uat", "sit", "prd"],lower(var.environment))
        error_message = "Value for variable ´environment´ is not available. Possible values are: dev, uat, sit, prd."
  }
}

# Terraform variables can have a defined default value. If a default value is defined and no value for the 
# respective variable is passed during module call, the module call will continue with the provided 
# default value.
variable "sqlServerVersion" {
    description = "Version of the SQL Server. Possible values are: 2.0 (for v11 Server) and 12.0 (for v12 Server). Default is set to '12.0'."
    type = string
    default ="12.0"
}

variable "azureAdAdminName" {
    type = string
    description = "Azure AD Entity name set as Administrator to enable AD authentication only on SQL Server."
}

variable "azureAdAdminId" {
    type = string
    description = "Azure AD Entity object ID set as Administrator to enable AD authentication only on SQL Server."
}

variable "privateEndpointSubnetId" {
    description = "VNet Subnet ID for private endpoint deployment."
    type = string
}

variable "privateLinkServiceId" {
    description = "Private Link Service ID for private endpoint deployment."
    type = string
}

variable "encryptionKeyKvId" {
    description = "Azure Keyvault object to store SQL Server encryption key."
    type = string
}

# Module variables can also be defined as an object with predefined attributes. In this example a 
# ´azurerm_resource_group´ object will be passed during the module call and its attributes ´name´ 
# and ´location´ will be used during module execution. 
# NOTE: It is not needed to add every attribute of the intended input object (´azurerm_resource_group´ 
#       in this example) to the variable. It is sufficient to define just those attributes that are 
#       referenced in the module.
variable "resourceGroup" {
    description = "Resoure Group object where the SQL Server will be deployed."
    type = object({
        name = string
        location = string
    })
}

main.tf

Contains the resources provisioned by the module. As can be seen in this example, defining a custom module for the MS SQL Server achieved multiple goals:

Encapsulates complexity of infrastructure setup. The necessary setup needs to be defined once and not be reinvented by other teams and/or projects again.
Standardization: The SQL Server setup of Team A will look similar to the setup of Team B.
Security and company policies: Security policies, such as restricting public access and AAD authentication only in the example, can be enforced within the module.

resource "azurerm_mssql_server" "msSqlSrv" {
  name                          = "${var.sqlServerName}-${var.environment}"
  resource_group_name           = var.resourceGroup.name
  location                      = var.resourceGroup.location
  version                       = var.sqlServerVersion
  minimum_tls_version           = "1.2"
  public_network_access_enabled = false

  identity {
    type         = "SystemAssigned"
  }

  azuread_administrator {
    azuread_authentication_only = true
    login_username              = var.azureAdAdminName
    object_id                   = var.azureAdAdminId
  }

  transparent_data_encryption_key_vault_key_id = azurerm_key_vault_key.cmk.id
}

resource "azurerm_role_assignment" "kvCryptoUserSqlSrv" {
  scope                = var.encryptionKeyKvId
  role_definition_name = "Key Vault Crypto Service Encryption User"
  principal_id         = azurerm_mssql_server.msSqlSrv.identity[0].principal_id
}

resource "azurerm_key_vault_key" "cmk" {

  name         = "${var.sqlServerName}-key"
  key_vault_id = var.encryptionKeyKvId
  key_type     = "RSA"
  key_size     = 2048

  key_opts = ["unwrapKey", "wrapKey"]
}

resource "azurerm_private_endpoint" "privateEndpointSqlSrv" {
  name                = "pe-${var.sqlServerName}-${var.environment}-sql"
  resource_group_name = var.resourceGroup.name
  location            = var.resourceGroup.location
  subnet_id           = var.privateEndpointSubnetId

  private_service_connection {
    name                           = "pe-${var.sqlServerName}-${var.environment}-sql-privateserviceconnection"
    private_connection_resource_id = var.privateLinkServiceId
    is_manual_connection           = false
  }
}

output.tf

Defines which resources and/or respective properties are accessible in the Terraform project where the module is called.

output "msSqlSrv" {
    description = "MS SQL Server Object created in module."
    value = azurerm_mssql_server.msSqlSrv
}

In the previous example the platform administrator went from configuring the Vnet, SQL Server, role assignments and private endpoints manually in the web UI, to adding them in the Terraform template for the target environment. Now, once the module is defined and ready to use, the platform administrator only needs to add one module call to the Terraform template for any future SQL Server deployment:

Considerations for designing shared Modules

A good module should provide a clear and structured documentation. The input variables i.e. interface, as well as the module itself should be documented in a way, that no handover from the developer to consumer is necessary.
The module should be designed as flexible as possible. A general approach can be: in case a property of the main resource is not restricted by security or company policies, it should be exposed as module variable. This approach provides flexibility for the module usage and hardens the interface against breaking changes.
Avoid feature bloat. While flexibility in its appliance to different use cases was previously mentioned as a design goal, in order to maintain a module, a clear scope of its functionality needs to be defined.
Versioning is important. Once a module is used by multiple teams, changes to the module can potentially affect many applications. A Terraform module should not simply be a collection of Terraform code. It should be treated as a reusable infrastructure building block with a defined interface, purpose and lifecycle.

Terraform Modules as a Self-Service Capability

Switching from manual infrastructure deployments through the cloud providers web UI to using IaC is a major step towards standardization not only between different environments, but also different teams and applications. Using modules as reusable building blocks can also solve an organizational problem: how can application teams request infrastructure without requiring the platform team to provision every requested change?

A possible solution is enabling the application teams to provision certain infrastructure parts themselves, by providing a comprehensive catalogue of Terraform modules. A module does not automatically have to become a self-service interface. An organization can continue to use certain modules only centrally within the platform team. More on that point in the following chapter.

Let us revisit the example of the platform administrator being requested to provision a SQL Server in an Azure subscription:

We went from provisioning and configuring all resources needed in the web UI to defining them in a Terraform template and finally to defining a standardized set of server configurations, role assignments as well as networking setup and encapsulated this in a custom module that is called in the template. The next step is to involve the requesting team directly: instead of creating a Ticket for the platform team, the application team can create a Pull Request in the repository governing their subscription and include the module call with the desired specifications directly. The platform administrator can then review the Pull Request and approve it for deployment.

This approach reduces operational overhead from the platform team and alleviates potential bottlenecks for the application team. The platform team therefore does not have to manually execute every infrastructure request, while the application team does not need to become an expert in the underlying infrastructure implementation.

Governance and Guardrails

While the self-service approach does reduce operational overhead of the platform team over time, there are some considerations to be made in advance:

Defining clear boundaries for what can be governed by self-service and what shouldn’t

Setting up a SQL Server, a storage account or a new container in an Azure subscription, managing the role assignments on the respective resources are tasks, which can be governed by self-service with a low risk of causing bigger issues. On the other hand, setting up a new subscription, setting a budget, integrating a new Vnet in the existing network infrastructure or managing an MS Entra tenant are tasks, where a self-service approach is not advisable. Such requests should still be implemented by the platform team or the respective networking/security specialists.

Setting up guardrails for application teams

Before providing access to the subscription repositories as well as the shared modules, a clear process, as well as guardrails and possible support workflows need to be defined. Here are a few questions to guide the process, that need to have clear answers before starting the self-service project:

What rules and conventions are employed in the subscription repositories? How are they enforced?
How do you ensure that no changes to resources are made, that are out of bound for the self-service approach?
How do you prevent simultaneous deployments in the same subscription by multiple users.
Is there a clear process for application teams on how to contribute on the subscription repositories?
Are there ways to circumvent the outlined process? If so, how can they be solved?
Is the documentation in place and comprehensive, so that application teams can start and discover the process autonomously?

Module Repositories and Versioning

A central module repository can provide a consistent place for teams to discover and consume approved shared modules. For a smaller number of shared modules, this is a good approach. Each module can be contained in its own directory and can be referenced accordingly in the module call in the template. With a rising number of shared modules and collaborators, a switch to a dedicated repository per module is preferred.

The modules should follow a clearly defined lifecycle should also be versioned accordingly. This directly ties to the usage of the shared modules: when a module is called in a terraform template, a source to the module definition needs to be provided. This can be a directory, a repository branch or a repository along with a tag. The latter option should always be chosen.

Conclusion

In this article we have seen, how managing infrastructure in a cloud environment has changed with the advancing adoption of Infrastructure as Code (with focus on Terraform) and what possibilities it provides in terms of standardization and scalability. It is also important to notice, that the outlined development from manual deployments in the web UI to managed self-service platform with standardized modules for every usecase, is only an outlook on how scalability in infrastructure management can be achieved and not a solution that fits every company. While for smaller companies with only a few application teams, having a small platform team maintaining the subscription and module repositories themselves might already be sufficient, bigger organizations will need to consider scalability.