Scaling Infrastructure Deployments with Terraform – From Manual Deployments to a Self-Service Platform

Why Infrastructure as Code?
- Predictable and reproducible deployments
- Standardization, security and policy compliance
Terraform Basics and Core Concepts
Terraform Modules as Building Blocks
Terraform Modules as a Self-Service Capability
- Governance and Guardrails
Conclusion

Inastructure management has changed significantly over the past decade. What was once managed manually through cloud consoles, scripts, and web UIs is increasingly becoming part of the software development lifecycle. Infrastructure as Code (IaC) is at the heart of this transformation. Instead of configuring infrastructure manually, infrastructure is defined in reusable templates, which are developed, versioned and applied by the platform team.

One of the most widely adopted tools for this approach is Terraform. This article provides an introduction to the basics of Terraform and explores how organizations can move from manual infrastructure provisioning to shared custom modules and ultimately to a self-service approach, enabling non-platform engineers while still guaranteeing compliance with organizational policies.

Why Infrastructure as Code?

Before looking at Terraform itself, it is worth understanding the problem Infrastructure as Code is trying to solve:

Predictable and reproducible deployments

Traditionally, infrastructure was often created manually. An administrator would log into a cloud console or web UI, create a virtual network, provision a database, configure access rules, and repeat the process for every environment. This approach may work for a small organization or a PoC. As infrastructure grows, however, manual resource management becomes increasingly difficult and error-prone:

Different environments may have slightly different configurations. Resources can be forgotten or configured incorrectly. Changes may not be documented properly, and reproducing an environment can become a time-consuming task.

With IaC, the aforementioned administrator needs to define what the virtual network, database, and access rules should look like in a parameterized template and apply the correct parameter set during deployment. This approach makes infrastructure deployment more predictable and reproducible, while changes can be tracked through version control.

With IaC the before mentioned administrator needs to define how the virtual network, database and access rules should look like in a parametrized template and apply the correct parameter set during the deployment. This approach makes infrastructure deployment more predictable and reproducible, while changes can be tracked through version control.

Standardization, security and policy compliance

Applying IaC and establishing company-wide templates and modules allows organizations to establish common infrastructure patterns. Naming conventions, tagging, networking configurations, as well as security and policy requirements can be met consistently by encapsulating them in custom modules (see “Terraform Modules as Building Blocks”).

Terraform Basics and Core Concepts

Declarative configuration

Terraform is an Infrastructure as Code tool by HashiCorp that allows infrastructure to be defined and managed through declarative configuration. This means that instead of scripting every individual step required to create the infrastructure resources in sequence, you describe the desired end state. Terraform then determines how to reach that state: it creates a directed acyclic graph (DAG) that defines the order of deployment and resolves dependencies between the defined resources.

Terraform state

To determine the deployment steps needed to bring the cloud environment into the desired state defined in the template, Terraform does not only compare the current state of the remote resources with their counterparts in the template. It also relies on the Terraform state file. This file contains a 1:1 mapping of the resources defined in the template to their real counterparts, as well as additional metadata, such as dependencies as of the last deployment. This is especially important when previously defined resources are decommissioned.

By default, the state is stored in a file in the local working directory. However, it is recommended to store the state in a remote location. With a remote state setup, Terraform can use state locking, which prevents multiple users from applying changes to the same project at the same time.

Providers

Unlike some other IaC tools, Terraform does not rely on one specific cloud provider or service. It uses providers to interact with different cloud services, such as AWS, Azure, or Google Cloud, but is not limited to those. While this article focuses on using Terraform with Azure, Terraform also offers providers for Kubernetes, VMware vSphere, and various other services and platforms, some of which can also run on-premises. In essence, a Terraform provider serves as a translator between Terraform code and the API calls of the underlying platform.

Modules

Modules allow Terraform resources and configurations to be packaged into reusable building blocks. For example, a module can contain an Azure SQL Server resource, as well as an encryption key resource and a private endpoint resource. Additionally, while certain properties of the SQL Server might still be exposed as input parameters for the module (more on that later), values of other properties, such as public access, can be enforced by hardcoding them in the module. This way, modules can help standardize infrastructure setups across multiple applications and enforce security and company policies.

Terraform Modules as Building Blocks

While Terraform technically does not rely on a given folder structure or naming convention, a typical setup of a Terraform module will contain the following files:

provider.tf
versions.tf
input.tf
output.tf
main.tf
README.md

In the following subsections, we will define a module for the previously discussed example of the platform administrator tasked with deploying a SQL Server. The following Azure resources will need to be deployed by the module:

MS SQL Server
Private endpoint to MS SQL Server
Customer-Managed Encryption Key (CMK)
Role assignment for the managed identity of the SQL Server in the Key Vault holding the CMK

provider.tf

The provider.tf file defines the providers the module needs to function. Additionally, the features block enables developers to define certain resource attributes at the provider level.

provider "azurerm" {
  features {}
}

versions.tf

The versions.tf file defines the required provider versions and prevents version drift between the provider used during module development and the one used in production.

terraform {
  required_version = ">= 1.16"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=> 5.9"
    }
  }
}

input.tf

Defines the variables used in the module. The specific values are passed in a module call in a Terraform project and are used to parametrize the resources that are defined in the main.tf file.

variable "sqlServerName" {
    description = "Name of the Azure SQL Server to be deployed."
    type = string
}

# Terraform variables can also contain a validation block in the definition. The value provided during 
# module call will be validated by the provided condition. If it evaluates to ´true´, the module call 
# will continue. If the condition evaluates to ´false´ the terraform plan/apply will fail with the 
#custom error message.
variable "environment" {
    description = "Environment for deployment. Possible values are: dev, uat, sit, prd."
    type = string
    validation {
        condition     = contains(["dev", "uat", "sit", "prd"],lower(var.environment))
        error_message = "Value for variable ´environment´ is not available. Possible values are: dev, uat, sit, prd."
  }
}

# Terraform variables can have a defined default value. If a default value is defined and no value for the 
# respective variable is passed during module call, the module call will continue with the provided 
# default value.
variable "sqlServerVersion" {
    description = "Version of the SQL Server. Possible values are: 2.0 (for v11 Server) and 12.0 (for v12 Server). Default is set to '12.0'."
    type = string
    default ="12.0"
}

variable "azureAdAdminName" {
    type = string
    description = "Azure AD Entity name set as Administrator to enable AD authentication only on SQL Server."
}

variable "azureAdAdminId" {
    type = string
    description = "Azure AD Entity object ID set as Administrator to enable AD authentication only on SQL Server."
}

variable "privateEndpointSubnetId" {
    description = "VNet Subnet ID for private endpoint deployment."
    type = string
}

variable "privateLinkServiceId" {
    description = "Private Link Service ID for private endpoint deployment."
    type = string
}

variable "encryptionKeyKvId" {
    description = "Azure Keyvault object to store SQL Server encryption key."
    type = string
}

# Module variables can also be defined as an object with predefined attributes. In this example a 
# ´azurerm_resource_group´ object will be passed during the module call and its attributes ´name´ 
# and ´location´ will be used during module execution. 
# NOTE: It is not needed to add every attribute of the intended input object (´azurerm_resource_group´ 
#       in this example) to the variable. It is sufficient to define just those attributes that are 
#       referenced in the module.
variable "resourceGroup" {
    description = "Resoure Group object where the SQL Server will be deployed."
    type = object({
        name = string
        location = string
    })
}

main.tf

Contains the resources provisioned by the module. As can be seen in this example, defining a custom module for the MS SQL Server achieves multiple goals:

Encapsulation: The complexity of the infrastructure setup is encapsulated. The setup only needs to be defined once and does not have to be reinvented by other teams or projects.
Standardization: The SQL Server setup of Team A will look similar to the setup of Team B.
Security and company policies: Security policies, such as restricting public access and allowing only Microsoft Entra ID authentication (as in the example), can be enforced within the module.

resource "azurerm_mssql_server" "msSqlSrv" {
  name                          = "${var.sqlServerName}-${var.environment}"
  resource_group_name           = var.resourceGroup.name
  location                      = var.resourceGroup.location
  version                       = var.sqlServerVersion
  minimum_tls_version           = "1.2"
  public_network_access_enabled = false

  identity {
    type         = "SystemAssigned"
  }

  azuread_administrator {
    azuread_authentication_only = true
    login_username              = var.azureAdAdminName
    object_id                   = var.azureAdAdminId
  }

  transparent_data_encryption_key_vault_key_id = azurerm_key_vault_key.cmk.id
}

resource "azurerm_role_assignment" "kvCryptoUserSqlSrv" {
  scope                = var.encryptionKeyKvId
  role_definition_name = "Key Vault Crypto Service Encryption User"
  principal_id         = azurerm_mssql_server.msSqlSrv.identity[0].principal_id
}

resource "azurerm_key_vault_key" "cmk" {

  name         = "${var.sqlServerName}-key"
  key_vault_id = var.encryptionKeyKvId
  key_type     = "RSA"
  key_size     = 2048

  key_opts = ["unwrapKey", "wrapKey"]
}

resource "azurerm_private_endpoint" "privateEndpointSqlSrv" {
  name                = "pe-${var.sqlServerName}-${var.environment}-sql"
  resource_group_name = var.resourceGroup.name
  location            = var.resourceGroup.location
  subnet_id           = var.privateEndpointSubnetId

  private_service_connection {
    name                           = "pe-${var.sqlServerName}-${var.environment}-sql-privateserviceconnection"
    private_connection_resource_id = var.privateLinkServiceId
    is_manual_connection           = false
  }
}

output.tf

Defines which resources and/or respective properties are accessible in the Terraform project where the module is called.

output "msSqlSrv" {
    description = "MS SQL Server Object created in module."
    value = azurerm_mssql_server.msSqlSrv
}

In the previous example, the platform administrator went from configuring the VNet, SQL Server, role assignments, and private endpoints manually in the web UI to adding them to the Terraform template for the target environment. Now, once the module is defined and ready to use, the platform administrator only needs to add one module call to the Terraform template for any future SQL Server deployment:

Considerations for Designing Shared Modules

A good module should come with clear, structured documentation. The input variables (i.e., the interface) as well as the module itself should be documented in a way that no handover from developer to consumer is necessary.
The module should be designed to be as flexible as possible. A general rule: if a property of the main resource is not restricted by security or company policies, it should be exposed as a module variable. This approach provides flexibility for module usage and hardens the interface against breaking changes.
Avoid feature bloat. While flexibility across different use cases was mentioned above as a design goal, a module can only be maintained if its scope is clearly defined.
Versioning is important. Once a module is used by multiple teams, changes to the module can potentially affect many applications. A Terraform module should not simply be a collection of Terraform code. It should be treated as a reusable infrastructure building block with a defined interface, purpose and lifecycle.

Terraform Modules as a Self-Service Capability

Switching from manual infrastructure deployments through the cloud provider’s web UI to using IaC is a major step towards standardization, not only between different environments but also between different teams and applications. Using modules as reusable building blocks can also solve an organizational problem: how can application teams request infrastructure without requiring the platform team to provision every requested change?

A possible solution is enabling the application teams to provision certain infrastructure parts themselves by providing a comprehensive catalog of Terraform modules. A module does not automatically have to become a self-service interface. An organization can continue to use certain modules only centrally within the platform team. More on that in the following section.

Let us revisit the example of the platform administrator who was asked to provision a SQL Server in an Azure subscription:

We went from provisioning and configuring all required resources in the web UI to defining them in a Terraform template, and finally to defining a standardized set of server configurations, role assignments, and networking setup, encapsulated in a custom module that is called in the template. The next step is to involve the requesting team directly: instead of creating a ticket for the platform team, the application team can create a pull request in the repository governing their subscription and include the module call with the desired specifications. The platform administrator can then review the pull request and approve it for deployment.

This approach reduces operational overhead for the platform team and alleviates potential bottlenecks for the application team. The platform team does not have to manually execute every infrastructure request, while the application team does not need to become an expert in the underlying infrastructure implementation.

Governance and Guardrails

While the self-service approach does reduce the operational overhead for the platform team over time, some points need to be considered in advance:

Defining clear boundaries for what can be governed by self-service and what shouldn’t

Setting up a SQL Server, a storage account, or a new container in an Azure subscription, as well as managing role assignments on these resources, are tasks that can be handled via self-service with a low risk of causing major issues. On the other hand, setting up a new subscription, setting a budget, integrating a new VNet into the existing network infrastructure, or managing a Microsoft Entra tenant are tasks for which a self-service approach is not advisable. Such requests should still be implemented by the platform team or the respective networking and security specialists.

Setting up guardrails for application teams

Before granting access to the subscription repositories and shared modules, a clear process, guardrails, and support workflows need to be defined. The following questions should have clear answers before the self-service project starts:

What rules and conventions are employed in the subscription repositories? How are they enforced?
How do you ensure that no changes are made to resources that are out of bounds for the self-service approach?
How do you prevent simultaneous deployments in the same subscription by multiple users?
Is there a clear process for application teams on how to contribute to the subscription repositories?
Are there ways to circumvent the outlined process? If so, how can they be prevented?
Is comprehensive documentation in place so that application teams can get started and understand the process on their own?

Module Repositories and Versioning

A central module repository can provide a consistent place for teams to discover and consume approved shared modules. For a smaller number of shared modules, this is a good approach. Each module can be contained in its own directory and can be referenced accordingly in the module call in the template. With a rising number of shared modules and collaborators, a switch to a dedicated repository per module is preferred.

The modules should follow a clearly defined lifecycle and be versioned accordingly. This ties directly to how shared modules are used: when a module is called in a Terraform template, a source for the module definition must be provided. This can be a directory, a repository branch, or a repository along with a tag. The latter option should always be chosen.

Conclusion

In this article, we have seen how managing infrastructure in a cloud environment has changed with the increasing adoption of Infrastructure as Code (with a focus on Terraform) and what possibilities it offers in terms of standardization and scalability. It is also important to note that the outlined progression from manual deployments in the web UI to a managed self-service platform with standardized modules for every use case is only one way to achieve scalability in infrastructure management, not a solution that fits every company. For smaller companies with only a few application teams, a small platform team maintaining the subscription and module repositories itself may be sufficient, while larger organizations will need to consider scalability.