Terraform in Production (Part 3): The Module Dilemma — Encapsulation vs. Over-Engineering

Every infrastructure team goes through the exact same rite of passage: "Let’s build a shared internal Terraform module library to standardize our cloud architecture and keep our code 100% DRY!"

It starts innocently enough. You wrap an S3 bucket or a VPC in a custom module with nice sensible defaults.

Six months later, you find yourself trapped in a labyrinth of 70 passthrough variables, diamond dependency conflicts, and dreading every provider major version bump.

Here is an honest reflection on Terraform module encapsulation, versioning friction, and why heavyweight modules often create more problems than they solve.

1. The Leaky Abstraction Trap

In standard software engineering (Python, Go, TypeScript), functions and classes excel at abstraction because you encapsulate logic and hide internal state:

// Software abstraction: easy to hide implementation logic
function calculatePricing(plan: PlanType, usage: number): number {
  return computeFormula(plan, usage); // Caller never needs to know internal details
}

Infrastructure as Code, however, is declarative state management, not procedural logic. When you abstract a cloud resource inside a module, you aren't hiding procedural algorithms—you are hiding resource arguments.

The minute Team B wants to deploy your "Standard S3 Bucket Module" but needs a custom lifecycle rule, specific CORS configuration, or a customer-managed KMS key, the abstraction shatters:

# The slippery slope of passthrough variables
variable "enable_cors" {
  type    = bool
  default = false
}

variable "cors_rules" {
  type    = any
  default = []
}

variable "lifecycle_rules" {
  type    = any
  default = []
}

variable "custom_kms_key_arn" {
  type    = string
  default = null
}

Within a year, your "reusable" module has 65 variables, 14 conditional dynamic blocks, and more boilerplate than the original resource declaration!

# The "God Module" Anti-Pattern: wrapper hell
resource "aws_s3_bucket_cors_configuration" "this" {
  count  = var.enable_cors && length(var.cors_rules) > 0 ? 1 : 0
  bucket = aws_s3_bucket.this.id

  dynamic "cors_rule" {
    for_each = var.cors_rules
    content {
      allowed_headers = try(cors_rule.value.allowed_headers, null)
      allowed_methods = cors_rule.value.allowed_methods
      allowed_origins = cors_rule.value.allowed_origins
      expose_headers  = try(cors_rule.value.expose_headers, null)
      max_age_seconds = try(cors_rule.value.max_age_seconds, null)
    }
  }
}

At that point, you haven't abstracted AWS. You have simply invented a slower, buggier, undocumented dialect of AWS.

2. The Versioning Minefield

How do you distribute modules across multiple projects or repositories?

module "vpc" {
  source = "git::https://github.com/my-org/terraform-modules.git//vpc?ref=v2.4.1"

  cidr_block = "10.0.0.0/16"
}

Pinning to Git tags (?ref=v2.4.1) is standard practice, but it introduces subtle maintenance bottlenecks:

Diamond Dependencies Across Providers

Suppose module-vpc v2 requires aws provider ~> 5.0, but your root module or a sibling module still relies on aws provider ~> 4.60 because of an unrelated deprecated resource. Terraform cannot load multiple provider versions in the same configuration. You are blocked from upgrading your network module until all other components are refactored.

The Upstream Bug Cascade

When HashiCorp introduces a breaking change or deprecates an attribute in a provider, a single bug in your shared module blocks 40 consumer microservices from running terraform apply.

3. When SHOULD You Write a Module?

Modules are not inherently evil; the problem is treating them as the default unit of code organization.

✅ When Modules Make Sense:

1. Multi-Resource Composition: When a single logical component always requires 3+ coupled resources that should never exist independently (e.g., an S3 bucket + CloudFront origin access control + Route53 alias + ACM certificate).
2. Repetition Within the Same Stack: When you need to instantiate identical worker queues or isolated tenant namespaces within a single root module using for_each:

   module "tenant_storage" {
     for_each  = toset(["customer-alpha", "customer-beta", "customer-gamma"])
     source    = "./modules/tenant-partition"
     tenant_id = each.key
   }
   

3. Firm Organizational Guardrails: Enforcing mandatory corporate security policies (e.g., forced KMS customer keys, automated flow logs, mandatory compliance tagging).

❌ When to Avoid Modules:

1. Thin 1:1 Wrappers: Wrapping a single aws_s3_bucket or cloudflare_record just to add 2 default tags. Use provider-level default_tags instead!
2. Premature Cross-Team Standardization: Trying to build a universal "all-in-one Kubernetes cluster" module for 10 teams with different needs before patterns stabilize.
3. Deep Nesting: Modules calling modules calling modules. Debugging error messages 4 levels deep in HCL is an exercise in pain.

[!TIP] Remember Sandi Metz’s Rule:
"Duplication is far cheaper than the wrong abstraction."
In infrastructure, copying and pasting 15 lines of explicit HCL across two stacks is vastly safer and easier to maintain than binding your entire company to a 500-line fragile wrapper module.

4. What's Next

Now that we know when to modularize and when to keep code flat, what happens when you need to refactor existing resources without destroying them?

In Part 4, we dive into Declarative State Surgery with moved and import blocks.

👈 Part 2: Air-Gapped CI/CD with GitHub Actions & Private Runners
👉 Read Part 4: Declarative State Surgery with moved and import Blocks →