Terraform in Production (Part 2): Air-Gapped CI/CD with GitHub Actions & Private Runners

In Part 1, we explored why Terraform's multi-provider ecosystem is the definitive way to orchestrate modern infrastructure.

Now, let’s confront one of the most glaring security vulnerabilities in modern DevOps: running terraform apply from public GitHub-hosted runners or directly from developer laptops.

1. The Public Runner Security Anti-Pattern

Public GitHub Actions runners live on Microsoft Azure’s public IP ranges. If your production environment is properly architected, your critical components:

• RDS / Aurora database clusters
• ElastiCache Redis clusters
• Kubernetes (EKS/GKE) private API control planes
• Internal HashiCorp Vault clusters

...reside strictly inside private subnets with zero public ingress.

If you attempt to run Terraform from public GitHub runners, you are backed into a corner:

1. Punching holes in your VPC security groups to allow traffic from dynamic, ever-changing public IP ranges (a massive security hazard).
2. Setting up ephemeral bastion proxies or complex VPN tunnels with credentials stored in GitHub.
3. Storing long-lived AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in GitHub repository secrets (keys that can be exfiltrated or abused).

2. The Production Architecture: Private Runners + OIDC

The battle-tested, zero-trust architecture relies on two primitives:

1. GitHub OpenID Connect (OIDC): Eliminates static cloud keys completely. GitHub mints a short-lived cryptographically signed JWT token that your cloud IAM provider verifies directly.
2. Ephemeral Self-Hosted Runners in Private Subnets: Run your GitHub Actions runners inside your private VPC (using Kubernetes with Actions Runner Controller (ARC) or auto-scaling EC2 instances).

flowchart TD
    subgraph GitHub Cloud
        PR[Developer Pull Request] --> GHA[GitHub Actions Workflow]
        GHA --> OIDC[GitHub OIDC Token]
    end

    subgraph Corporate Private VPC
        direction TB
        Runner[Ephemeral Self-Hosted Runner<br/>ARC / Private EC2]
        Vault[Internal HashiCorp Vault]
        EKS[Private EKS / API]
        RDS[Private RDS Database]
        
        Runner -->|Outbound HTTPS 443 only| GHA
        Runner -->|Direct Private Peering| Vault
        Runner -->|Direct Private Access| EKS
        Runner -->|Direct Private Access| RDS
    end

    subgraph Cloud IAM
        IAM[AWS IAM / Cloud Identity]
        OIDC -->|Federated AssumeRoleWithWebIdentity| IAM
        IAM -.->|Short-Lived Token| Runner
    end

Why This Architecture Wins:

• Zero Inbound Ports: Self-hosted runners communicate with GitHub via outbound HTTPS (port 443) long-polling. You do not open a single inbound firewall port into your network!
• Direct Private Reachability: Because the runner resides inside your private VPC, it has native, high-speed, private network routing to your internal databases, Kubernetes clusters, and Vault instances.
• Zero Static Credentials: Authentication relies on cryptographic OIDC claims tied strictly to your repository, branch, and environment.

3. Production GitHub Actions Workflow

Here is a hardened GitHub Actions workflow executing strictly on private self-hosted runners:

# .github/workflows/terraform.yml
name: "Terraform Production Deployment"

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  terraform:
    name: "Terraform Pipeline"
    # Execute strictly on private self-hosted runners inside the VPC
    runs-on: [ self-hosted, linux, private-vpc ]
    
    permissions:
      id-token: write # Required for GitHub OIDC
      contents: read
      pull-requests: write

    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Authenticate to AWS via OIDC (No Static Keys!)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-terraform-pipeline
          role-session-name: gha-tf-${{ github.run_id }}
          aws-region: eu-west-1

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.5

      - name: Terraform Init
        run: terraform init

      - name: Terraform Plan
        id: plan
        if: github.event_name == 'pull_request'
        run: terraform plan -no-color -out=tfplan
        
      - name: Terraform Apply (Main Branch Only)
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        run: terraform apply -auto-approve tfplan

The Pull Request Safety Net

In pull requests, the workflow generates a plan and posts the summary directly to the PR comments. Team members can review the exact diff before any changes are merged.

When merged to main, the runner executes terraform apply automatically in the same private execution context.

4. Summary & What's Next

By combining GitHub OIDC with ephemeral self-hosted runners in private subnets:

• No static credentials exist anywhere in GitHub Secrets.
• No public ingress is opened to your VPC.
• The control plane is completely air-gapped from the public internet.

In Part 3, we tackle the most controversial topic in Terraform: The Module Dilemma — when modules create more problems than they solve, and how to avoid the "God Module" trap.

👈 Part 1: Why HCL Beats Cloud-Native IaC
👉 Read Part 3: The Module Dilemma — Encapsulation vs. Over-Engineering →