The Anatomy of a Production Terraform Module

The Anatomy of a Production Terraform Module

As your infrastructure grows, copy-pasting resource definitions across different microservices or environments becomes unsustainable. If you have 20 services, and each requires a standard PostgreSQL instance, copying 100 lines of RDS code 20 times means that a simple upgrade (like moving from PostgreSQL 14 to 15) requires editing 20 separate files.

To solve this, we create Terraform Modules.

A module is a reusable container for multiple cloud resources that are used together. In this lesson, we will build a production-grade SQS Queue Module that automatically attaches a Dead Letter Queue (DLQ), retry logic, encryption, and CloudWatch alert boundaries.

Module Architecture Design

A professional module must be completely self-contained. It lives in its own directory (or separate Git repository) and follows a strict layout:

modules/sqs-queue/
├── main.tf        # Resources (SQS, DLQ, KMS, CloudWatch)
├── variables.tf   # Module Inputs with rigid validations
├── outputs.tf     # Module Outputs exposed to consumers
└── README.md      # Usage and integration instructions

Step 1: Define Inputs (variables.tf)

We must parameterize our SQS module so that different services can customize their queue names, visibility timeouts, and retention periods:

# modules/sqs-queue/variables.tf

variable "queue_name" {
  description = "Base name of the SQS queue"
  type        = string
}

variable "environment" {
  description = "Target deployment environment"
  type        = string
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

variable "visibility_timeout_seconds" {
  description = "Duration (in seconds) that received messages are hidden from consumers"
  type        = number
  default     = 30
}

variable "message_retention_seconds" {
  description = "Duration (in seconds) that SQS retains a message"
  type        = number
  default     = 345600 # 4 days
}

variable "max_receive_count" {
  description = "Number of times a message can be received before routing to the DLQ"
  type        = number
  default     = 5
}

Step 2: Implement the Resources (main.tf)

Inside the SQS module, we encapsulate SQS creation, Dead Letter Queue (DLQ) linking, and KMS customer-managed key encryption for maximum data-at-rest protection.

# modules/sqs-queue/main.tf

# 1. Create KMS Key for Queue Encryption
resource "aws_kms_key" "sqs" {
  description             = "KMS key for SQS Queue: ${var.queue_name}"
  deletion_window_in_days = 7
  enable_key_rotation     = true

  tags = {
    Name        = "kms-sqs-${var.queue_name}"
    Environment = var.environment
  }
}

# 2. Create the Dead Letter Queue (DLQ)
resource "aws_sqs_queue" "dlq" {
  name                      = "${var.queue_name}-dlq-${var.environment}"
  message_retention_seconds = 1209600 # 14 days (long retention for debugging)
  
  kms_master_key_id                 = aws_kms_key.sqs.id
  kms_data_key_reuse_period_seconds = 300

  tags = {
    Name        = "${var.queue_name}-dlq-${var.environment}"
    Environment = var.environment
  }
}

# 3. Create the Primary SQS Queue
resource "aws_sqs_queue" "primary" {
  name                       = "${var.queue_name}-${var.environment}"
  visibility_timeout_seconds = var.visibility_timeout_seconds
  message_retention_seconds  = var.message_retention_seconds
  
  kms_master_key_id                 = aws_kms_key.sqs.id
  kms_data_key_reuse_period_seconds = 300

  # Redrive policy linking SQS to the DLQ
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = var.max_receive_count
  })

  tags = {
    Name        = "${var.queue_name}-${var.environment}"
    Environment = var.environment
  }
}

# 4. Automate Alert Boundary: Alarm on DLQ depth > 0
resource "aws_cloudwatch_metric_alarm" "dlq_not_empty" {
  alarm_name          = "sqs-${var.queue_name}-dlq-not-empty-${var.environment}"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "1"
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = "60"
  statistic           = "Sum"
  threshold           = "0" # Trigger alarm if even 1 message lands in DLQ
  alarm_description   = "Alarm when messages land in SQS DLQ: ${var.queue_name}"

  dimensions = {
    QueueName = aws_sqs_queue.dlq.name
  }
}

Step 3: Expose Outputs (outputs.tf)

Outputs allow consumer modules to reference SQS identifiers (like the SQS ARN or SQS URL) to pass them into application containers:

# modules/sqs-queue/outputs.tf

output "queue_url" {
  value       = aws_sqs_queue.primary.url
  description = "The URL of the primary SQS Queue"
}

output "queue_arn" {
  value       = aws_sqs_queue.primary.arn
  description = "The ARN of the primary SQS Queue"
}

output "dlq_url" {
  value       = aws_sqs_queue.dlq.url
  description = "The URL of the SQS Dead Letter Queue"
}

Step 4: Consume the Module in Your Application

Now, backend developers don't have to deal with SQS DLQ setups, encryption KMS policies, or CloudWatch alarms. They simply declare their dependency on your module:

# main.tf

module "order_processing_queue" {
  source = "./modules/sqs-queue"

  queue_name                 = "order-processing"
  environment                = "prod"
  visibility_timeout_seconds = 60
  max_receive_count          = 3
}

# Pass output directly into app task definition
output "queue_endpoint" {
  value = module.order_processing_queue.queue_url
}

By designing modules with rigorous defaults, you encapsulate complex, secure patterns once and scale them across your entire engineering organization with zero code repetition.

Next Steps

Now that we understand module anatomy, we must tackle Secrets Management. In the next lesson, we'll learn how to handle sensitive database credentials, passwords, and API keys securely, preventing leakage inside state files or repository branches.