Secrets Management: AWS Secrets Manager & KMS Integration

One of the most common causes of corporate data breaches is the hardcoding of API keys, database passwords, and TLS private keys inside source code. In a Terraform environment, there are two distinct vectors where secrets can leak:

Source Code: Committing plaintext variables or .tfvars files to Git.
State Files: Terraform state files (terraform.tfstate) store all outputs, resources, and credentials in plaintext.

To secure our infrastructure, we must separate Infra Configuration from Runtime Secrets. We achieve this by provisioning credentials through AWS Secrets Manager and encrypting data using customer-managed KMS (Key Management Service) keys.

[ Developer / CI ] ── Provisions placeholder ──> [ AWS Secrets Manager ]
                                                        │ (Encrypted with KMS)
                                                        ▼
[ App container (ECS Fargate) ] ── Fetches secret ─────> [ Decrypted Password ]
                                   at launch (IAM gated)

Step 1: Provision a KMS Customer-Managed Key

We avoid using the default AWS SQS or Secrets Manager service keys because we cannot customize access policies, rotation schedules, or audits on default keys. We create our own:

# modules/secrets/main.tf

# 1. Create a Customer-Managed Key (CMK)
resource "aws_kms_key" "secrets" {
  description             = "KMS Key for Sensitive Secrets Manager Credentials"
  deletion_window_in_days = 7
  enable_key_rotation     = true # Automatic annual rotation

  tags = {
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# 2. Define a clean alias for the key
resource "aws_kms_alias" "secrets_alias" {
  name          = "alias/app-secrets-${var.environment}"
  target_key_id = aws_kms_key.secrets.key_id
}

Step 2: Provision an AWS Secrets Manager Shell

We will create a Secrets Manager container to house database credentials, but we will not hardcode the password value in HCL. Instead, we provision a "shell" secret, and allow engineers to populate the value manually in the AWS Console once, or dynamically inject it at runtime:

# modules/secrets/main.tf (continued)

# Create the Secret Container
resource "aws_secretsmanager_secret" "db_credentials" {
  name                    = "${var.environment}-database-credentials"
  description             = "Production RDS PostgreSQL Credentials"
  kms_key_id              = aws_kms_key.secrets.arn # Encrypt using our custom KMS key
  recovery_window_in_days = 7                       # Prevent immediate deletion

  tags = {
    Environment = var.environment
  }
}

# Provision a default JSON structure as a placeholder
resource "aws_secretsmanager_secret_version" "db_credentials_placeholder" {
  secret_id     = aws_secretsmanager_secret.db_credentials.id
  secret_string = jsonencode({
    username = "admin"
    password = "CHANGE_ME_IN_AWS_CONSOLE" # Placeholder to be overridden securely
  })
  
  # Prevent Terraform from resetting manual console password updates on subsequent applies!
  lifecycle {
    ignore_changes = [secret_string]
  }
}

By adding ignore_changes = [secret_string], developers can securely log into the AWS Console once, change the database password to a cryptographically random 32-character string, and subsequent terraform apply commands will not overwrite or expose that password in the state file.

Step 3: Fetching Secrets at Runtime inside ECS Tasks

When deploying application containers (e.g. inside AWS ECS Fargate), you should never pass the database password as an environment variable in plaintext. Instead, you reference the Secrets Manager ARN directly in the container definition, allowing ECS to securely retrieve and inject the secret at container launch:

# Task Definition Container Environment block:
"secrets": [
  {
    "name": "DB_PASSWORD",
    "valueFrom": "${aws_secretsmanager_secret.db_credentials.arn}:password::"
  }
]

To authorize this retrieval, we attach a strict least-privilege IAM policy to the ECS Task Execution Role:

# Allow ECS Task Execution to read database secret
data "aws_iam_policy_document" "ecs_secrets_policy" {
  statement {
    effect    = "Allow"
    actions   = ["secretsmanager:GetSecretValue"]
    resources = [aws_secretsmanager_secret.db_credentials.arn]
  }

  statement {
    effect    = "Allow"
    actions   = ["kms:Decrypt"]
    resources = [aws_kms_key.secrets.arn]
  }
}

Through this architecture, your secrets are encrypted at rest, audited dynamically in AWS CloudTrail, and never committed to source control or exposed in local environments.

Next Steps

Now that we have established a secure network, passwordless CI/CD authentication, and professional secrets management, we are ready to move to Module 4: AWS Compute & Storage Provisioning. In the next lesson, we will provision a highly available, Multi-AZ relational database using AWS RDS PostgreSQL.

Secrets Management: AWS Secrets Manager & KMS Integration

Provision, secure, and automate production-grade cloud infrastructure at scale.