703 lines
14 KiB
Markdown
703 lines
14 KiB
Markdown
# Secrets Management Patterns
|
|
|
|
This document outlines the secure patterns for managing secrets in GitLab stack projects.
|
|
|
|
## Table of Contents
|
|
1. [Core Security Principles](#core-security-principles)
|
|
2. [Directory and File Structure](#directory-and-file-structure)
|
|
3. [Docker Secrets Integration](#docker-secrets-integration)
|
|
4. [Secret Detection Patterns](#secret-detection-patterns)
|
|
5. [Migration Patterns](#migration-patterns)
|
|
6. [Common Secret Types](#common-secret-types)
|
|
|
|
---
|
|
|
|
## Core Security Principles
|
|
|
|
### The Golden Rules
|
|
|
|
**NEVER**:
|
|
- ❌ Put secrets in .env files
|
|
- ❌ Put secrets in docker-compose.yml environment variables
|
|
- ❌ Commit secrets to git
|
|
- ❌ Use world-readable permissions
|
|
- ❌ Store secrets as root-owned files
|
|
- ❌ Hardcode secrets in application code
|
|
- ❌ Log secret values
|
|
- ❌ Pass secrets via command-line arguments
|
|
|
|
**ALWAYS**:
|
|
- ✅ Use Docker secrets mechanism
|
|
- ✅ Store secret files in ./secrets directory
|
|
- ✅ Set 700 permissions on ./secrets directory
|
|
- ✅ Set 600 permissions on secret files
|
|
- ✅ Add ./secrets/* to .gitignore
|
|
- ✅ Use cryptographically secure random generation
|
|
- ✅ Rotate secrets regularly
|
|
- ✅ Audit secret usage
|
|
|
|
---
|
|
|
|
## Directory and File Structure
|
|
|
|
### Standard ./secrets Directory Layout
|
|
|
|
```
|
|
./secrets/
|
|
├── .gitkeep # Only file tracked by git
|
|
├── db_password # Database password
|
|
├── db_root_password # Database root password
|
|
├── api_key # External API key
|
|
├── jwt_secret # JWT signing secret
|
|
├── oauth_client_secret # OAuth secret
|
|
├── smtp_password # Email password
|
|
├── encryption_key # Application encryption key
|
|
└── ssl/
|
|
├── cert.pem # SSL certificate
|
|
└── key.pem # SSL private key
|
|
```
|
|
|
|
### Permissions Reference
|
|
|
|
```bash
|
|
# Directory permissions
|
|
drwx------ ./secrets/ # 700 (owner only)
|
|
|
|
# File permissions
|
|
-rw------- db_password # 600 (owner read/write)
|
|
-rw------- api_key # 600
|
|
-rw------- jwt_secret # 600
|
|
|
|
# Ownership
|
|
user:user all files and directories # NOT root
|
|
```
|
|
|
|
### Setting Up Proper Permissions
|
|
|
|
```bash
|
|
# Create secrets directory
|
|
mkdir -p ./secrets
|
|
chmod 700 ./secrets
|
|
|
|
# Create secret file
|
|
echo -n "secret-value" > ./secrets/db_password
|
|
chmod 600 ./secrets/db_password
|
|
|
|
# Verify permissions
|
|
ls -la ./secrets/
|
|
# Expected: drwx------ ... ./secrets/
|
|
# Expected: -rw------- ... db_password
|
|
|
|
# Fix ownership if root-owned
|
|
sudo chown -R $(id -u):$(id -g) ./secrets/
|
|
```
|
|
|
|
---
|
|
|
|
## Docker Secrets Integration
|
|
|
|
### Top-Level Secrets Definition
|
|
|
|
**File-based secrets** (preferred for development/single-host):
|
|
|
|
```yaml
|
|
secrets:
|
|
db_password:
|
|
file: ./secrets/db_password
|
|
|
|
api_key:
|
|
file: ./secrets/api_key
|
|
|
|
jwt_secret:
|
|
file: ./secrets/jwt_secret
|
|
|
|
# SSL certificates
|
|
ssl_cert:
|
|
file: ./secrets/ssl/cert.pem
|
|
|
|
ssl_key:
|
|
file: ./secrets/ssl/key.pem
|
|
```
|
|
|
|
**External secrets** (for production/swarm):
|
|
|
|
```yaml
|
|
secrets:
|
|
db_password:
|
|
external: true
|
|
name: prod_db_password
|
|
|
|
api_key:
|
|
external: true
|
|
name: prod_api_key_v2
|
|
```
|
|
|
|
### Service Secret Usage
|
|
|
|
**Basic usage**:
|
|
|
|
```yaml
|
|
services:
|
|
app:
|
|
image: myapp:latest
|
|
secrets:
|
|
- db_password
|
|
- api_key
|
|
- jwt_secret
|
|
# Secrets mounted at /run/secrets/secret_name
|
|
```
|
|
|
|
**Containers with native Docker secrets support**:
|
|
|
|
```yaml
|
|
services:
|
|
postgres:
|
|
image: postgres:16-alpine
|
|
secrets:
|
|
- db_password
|
|
environment:
|
|
POSTGRES_DB: myapp
|
|
POSTGRES_USER: appuser
|
|
# Use _FILE suffix to point to secret
|
|
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
|
|
```
|
|
|
|
**Supported containers with _FILE suffix**:
|
|
- PostgreSQL: `POSTGRES_PASSWORD_FILE`
|
|
- MySQL/MariaDB: `MYSQL_ROOT_PASSWORD_FILE`, `MYSQL_PASSWORD_FILE`
|
|
- MongoDB: Various `_FILE` variables
|
|
- Redis: Configuration file can read from secret path
|
|
|
|
**Containers requiring docker-entrypoint.sh**:
|
|
|
|
```yaml
|
|
services:
|
|
custom_app:
|
|
image: myapp:latest
|
|
entrypoint: /docker-entrypoint.sh
|
|
command: ["npm", "start"]
|
|
volumes:
|
|
- ./docker-entrypoint.sh:/docker-entrypoint.sh:ro
|
|
secrets:
|
|
- api_key
|
|
- jwt_secret
|
|
# docker-entrypoint.sh loads secrets into environment
|
|
```
|
|
|
|
---
|
|
|
|
## Secret Detection Patterns
|
|
|
|
### Patterns That Indicate Secrets
|
|
|
|
**Variable Name Patterns** (case-insensitive):
|
|
|
|
```regex
|
|
.*PASSWORD.*
|
|
.*SECRET.*
|
|
.*KEY.*
|
|
.*TOKEN.*
|
|
.*API.*
|
|
.*AUTH.*
|
|
.*CREDENTIAL.*
|
|
.*PRIVATE.*
|
|
.*CERT.*
|
|
```
|
|
|
|
**Value Patterns**:
|
|
|
|
```regex
|
|
# Base64-encoded (long strings)
|
|
^[A-Za-z0-9+/]{40,}={0,2}$
|
|
|
|
# Hex strings (64+ chars)
|
|
^[a-f0-9]{64,}$
|
|
|
|
# JWT tokens
|
|
^eyJ[A-Za-z0-9-_]+\.[A-Za-z0-9-_]+\.[A-Za-z0-9-_]+$
|
|
|
|
# API keys (common formats)
|
|
^sk_live_[A-Za-z0-9]{24,}$
|
|
^pk_live_[A-Za-z0-9]{24,}$
|
|
|
|
# UUID format
|
|
^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$
|
|
```
|
|
|
|
### Scanning .env for Secrets
|
|
|
|
**Examples of secrets in .env** (BAD):
|
|
|
|
```bash
|
|
# ❌ BAD - These are secrets and should NOT be in .env
|
|
DB_PASSWORD=supersecret123
|
|
API_KEY=sk_live_abc123xyz789
|
|
JWT_SECRET=my-super-secret-jwt-key
|
|
STRIPE_SECRET_KEY=sk_test_abc123
|
|
OAUTH_CLIENT_SECRET=oauth-secret-123
|
|
ENCRYPTION_KEY=aes256-key-here
|
|
ADMIN_PASSWORD=admin123
|
|
SMTP_PASSWORD=email-password
|
|
PRIVATE_KEY=-----BEGIN PRIVATE KEY-----
|
|
```
|
|
|
|
**What SHOULD be in .env** (GOOD):
|
|
|
|
```bash
|
|
# ✅ GOOD - Non-secret configuration
|
|
APP_NAME=myapp
|
|
APP_ENV=production
|
|
APP_DEBUG=false
|
|
APP_URL=https://example.com
|
|
|
|
# Database connection (NOT credentials)
|
|
DB_HOST=postgres
|
|
DB_PORT=5432
|
|
DB_NAME=myapp_production
|
|
DB_USER=appuser
|
|
# DB_PASSWORD is in ./secrets/db_password
|
|
|
|
# Redis
|
|
REDIS_HOST=redis
|
|
REDIS_PORT=6379
|
|
|
|
# Ports
|
|
WEB_PORT=80
|
|
API_PORT=8080
|
|
|
|
# Feature flags
|
|
ENABLE_CACHING=true
|
|
ENABLE_LOGGING=true
|
|
```
|
|
|
|
### Scanning docker-compose.yml for Secrets
|
|
|
|
**Bad patterns** (secrets in environment):
|
|
|
|
```yaml
|
|
# ❌ BAD - Secrets in environment variables
|
|
services:
|
|
app:
|
|
environment:
|
|
DB_PASSWORD: supersecret123 # ❌ CRITICAL
|
|
API_KEY: sk_live_abc123 # ❌ CRITICAL
|
|
JWT_SECRET: my-jwt-secret # ❌ CRITICAL
|
|
|
|
postgres:
|
|
environment:
|
|
POSTGRES_PASSWORD: dbpassword123 # ❌ CRITICAL
|
|
```
|
|
|
|
**Good patterns** (using Docker secrets):
|
|
|
|
```yaml
|
|
# ✅ GOOD - Using Docker secrets
|
|
services:
|
|
app:
|
|
secrets:
|
|
- db_password
|
|
- api_key
|
|
- jwt_secret
|
|
environment:
|
|
# Only non-secret config in environment
|
|
DB_HOST: postgres
|
|
DB_PORT: 5432
|
|
DB_NAME: myapp
|
|
|
|
postgres:
|
|
secrets:
|
|
- db_password
|
|
environment:
|
|
POSTGRES_DB: myapp
|
|
POSTGRES_USER: appuser
|
|
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
|
|
|
|
secrets:
|
|
db_password:
|
|
file: ./secrets/db_password
|
|
api_key:
|
|
file: ./secrets/api_key
|
|
jwt_secret:
|
|
file: ./secrets/jwt_secret
|
|
```
|
|
|
|
---
|
|
|
|
## Migration Patterns
|
|
|
|
### Pattern 1: Migrate from .env to Docker Secrets
|
|
|
|
**Before** (.env file):
|
|
|
|
```bash
|
|
DB_PASSWORD=mysecretpass
|
|
API_KEY=sk_live_abc123xyz
|
|
JWT_SECRET=my-jwt-secret-key
|
|
```
|
|
|
|
**Migration steps**:
|
|
|
|
```bash
|
|
# 1. Create secret files
|
|
echo -n "mysecretpass" > ./secrets/db_password
|
|
echo -n "sk_live_abc123xyz" > ./secrets/api_key
|
|
echo -n "my-jwt-secret-key" > ./secrets/jwt_secret
|
|
|
|
# 2. Set permissions
|
|
chmod 600 ./secrets/*
|
|
|
|
# 3. Remove from .env
|
|
sed -i '/DB_PASSWORD=/d' .env
|
|
sed -i '/API_KEY=/d' .env
|
|
sed -i '/JWT_SECRET=/d' .env
|
|
```
|
|
|
|
**After** (.env file):
|
|
|
|
```bash
|
|
# Secrets moved to Docker secrets in ./secrets/
|
|
# DB_PASSWORD: ./secrets/db_password
|
|
# API_KEY: ./secrets/api_key
|
|
# JWT_SECRET: ./secrets/jwt_secret
|
|
```
|
|
|
|
### Pattern 2: Migrate from docker-compose.yml environment to Secrets
|
|
|
|
**Before**:
|
|
|
|
```yaml
|
|
services:
|
|
app:
|
|
environment:
|
|
DB_HOST: postgres
|
|
DB_PASSWORD: supersecret123 # ❌ Secret in compose
|
|
API_KEY: sk_live_abc123 # ❌ Secret in compose
|
|
```
|
|
|
|
**Migration**:
|
|
|
|
```bash
|
|
# Extract values and create secret files
|
|
echo -n "supersecret123" > ./secrets/db_password
|
|
echo -n "sk_live_abc123" > ./secrets/api_key
|
|
chmod 600 ./secrets/*
|
|
```
|
|
|
|
**After**:
|
|
|
|
```yaml
|
|
services:
|
|
app:
|
|
secrets:
|
|
- db_password
|
|
- api_key
|
|
environment:
|
|
DB_HOST: postgres
|
|
# Secrets loaded from /run/secrets/
|
|
|
|
secrets:
|
|
db_password:
|
|
file: ./secrets/db_password
|
|
api_key:
|
|
file: ./secrets/api_key
|
|
```
|
|
|
|
### Pattern 3: Create docker-entrypoint.sh for Legacy Containers
|
|
|
|
When container doesn't support `_FILE` variables:
|
|
|
|
**docker-entrypoint.sh**:
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
set -e
|
|
|
|
# Function to load secret into environment variable
|
|
load_secret() {
|
|
local secret_name=$1
|
|
local env_var=$2
|
|
local secret_file="/run/secrets/${secret_name}"
|
|
|
|
if [ -f "$secret_file" ]; then
|
|
export "${env_var}=$(cat "$secret_file")"
|
|
echo "✓ Loaded secret: $secret_name -> $env_var"
|
|
else
|
|
echo "✗ ERROR: Secret file not found: $secret_file" >&2
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
# Load all required secrets
|
|
load_secret "db_password" "DB_PASSWORD"
|
|
load_secret "api_key" "API_KEY"
|
|
load_secret "jwt_secret" "JWT_SECRET"
|
|
|
|
# Execute the original command
|
|
exec "$@"
|
|
```
|
|
|
|
**docker-compose.yml**:
|
|
|
|
```yaml
|
|
services:
|
|
legacy_app:
|
|
image: legacy-app:latest
|
|
entrypoint: /docker-entrypoint.sh
|
|
command: ["node", "server.js"]
|
|
volumes:
|
|
- ./docker-entrypoint.sh:/docker-entrypoint.sh:ro
|
|
secrets:
|
|
- db_password
|
|
- api_key
|
|
- jwt_secret
|
|
```
|
|
|
|
**Set permissions**:
|
|
|
|
```bash
|
|
chmod +x docker-entrypoint.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Common Secret Types
|
|
|
|
### 1. Database Passwords
|
|
|
|
**Generate**:
|
|
```bash
|
|
openssl rand -base64 32 | tr -d '/+=' | head -c 32 > ./secrets/db_password
|
|
chmod 600 ./secrets/db_password
|
|
```
|
|
|
|
**Use with PostgreSQL**:
|
|
```yaml
|
|
services:
|
|
postgres:
|
|
image: postgres:16-alpine
|
|
secrets:
|
|
- db_password
|
|
environment:
|
|
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
|
|
```
|
|
|
|
### 2. API Keys
|
|
|
|
**Generate**:
|
|
```bash
|
|
openssl rand -hex 32 > ./secrets/api_key
|
|
chmod 600 ./secrets/api_key
|
|
```
|
|
|
|
**Format**: 64 hex characters
|
|
|
|
### 3. JWT Secrets
|
|
|
|
**Generate**:
|
|
```bash
|
|
openssl rand -base64 64 > ./secrets/jwt_secret
|
|
chmod 600 ./secrets/jwt_secret
|
|
```
|
|
|
|
**Format**: Base64-encoded, 64+ characters
|
|
|
|
### 4. Encryption Keys
|
|
|
|
**Generate AES-256 key**:
|
|
```bash
|
|
openssl rand -hex 32 > ./secrets/encryption_key
|
|
chmod 600 ./secrets/encryption_key
|
|
```
|
|
|
|
**Format**: 32 bytes hex (256-bit)
|
|
|
|
### 5. Session Secrets
|
|
|
|
**Generate**:
|
|
```bash
|
|
openssl rand -base64 32 > ./secrets/session_secret
|
|
chmod 600 ./secrets/session_secret
|
|
```
|
|
|
|
### 6. OAuth Client Secrets
|
|
|
|
**Usually provided by OAuth provider**, store securely:
|
|
```bash
|
|
echo -n "provider-given-secret" > ./secrets/oauth_client_secret
|
|
chmod 600 ./secrets/oauth_client_secret
|
|
```
|
|
|
|
### 7. SSL/TLS Certificates and Keys
|
|
|
|
**Store certificate and key separately**:
|
|
|
|
```bash
|
|
# Certificate (can be less restrictive)
|
|
cp cert.pem ./secrets/ssl/cert.pem
|
|
chmod 644 ./secrets/ssl/cert.pem
|
|
|
|
# Private key (must be restrictive)
|
|
cp key.pem ./secrets/ssl/key.pem
|
|
chmod 600 ./secrets/ssl/key.pem
|
|
```
|
|
|
|
**Use in compose**:
|
|
```yaml
|
|
secrets:
|
|
ssl_cert:
|
|
file: ./secrets/ssl/cert.pem
|
|
ssl_key:
|
|
file: ./secrets/ssl/key.pem
|
|
|
|
services:
|
|
nginx:
|
|
secrets:
|
|
- ssl_cert
|
|
- ssl_key
|
|
# Mounted at /run/secrets/ssl_cert and /run/secrets/ssl_key
|
|
```
|
|
|
|
---
|
|
|
|
## Git Protection Patterns
|
|
|
|
### .gitignore Configuration
|
|
|
|
**Comprehensive .gitignore**:
|
|
|
|
```gitignore
|
|
# Secrets directory - NEVER commit
|
|
/secrets/
|
|
/secrets/*
|
|
|
|
# Allow only .gitkeep
|
|
!secrets/.gitkeep
|
|
|
|
# Backup files
|
|
*.old
|
|
*.backup
|
|
*.bak
|
|
*~
|
|
|
|
# Environment files (may contain secrets)
|
|
.env
|
|
.env.local
|
|
.env.*.local
|
|
.env.production
|
|
|
|
# Common secret file patterns
|
|
*password*.txt
|
|
*secret*.txt
|
|
*key*.txt
|
|
*token*.txt
|
|
*credential*.txt
|
|
|
|
# SSL/TLS
|
|
*.pem
|
|
*.key
|
|
*.crt
|
|
*.p12
|
|
*.pfx
|
|
|
|
# SSH keys
|
|
id_rsa
|
|
id_ed25519
|
|
*.ppk
|
|
```
|
|
|
|
### Checking Git Status
|
|
|
|
**Verify secrets aren't staged**:
|
|
```bash
|
|
# Check for secrets in staging
|
|
git status | grep secrets/
|
|
|
|
# Should only show .gitkeep if anything
|
|
# If other files shown, they're staged (BAD!)
|
|
```
|
|
|
|
**Check git history**:
|
|
```bash
|
|
# Search for secrets in history
|
|
git log --all --full-history -- ./secrets/
|
|
|
|
# Search for specific patterns
|
|
git log -p --all -S "password"
|
|
git log -p --all -S "secret"
|
|
```
|
|
|
|
**Remove secrets from git history** (if committed):
|
|
```bash
|
|
# Using git-filter-repo (recommended)
|
|
git filter-repo --path secrets/ --invert-paths
|
|
|
|
# Or BFG Repo-Cleaner
|
|
bfg --delete-folders secrets
|
|
```
|
|
|
|
---
|
|
|
|
## Secret Rotation Patterns
|
|
|
|
### Safe Rotation Procedure
|
|
|
|
```bash
|
|
# 1. Backup current secret
|
|
cp ./secrets/api_key ./secrets/api_key.$(date +%Y%m%d).old
|
|
|
|
# 2. Generate new secret
|
|
openssl rand -hex 32 > ./secrets/api_key
|
|
|
|
# 3. Test with new secret
|
|
docker compose up -d
|
|
docker compose logs app # Check for errors
|
|
|
|
# 4. If successful, remove old backup after grace period
|
|
# rm ./secrets/api_key.*.old
|
|
```
|
|
|
|
### Rotation Tracking
|
|
|
|
**Create .secrets/metadata.yml** (not tracked):
|
|
|
|
```yaml
|
|
db_password:
|
|
created: 2025-01-15
|
|
rotated: 2025-10-20
|
|
rotation_interval_days: 90
|
|
|
|
api_key:
|
|
created: 2025-01-15
|
|
rotated: 2025-10-20
|
|
rotation_interval_days: 90
|
|
```
|
|
|
|
---
|
|
|
|
## Security Checklist
|
|
|
|
### Pre-Deployment
|
|
|
|
- [ ] All secrets in ./secrets directory
|
|
- [ ] Directory permissions: 700
|
|
- [ ] File permissions: 600
|
|
- [ ] No root-owned files
|
|
- [ ] ./secrets/* in .gitignore
|
|
- [ ] No secrets in .env
|
|
- [ ] No secrets in docker-compose.yml environment
|
|
- [ ] All referenced secrets exist
|
|
- [ ] docker-entrypoint.sh only when necessary
|
|
- [ ] No secrets in git history
|
|
|
|
### Post-Deployment
|
|
|
|
- [ ] Services can access secrets
|
|
- [ ] No secrets in container logs
|
|
- [ ] Secrets mounted at /run/secrets/
|
|
- [ ] No permission errors
|
|
- [ ] Rotation schedule established
|
|
|
|
---
|
|
|
|
*These patterns ensure secrets are managed securely throughout the stack lifecycle.*
|