gh-agentsecops-secopsagentkit/skills/devsecops/secrets-gitleaks/references/remediation_guide.md

Secret Remediation Guide

Comprehensive procedures for remediating exposed secrets detected by Gitleaks.

Table of Contents

• Immediate Response
• Remediation Workflow
• Git History Cleanup
• Cloud Provider Specific
• Database Credentials
• API Keys and Tokens
• Post-Remediation

Immediate Response

When secrets are detected, follow this priority order:

1. Assess Exposure (0-15 minutes)

Questions to answer immediately:

• Is the repository public or private?
• Has the commit been pushed to remote?
• How long has the secret been exposed?
• What systems does this credential access?

Actions:

# Check if commit is pushed
git log origin/main..HEAD  # If output, not yet pushed

# Check repository visibility
gh repo view --json visibility

# Check commit age
git log -1 --format="%ar" <commit-sha>

2. Rotate Credentials (0-30 minutes)

CRITICAL: Rotate the exposed credential immediately, regardless of exposure duration.

Priority order:

1. Production credentials - Immediate rotation
2. Payment/financial systems - Immediate rotation
3. Customer data access - Immediate rotation
4. Development/test credentials - Rotate within 24 hours

3. Review Access Logs (30-60 minutes)

Check for unauthorized access:

• Cloud provider audit logs (CloudTrail, Cloud Audit Logs, Activity Log)
• Application logs showing authentication attempts
• Database connection logs
• API usage logs

4. Remove from Code (0-24 hours)

Remove secret from current code and optionally from git history.

Remediation Workflow

Step 1: Rotate the Credential

Before removing from code, rotate the credential to prevent race conditions.

Cloud Providers

AWS:

# Deactivate compromised key
aws iam update-access-key \
  --access-key-id AKIA... \
  --status Inactive \
  --user-name username

# Create new key
aws iam create-access-key --user-name username

# Delete old key after updating applications
aws iam delete-access-key \
  --access-key-id AKIA... \
  --user-name username

GCP:

# Delete service account key
gcloud iam service-accounts keys delete KEY_ID \
  --iam-account=SERVICE_ACCOUNT_EMAIL

# Create new key
gcloud iam service-accounts keys create new-key.json \
  --iam-account=SERVICE_ACCOUNT_EMAIL

Azure:

# Regenerate storage account key
az storage account keys renew \
  --account-name ACCOUNT_NAME \
  --key primary

# List keys to verify
az storage account keys list \
  --account-name ACCOUNT_NAME

API Tokens

GitHub:

1. Navigate to Settings > Developer settings > Personal access tokens
2. Find the compromised token (check "Last used" column)
3. Click "Delete"
4. Generate new token with minimal required scopes

Stripe:

1. Log into Stripe Dashboard
2. Navigate to Developers > API keys
3. Click "Roll" on the compromised key
4. Update all applications with new key

Generic API Key:

1. Access provider's console/dashboard
2. Locate API key management
3. Revoke/delete compromised key
4. Generate new key
5. Update applications
6. Test connectivity

Step 2: Remove from Current Code

Replace hardcoded secrets with environment variables or secret management:

Before (insecure):

API_KEY = "sk_live_51ABC123..."
db_password = "MyP@ssw0rd123"

After (secure):

import os

API_KEY = os.environ.get("STRIPE_API_KEY")
if not API_KEY:
    raise ValueError("STRIPE_API_KEY environment variable not set")

db_password = os.environ.get("DB_PASSWORD")

Using secret management:

from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = SecretClient(vault_url="https://myvault.vault.azure.net/", credential=credential)

db_password = client.get_secret("database-password").value

Step 3: Commit the Fix

# Add changes
git add .

# Commit with clear message
git commit -m "refactor: Move API credentials to environment variables

- Replace hardcoded Stripe API key with environment variable
- Replace database password with AWS Secrets Manager reference
- Add validation for required environment variables

Addresses: Secret exposure detected by Gitleaks scan"

# Push
git push origin main

Git History Cleanup

If secrets are in pushed commits, consider removing from git history.

Decision Matrix

Scenario	Action	Reason
Public repo, secret exposed	Mandatory history rewrite	Secret is public knowledge
Private repo, < 24 hours, < 5 collaborators	Recommended history rewrite	Minimal disruption
Private repo, > 1 week, > 10 collaborators	Optional - Rotate only	High coordination cost
Production repo with CI/CD	Coordinate carefully	May break automation

Method 1: git-filter-repo (Recommended)

Install:

pip install git-filter-repo

Remove specific file from all history:

# Backup first
git clone --mirror <repo-url> backup-repo.git

# Remove file
git filter-repo --path config/secrets.yaml --invert-paths

# Force push
git push origin --force --all

Remove secrets matching pattern:

# Use callback for complex filtering
git filter-repo --replace-text <(echo 'regex:sk_live_[a-zA-Z0-9]{24}==>REDACTED')

Method 2: BFG Repo-Cleaner

Download:

# macOS
brew install bfg

# Or download JAR from https://rtyley.github.io/bfg-repo-cleaner/

Remove specific file:

# Clone mirror
git clone --mirror <repo-url> repo-mirror.git
cd repo-mirror.git

# Remove file
bfg --delete-files secrets.env

# Clean up
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Force push
git push

Remove secrets by pattern:

# Create replacements.txt
echo "PASSWORD1==>***REMOVED***" > replacements.txt
echo "sk_live_51ABC==>***REMOVED***" >> replacements.txt

# Run BFG
bfg --replace-text replacements.txt repo-mirror.git

Method 3: Interactive Rebase (Small Changes)

For recent commits not yet widely distributed:

# Rebase last N commits
git rebase -i HEAD~5

# In editor, mark commits to 'edit'
# When stopped at each commit:
git rm config/secrets.yaml
git commit --amend --no-edit
git rebase --continue

# Force push
git push --force-with-lease

Post-Rewrite Coordination

After rewriting history:

1. Notify team immediately:

URGENT: Git history rewritten to remove exposed credentials.

Action required for all developers:
1. Commit/stash any local changes
2. Run: git fetch origin && git reset --hard origin/main
3. Delete and re-clone if issues persist

Contact security team with questions.

2. Update CI/CD:

   • Invalidate old caches
   • May need to reconfigure webhooks
   • Update any hardcoded commit references

3. Update branch protection:

   • May need to temporarily disable
   • Re-enable after force push completes

Cloud Provider Specific

AWS

Check for unauthorized access:

# List recent API calls for access key
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=compromised-user \
  --max-results 50 \
  --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S)

Revoke all sessions:

# Attach policy to deny all actions
aws iam put-user-policy \
  --user-name compromised-user \
  --policy-name DenyAll \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*"}]}'

GCP

Check audit logs:

gcloud logging read "protoPayload.authenticationInfo.principalEmail=SERVICE_ACCOUNT_EMAIL" \
  --limit 100 \
  --format json

Disable service account:

gcloud iam service-accounts disable SERVICE_ACCOUNT_EMAIL

Azure

Review activity logs:

az monitor activity-log list \
  --start-time 2024-01-01T00:00:00Z \
  --resource-id /subscriptions/SUBSCRIPTION_ID

Revoke access:

# Regenerate keys
az storage account keys renew \
  --account-name STORAGE_ACCOUNT \
  --key primary

Database Credentials

PostgreSQL

-- Change password
ALTER USER app_user WITH PASSWORD 'new_secure_password';

-- View recent connections
SELECT datname, usename, client_addr, backend_start
FROM pg_stat_activity
WHERE usename = 'app_user'
ORDER BY backend_start DESC;

-- Kill active connections (if suspicious)
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE usename = 'app_user' AND client_addr != 'trusted_ip';

MySQL

-- Change password
ALTER USER 'app_user'@'%' IDENTIFIED BY 'new_secure_password';
FLUSH PRIVILEGES;

-- View recent connections
SELECT * FROM information_schema.PROCESSLIST
WHERE USER = 'app_user';

-- Kill connections
KILL CONNECTION process_id;

MongoDB

// Change password
use admin
db.changeUserPassword("app_user", "new_secure_password")

// View recent operations
db.currentOp({ "active": true })

// Kill operation
db.killOp(opid)

API Keys and Tokens

GitHub

Audit unauthorized access:

# List recent events for token
gh api /users/{username}/events/public | jq '.[] | {type, repo: .repo.name, created_at}'

Revoke all tokens (if compromised account):

1. Settings > Developer settings > Personal access tokens
2. Select all tokens
3. Click "Delete"

Slack

Check workspace audit logs:

1. Go to workspace settings (admin required)
2. Navigate to Logs > Audit Logs
3. Filter by token usage

Regenerate token:

1. Go to api.slack.com/apps
2. Select your app
3. Navigate to OAuth & Permissions
4. Click "Regenerate" on token

Post-Remediation

1. Implement Prevention

Pre-commit hooks:

# Install Gitleaks pre-commit hook
cd /path/to/repo
cat << 'EOF' > .git/hooks/pre-commit
#!/bin/sh
gitleaks protect --verbose --redact --staged
EOF
chmod +x .git/hooks/pre-commit

CI/CD checks:

# .github/workflows/secrets-scan.yml
name: Secret Scanning
on: [push, pull_request]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

2. Update Secret Management

Migrate to proper secret management:

Environment variables (minimal):

# .env (never commit!)
DATABASE_URL=postgresql://user:pass@host:5432/db
API_KEY=sk_live_...

# .gitignore
.env
.env.local

Secret management services:

• AWS: Secrets Manager, Systems Manager Parameter Store
• GCP: Secret Manager
• Azure: Key Vault
• HashiCorp: Vault
• Kubernetes: Secrets

3. Document Incident

Create incident report including:

• Timeline: When secret was committed, detected, remediated
• Exposure: Duration, repository visibility, access scope
• Impact: Systems accessed, data at risk, unauthorized activity
• Response: Rotation completed, logs reviewed, history cleaned
• Prevention: Controls implemented to prevent recurrence

4. Team Training

Conduct training on:

• Using environment variables and secret management
• Pre-commit hooks and local scanning
• Recognizing secrets in code review
• Incident response procedures

5. Compliance Notifications

If required by regulations:

• GDPR: Notify supervisory authority within 72 hours if personal data at risk
• PCI-DSS: Notify card brands and processor if payment data affected
• SOC2: Document in compliance report, may trigger audit
• HIPAA: Notify covered entities if PHI exposed

Prevention Checklist

• Credential rotated and old credential deactivated
• Access logs reviewed for unauthorized activity
• Secret removed from current code
• Git history cleaned (if applicable)
• Team notified of credential change
• Applications updated with new credential
• Pre-commit hooks installed
• CI/CD secret scanning enabled
• Secret management solution implemented
• Incident documented
• Compliance notifications sent (if required)
• Team training scheduled

Emergency Contacts

Maintain contact list for rapid response:

• Security Team: security@company.com
• DevOps On-Call: devops-oncall@company.com
• Cloud Provider Support: Account-specific
• Compliance Officer: compliance@company.com