gh-agentsecops-secopsagentkit/skills/devsecops/secrets-gitleaks/references/remediation_guide.md

# Secret Remediation Guide

Comprehensive procedures for remediating exposed secrets detected by Gitleaks.

## Table of Contents

- [Immediate Response](#immediate-response)
- [Remediation Workflow](#remediation-workflow)
- [Git History Cleanup](#git-history-cleanup)
- [Cloud Provider Specific](#cloud-provider-specific)
- [Database Credentials](#database-credentials)
- [API Keys and Tokens](#api-keys-and-tokens)
- [Post-Remediation](#post-remediation)

## Immediate Response

When secrets are detected, follow this priority order:

### 1. Assess Exposure (0-15 minutes)

**Questions to answer immediately:**
- Is the repository public or private?
- Has the commit been pushed to remote?
- How long has the secret been exposed?
- What systems does this credential access?

**Actions:**
```bash
# Check if commit is pushed
git log origin/main..HEAD  # If output, not yet pushed

# Check repository visibility
gh repo view --json visibility

# Check commit age
git log -1 --format="%ar" <commit-sha>
```

### 2. Rotate Credentials (0-30 minutes)

**CRITICAL**: Rotate the exposed credential immediately, regardless of exposure duration.

Priority order:
1. **Production credentials** - Immediate rotation
2. **Payment/financial systems** - Immediate rotation
3. **Customer data access** - Immediate rotation
4. **Development/test credentials** - Rotate within 24 hours

### 3. Review Access Logs (30-60 minutes)

Check for unauthorized access:
- Cloud provider audit logs (CloudTrail, Cloud Audit Logs, Activity Log)
- Application logs showing authentication attempts
- Database connection logs
- API usage logs

### 4. Remove from Code (0-24 hours)

Remove secret from current code and optionally from git history.

## Remediation Workflow

### Step 1: Rotate the Credential

**Before removing from code**, rotate the credential to prevent race conditions.

#### Cloud Providers

**AWS**:
```bash
# Deactivate compromised key
aws iam update-access-key \
  --access-key-id AKIA... \
  --status Inactive \
  --user-name username

# Create new key
aws iam create-access-key --user-name username

# Delete old key after updating applications
aws iam delete-access-key \
  --access-key-id AKIA... \
  --user-name username
```

**GCP**:
```bash
# Delete service account key
gcloud iam service-accounts keys delete KEY_ID \
  --iam-account=SERVICE_ACCOUNT_EMAIL

# Create new key
gcloud iam service-accounts keys create new-key.json \
  --iam-account=SERVICE_ACCOUNT_EMAIL
```

**Azure**:
```bash
# Regenerate storage account key
az storage account keys renew \
  --account-name ACCOUNT_NAME \
  --key primary

# List keys to verify
az storage account keys list \
  --account-name ACCOUNT_NAME
```

#### API Tokens

**GitHub**:
1. Navigate to Settings > Developer settings > Personal access tokens
2. Find the compromised token (check "Last used" column)
3. Click "Delete"
4. Generate new token with minimal required scopes

**Stripe**:
1. Log into Stripe Dashboard
2. Navigate to Developers > API keys
3. Click "Roll" on the compromised key
4. Update all applications with new key

**Generic API Key**:
1. Access provider's console/dashboard
2. Locate API key management
3. Revoke/delete compromised key
4. Generate new key
5. Update applications
6. Test connectivity

### Step 2: Remove from Current Code

Replace hardcoded secrets with environment variables or secret management:

**Before** (insecure):
```python
API_KEY = "sk_live_51ABC123..."
db_password = "MyP@ssw0rd123"
```

**After** (secure):
```python
import os

API_KEY = os.environ.get("STRIPE_API_KEY")
if not API_KEY:
    raise ValueError("STRIPE_API_KEY environment variable not set")

db_password = os.environ.get("DB_PASSWORD")
```

**Using secret management**:
```python
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = SecretClient(vault_url="https://myvault.vault.azure.net/", credential=credential)

db_password = client.get_secret("database-password").value
```

### Step 3: Commit the Fix

```bash
# Add changes
git add .

# Commit with clear message
git commit -m "refactor: Move API credentials to environment variables

- Replace hardcoded Stripe API key with environment variable
- Replace database password with AWS Secrets Manager reference
- Add validation for required environment variables

Addresses: Secret exposure detected by Gitleaks scan"

# Push
git push origin main
```

## Git History Cleanup

If secrets are in pushed commits, consider removing from git history.

### Decision Matrix

| Scenario | Action | Reason |
|----------|--------|--------|
| Public repo, secret exposed | **Mandatory** history rewrite | Secret is public knowledge |
| Private repo, < 24 hours, < 5 collaborators | **Recommended** history rewrite | Minimal disruption |
| Private repo, > 1 week, > 10 collaborators | **Optional** - Rotate only | High coordination cost |
| Production repo with CI/CD | **Coordinate carefully** | May break automation |

### Method 1: git-filter-repo (Recommended)

Install:
```bash
pip install git-filter-repo
```

Remove specific file from all history:
```bash
# Backup first
git clone --mirror <repo-url> backup-repo.git

# Remove file
git filter-repo --path config/secrets.yaml --invert-paths

# Force push
git push origin --force --all
```

Remove secrets matching pattern:
```bash
# Use callback for complex filtering
git filter-repo --replace-text <(echo 'regex:sk_live_[a-zA-Z0-9]{24}==>REDACTED')
```

### Method 2: BFG Repo-Cleaner

Download:
```bash
# macOS
brew install bfg

# Or download JAR from https://rtyley.github.io/bfg-repo-cleaner/
```

Remove specific file:
```bash
# Clone mirror
git clone --mirror <repo-url> repo-mirror.git
cd repo-mirror.git

# Remove file
bfg --delete-files secrets.env

# Clean up
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Force push
git push
```

Remove secrets by pattern:
```bash
# Create replacements.txt
echo "PASSWORD1==>***REMOVED***" > replacements.txt
echo "sk_live_51ABC==>***REMOVED***" >> replacements.txt

# Run BFG
bfg --replace-text replacements.txt repo-mirror.git
```

### Method 3: Interactive Rebase (Small Changes)

For recent commits not yet widely distributed:

```bash
# Rebase last N commits
git rebase -i HEAD~5

# In editor, mark commits to 'edit'
# When stopped at each commit:
git rm config/secrets.yaml
git commit --amend --no-edit
git rebase --continue

# Force push
git push --force-with-lease
```

### Post-Rewrite Coordination

After rewriting history:

1. **Notify team immediately**:
```text
URGENT: Git history rewritten to remove exposed credentials.

Action required for all developers:
1. Commit/stash any local changes
2. Run: git fetch origin && git reset --hard origin/main
3. Delete and re-clone if issues persist

Contact security team with questions.
```

2. **Update CI/CD**:
   - Invalidate old caches
   - May need to reconfigure webhooks
   - Update any hardcoded commit references

3. **Update branch protection**:
   - May need to temporarily disable
   - Re-enable after force push completes

## Cloud Provider Specific

### AWS

**Check for unauthorized access**:
```bash
# List recent API calls for access key
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=compromised-user \
  --max-results 50 \
  --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S)
```

**Revoke all sessions**:
```bash
# Attach policy to deny all actions
aws iam put-user-policy \
  --user-name compromised-user \
  --policy-name DenyAll \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*"}]}'
```

### GCP

**Check audit logs**:
```bash
gcloud logging read "protoPayload.authenticationInfo.principalEmail=SERVICE_ACCOUNT_EMAIL" \
  --limit 100 \
  --format json
```

**Disable service account**:
```bash
gcloud iam service-accounts disable SERVICE_ACCOUNT_EMAIL
```

### Azure

**Review activity logs**:
```bash
az monitor activity-log list \
  --start-time 2024-01-01T00:00:00Z \
  --resource-id /subscriptions/SUBSCRIPTION_ID
```

**Revoke access**:
```bash
# Regenerate keys
az storage account keys renew \
  --account-name STORAGE_ACCOUNT \
  --key primary
```

## Database Credentials

### PostgreSQL

```sql
-- Change password
ALTER USER app_user WITH PASSWORD 'new_secure_password';

-- View recent connections
SELECT datname, usename, client_addr, backend_start
FROM pg_stat_activity
WHERE usename = 'app_user'
ORDER BY backend_start DESC;

-- Kill active connections (if suspicious)
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE usename = 'app_user' AND client_addr != 'trusted_ip';
```

### MySQL

```sql
-- Change password
ALTER USER 'app_user'@'%' IDENTIFIED BY 'new_secure_password';
FLUSH PRIVILEGES;

-- View recent connections
SELECT * FROM information_schema.PROCESSLIST
WHERE USER = 'app_user';

-- Kill connections
KILL CONNECTION process_id;
```

### MongoDB

```javascript
// Change password
use admin
db.changeUserPassword("app_user", "new_secure_password")

// View recent operations
db.currentOp({ "active": true })

// Kill operation
db.killOp(opid)
```

## API Keys and Tokens

### GitHub

**Audit unauthorized access**:
```bash
# List recent events for token
gh api /users/{username}/events/public | jq '.[] | {type, repo: .repo.name, created_at}'
```

**Revoke all tokens** (if compromised account):
1. Settings > Developer settings > Personal access tokens
2. Select all tokens
3. Click "Delete"

### Slack

**Check workspace audit logs**:
1. Go to workspace settings (admin required)
2. Navigate to Logs > Audit Logs
3. Filter by token usage

**Regenerate token**:
1. Go to api.slack.com/apps
2. Select your app
3. Navigate to OAuth & Permissions
4. Click "Regenerate" on token

## Post-Remediation

### 1. Implement Prevention

**Pre-commit hooks**:
```bash
# Install Gitleaks pre-commit hook
cd /path/to/repo
cat << 'EOF' > .git/hooks/pre-commit
#!/bin/sh
gitleaks protect --verbose --redact --staged
EOF
chmod +x .git/hooks/pre-commit
```

**CI/CD checks**:
```yaml
# .github/workflows/secrets-scan.yml
name: Secret Scanning
on: [push, pull_request]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

### 2. Update Secret Management

Migrate to proper secret management:

**Environment variables** (minimal):
```bash
# .env (never commit!)
DATABASE_URL=postgresql://user:pass@host:5432/db
API_KEY=sk_live_...

# .gitignore
.env
.env.local
```

**Secret management services**:
- AWS: Secrets Manager, Systems Manager Parameter Store
- GCP: Secret Manager
- Azure: Key Vault
- HashiCorp: Vault
- Kubernetes: Secrets

### 3. Document Incident

Create incident report including:
- **Timeline**: When secret was committed, detected, remediated
- **Exposure**: Duration, repository visibility, access scope
- **Impact**: Systems accessed, data at risk, unauthorized activity
- **Response**: Rotation completed, logs reviewed, history cleaned
- **Prevention**: Controls implemented to prevent recurrence

### 4. Team Training

Conduct training on:
- Using environment variables and secret management
- Pre-commit hooks and local scanning
- Recognizing secrets in code review
- Incident response procedures

### 5. Compliance Notifications

If required by regulations:
- **GDPR**: Notify supervisory authority within 72 hours if personal data at risk
- **PCI-DSS**: Notify card brands and processor if payment data affected
- **SOC2**: Document in compliance report, may trigger audit
- **HIPAA**: Notify covered entities if PHI exposed

## Prevention Checklist

- [ ] Credential rotated and old credential deactivated
- [ ] Access logs reviewed for unauthorized activity
- [ ] Secret removed from current code
- [ ] Git history cleaned (if applicable)
- [ ] Team notified of credential change
- [ ] Applications updated with new credential
- [ ] Pre-commit hooks installed
- [ ] CI/CD secret scanning enabled
- [ ] Secret management solution implemented
- [ ] Incident documented
- [ ] Compliance notifications sent (if required)
- [ ] Team training scheduled

## Emergency Contacts

Maintain contact list for rapid response:
- **Security Team**: security@company.com
- **DevOps On-Call**: devops-oncall@company.com
- **Cloud Provider Support**: Account-specific
- **Compliance Officer**: compliance@company.com