zhongwei/gh-agentsecops-secopsagentkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
Zhongwei Li
2025-11-29 17:51:02 +08:00
commit ff1f4bd119
252 changed files with 72682 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

502
skills/devsecops/secrets-gitleaks/SKILL.md Normal file
View File

@@ -0,0 +1,502 @@
---
name: secrets-gitleaks
description: >
Hardcoded secret detection and prevention in git repositories and codebases using Gitleaks.
Identifies passwords, API keys, tokens, and credentials through regex-based pattern matching
and entropy analysis. Use when: (1) Scanning repositories for exposed secrets and credentials,
(2) Implementing pre-commit hooks to prevent secret leakage, (3) Integrating secret detection
into CI/CD pipelines, (4) Auditing codebases for compliance violations (PCI-DSS, SOC2, GDPR),
(5) Establishing baseline secret detection and tracking new exposures, (6) Remediating
historical secret exposures in git history.
version: 0.1.0
maintainer: SirAppSec
category: devsecops
tags: [secrets, gitleaks, secret-scanning, devsecops, ci-cd, credentials, api-keys, compliance]
frameworks: [OWASP, CWE, PCI-DSS, SOC2, GDPR]
dependencies:
tools: [gitleaks, git]
references:
- https://github.com/gitleaks/gitleaks
- https://owasp.org/Top10/A07_2021-Identification_and_Authentication_Failures/
- https://cwe.mitre.org/data/definitions/798.html
---
# Secrets Detection with Gitleaks
## Overview
Gitleaks is a secret detection tool that scans git repositories, files, and directories for hardcoded credentials including passwords, API keys, tokens, and other sensitive information. It uses regex-based pattern matching combined with Shannon entropy analysis to identify secrets that could lead to unauthorized access if exposed.
This skill provides comprehensive guidance for integrating Gitleaks into DevSecOps workflows, from pre-commit hooks to CI/CD pipelines, with emphasis on preventing secret leakage before code reaches production.
## Quick Start
Scan current repository for secrets:
```bash
# Install gitleaks
brew install gitleaks # macOS
# or: docker pull zricethezav/gitleaks:latest
# Scan current git repository
gitleaks detect -v
# Scan specific directory
gitleaks detect --source /path/to/code -v
# Generate report
gitleaks detect --report-path gitleaks-report.json --report-format json
```
## Core Workflows
### 1. Repository Scanning
Scan existing repositories to identify exposed secrets:
```bash
# Full repository scan with verbose output
gitleaks detect -v --source /path/to/repo
# Scan with custom configuration
gitleaks detect --config .gitleaks.toml -v
# Generate JSON report for further analysis
gitleaks detect --report-path findings.json --report-format json
# Generate SARIF report for GitHub/GitLab integration
gitleaks detect --report-path findings.sarif --report-format sarif
```
**When to use**: Initial security audit, compliance checks, incident response.
### 2. Pre-Commit Hook Protection
Prevent secrets from being committed in the first place:
```bash
# Install pre-commit hook (run in repository root)
cat << 'EOF' > .git/hooks/pre-commit
#!/bin/sh
gitleaks protect --verbose --redact --staged
EOF
chmod +x .git/hooks/pre-commit
```
Use the bundled script for automated hook installation:
```bash
./scripts/install_precommit.sh
```
**When to use**: Developer workstation setup, team onboarding, mandatory security controls.
### 3. CI/CD Pipeline Integration
#### GitHub Actions
```yaml
name: gitleaks
on: [push, pull_request]
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
#### GitLab CI
```yaml
gitleaks:
image: zricethezav/gitleaks:latest
stage: test
script:
- gitleaks detect --report-path gitleaks.json --report-format json --verbose
artifacts:
paths:
- gitleaks.json
when: always
allow_failure: false
```
**When to use**: Automated security gates, pull request checks, release validation.
### 4. Baseline and Incremental Scanning
Establish security baseline and track only new secrets:
```bash
# Create initial baseline
gitleaks detect --report-path baseline.json --report-format json
# Subsequent scans detect only new secrets
gitleaks detect --baseline-path baseline.json --report-path new-findings.json -v
```
**When to use**: Legacy codebase remediation, phased rollout, compliance tracking.
### 5. Configuration Customization
Create custom `.gitleaks.toml` configuration:
```toml
title = "Custom Gitleaks Configuration"
[extend]
# Extend default config with custom rules
useDefault = true
[[rules]]
id = "custom-api-key"
description = "Custom API Key Pattern"
regex = '''(?i)(custom_api_key|custom_secret)[\s]*[=:][\s]*['"][a-zA-Z0-9]{32,}['"]'''
tags = ["api-key", "custom"]
[allowlist]
description = "Global allowlist"
paths = [
'''\.md$''', # Ignore markdown files
'''test/fixtures/''', # Ignore test fixtures
]
stopwords = [
'''EXAMPLE''', # Ignore example keys
'''PLACEHOLDER''',
]
```
Use bundled configuration templates in `assets/`:
- `assets/config-strict.toml` - Strict detection (low false negatives)
- `assets/config-balanced.toml` - Balanced detection (recommended)
- `assets/config-custom.toml` - Template for custom rules
**When to use**: Reducing false positives, adding proprietary secret patterns, organizational standards.
## Security Considerations
### Sensitive Data Handling
- **Secret Redaction**: Always use `--redact` flag in logs and reports to prevent accidental secret exposure
- **Report Security**: Gitleaks reports contain detected secrets - treat as confidential, encrypt at rest
- **Git History**: Detected secrets in git history require complete removal using tools like `git filter-repo` or `BFG Repo-Cleaner`
- **Credential Rotation**: All exposed secrets must be rotated immediately, even if removed from code
### Access Control
- **CI/CD Permissions**: Gitleaks scans require read access to repository content and git history
- **Report Access**: Restrict access to scan reports containing sensitive findings
- **Baseline Files**: Baseline JSON files contain secret metadata - protect with same controls as findings
### Audit Logging
Log the following for compliance and incident response:
- Scan execution timestamps and scope (repository, branch, commit range)
- Number and types of secrets detected
- Remediation actions taken (credential rotation, commit history cleanup)
- False positive classifications and allowlist updates
### Compliance Requirements
- **PCI-DSS 3.2.1**: Requirement 6.5.3 - Prevent hardcoded credentials in payment applications
- **SOC2**: CC6.1 - Logical access controls prevent unauthorized credential exposure
- **GDPR**: Article 32 - Appropriate security measures for processing personal data credentials
- **CWE-798**: Use of Hard-coded Credentials
- **CWE-259**: Use of Hard-coded Password
- **OWASP A07:2021**: Identification and Authentication Failures
## Bundled Resources
### Scripts (`scripts/`)
- `install_precommit.sh` - Automated pre-commit hook installation with configuration prompts
- `scan_and_report.py` - Comprehensive scanning with multiple output formats and severity classification
- `baseline_manager.py` - Baseline creation, comparison, and incremental scan management
### References (`references/`)
- `detection_rules.md` - Comprehensive list of built-in Gitleaks detection rules with CWE mappings
- `remediation_guide.md` - Step-by-step secret remediation procedures including git history cleanup
- `false_positives.md` - Common false positive patterns and allowlist configuration strategies
- `compliance_mapping.md` - Detailed mapping to PCI-DSS, SOC2, GDPR, and OWASP requirements
### Assets (`assets/`)
- `config-strict.toml` - High-sensitivity configuration (maximum detection)
- `config-balanced.toml` - Production-ready balanced configuration
- `config-custom.toml` - Template with inline documentation for custom rules
- `precommit-config.yaml` - Pre-commit framework configuration
- `github-action.yml` - Complete GitHub Actions workflow template
- `gitlab-ci.yml` - Complete GitLab CI pipeline template
## Common Patterns
### Pattern 1: Initial Repository Audit
First-time secret scanning for security assessment:
```bash
# 1. Clone repository with full history
git clone --mirror https://github.com/org/repo.git audit-repo
cd audit-repo
# 2. Run comprehensive scan
gitleaks detect --report-path audit-report.json --report-format json -v
# 3. Generate human-readable report
./scripts/scan_and_report.py --input audit-report.json --format markdown --output audit-report.md
# 4. Review findings and classify false positives
# Edit .gitleaks.toml to add allowlist entries
# 5. Create baseline for future scans
cp audit-report.json baseline.json
```
### Pattern 2: Developer Workstation Setup
Protect developers from accidental secret commits:
```bash
# 1. Install gitleaks locally
brew install gitleaks # macOS
# or use package manager for your OS
# 2. Install pre-commit hook
./scripts/install_precommit.sh
# 3. Test hook with dummy commit
echo "api_key = 'EXAMPLE_KEY_12345'" > test.txt
git add test.txt
git commit -m "test" # Should be blocked by gitleaks
# 4. Clean up test
git reset HEAD~1
rm test.txt
```
### Pattern 3: CI/CD Pipeline with Baseline
Progressive secret detection in continuous integration:
```bash
# In CI pipeline script:
# 1. Check if baseline exists
if [ -f ".gitleaks-baseline.json" ]; then
# Incremental scan - only new secrets
gitleaks detect \
--baseline-path .gitleaks-baseline.json \
--report-path new-findings.json \
--report-format json \
--exit-code 1 # Fail on new secrets
else
# Initial scan - create baseline
gitleaks detect \
--report-path .gitleaks-baseline.json \
--report-format json \
--exit-code 0 # Don't fail on first scan
fi
# 2. Generate SARIF for GitHub Security tab
if [ -f "new-findings.json" ] && [ -s "new-findings.json" ]; then
gitleaks detect \
--baseline-path .gitleaks-baseline.json \
--report-path results.sarif \
--report-format sarif
fi
```
### Pattern 4: Custom Rule Development
Add organization-specific secret patterns:
```toml
# Add to .gitleaks.toml
[[rules]]
id = "acme-corp-api-key"
description = "ACME Corp Internal API Key"
regex = '''(?i)acme[_-]?api[_-]?key[\s]*[=:][\s]*['"]?([a-f0-9]{40})['"]?'''
secretGroup = 1
tags = ["api-key", "acme-internal"]
[[rules]]
id = "acme-corp-database-password"
description = "ACME Corp Database Password Format"
regex = '''(?i)(db_pass|database_password)[\s]*[=:][\s]*['"]([A-Z][a-z0-9@#$%]{15,})['"]'''
secretGroup = 2
tags = ["password", "database", "acme-internal"]
# Test custom rules
# gitleaks detect --config .gitleaks.toml -v
```
## Integration Points
### CI/CD Integration
- **GitHub Actions**: Use `gitleaks/gitleaks-action@v2` for native integration with Security tab
- **GitLab CI**: Docker-based scanning with artifact retention for audit trails
- **Jenkins**: Execute via Docker or installed binary in pipeline stages
- **CircleCI**: Docker executor with orb support
- **Azure Pipelines**: Task-based integration with results publishing
### Security Tools Ecosystem
- **SIEM Integration**: Export JSON findings to Splunk, ELK, or Datadog for centralized monitoring
- **Vulnerability Management**: Import SARIF reports into Snyk, SonarQube, or Checkmarx
- **Secret Management**: Integrate findings with HashiCorp Vault or AWS Secrets Manager rotation workflows
- **Ticketing Systems**: Automated Jira/ServiceNow ticket creation for remediation tracking
### SDLC Integration
- **Design Phase**: Include secret detection requirements in security architecture reviews
- **Development**: Pre-commit hooks provide immediate feedback to developers
- **Code Review**: PR/MR checks prevent secrets from reaching main branches
- **Testing**: Scan test environments and infrastructure-as-code
- **Deployment**: Final validation gate before production release
- **Operations**: Periodic scanning of deployed configurations and logs
## Troubleshooting
### Issue: Too Many False Positives
**Symptoms**: Legitimate code patterns flagged as secrets (test fixtures, examples, placeholders)
**Solution**:
1. Review findings to identify patterns: `grep -i "example\|test\|placeholder" gitleaks-report.json`
2. Add to allowlist in `.gitleaks.toml`:
```toml
[allowlist]
paths = ['''test/''', '''examples/''', '''\.md$''']
stopwords = ["EXAMPLE", "PLACEHOLDER", "YOUR_API_KEY_HERE"]
```
3. Use commit allowlists for specific false positives:
```toml
[allowlist]
commits = ["commit-sha-here"]
```
4. Consult `references/false_positives.md` for common patterns
### Issue: Performance Issues on Large Repositories
**Symptoms**: Scans taking excessive time (>10 minutes), high memory usage
**Solution**:
1. Use `--log-opts` to limit git history: `gitleaks detect --log-opts="--since=2024-01-01"`
2. Scan specific branches: `gitleaks detect --log-opts="origin/main"`
3. Use baseline approach to scan only recent changes
4. Consider shallow clone for initial scans: `git clone --depth=1000`
5. Parallelize scans across multiple branches or subdirectories
### Issue: Pre-commit Hook Blocking Valid Commits
**Symptoms**: Developers unable to commit code with legitimate patterns
**Solution**:
1. Add inline comment to bypass hook: `# gitleaks:allow`
2. Update `.gitleaks.toml` allowlist for the specific pattern
3. Use `--redact` to safely review findings: `gitleaks protect --staged --redact`
4. Temporary bypass (use with caution): `git commit --no-verify`
5. Review with security team if pattern is genuinely needed
### Issue: Secrets Found in Git History
**Symptoms**: Secrets detected in old commits, already removed from current code
**Solution**:
1. Rotate compromised credentials immediately (highest priority)
2. For public repositories, consider full history rewrite using:
- `git filter-repo` (recommended): `git filter-repo --path-glob '*.env' --invert-paths`
- BFG Repo-Cleaner: `bfg --delete-files credentials.json`
3. Force-push cleaned history: `git push --force`
4. Notify all contributors to rebase/re-clone
5. See `references/remediation_guide.md` for detailed procedures
6. Document incident in security audit log
### Issue: Custom Secret Patterns Not Detected
**Symptoms**: Organization-specific secrets not caught by default rules
**Solution**:
1. Develop regex pattern: Test at regex101.com with sample secrets
2. Add custom rule to `.gitleaks.toml`:
```toml
[[rules]]
id = "custom-secret-id"
description = "Description"
regex = '''your-pattern-here'''
secretGroup = 1 # Capture group containing actual secret
```
3. Test pattern: `gitleaks detect --config .gitleaks.toml -v --no-git`
4. Consider entropy threshold if pattern is ambiguous:
```toml
[[rules.Entropies]]
Min = "3.5"
Max = "7.0"
Group = "1"
```
5. Validate with known true positives and negatives
## Advanced Configuration
### Entropy-Based Detection
For secrets without clear patterns, use Shannon entropy analysis:
```toml
[[rules]]
id = "high-entropy-strings"
description = "High entropy strings that may be secrets"
regex = '''[a-zA-Z0-9]{32,}'''
entropy = 4.5 # Shannon entropy threshold
secretGroup = 0
```
### Composite Rules (v8.28.0+)
Detect secrets spanning multiple lines or requiring context:
```toml
[[rules]]
id = "multi-line-secret"
description = "API key with usage context"
regex = '''api_key[\s]*='''
[[rules.composite]]
pattern = '''initialize_client'''
location = "line" # Must be within same line proximity
distance = 5 # Within 5 lines
```
### Global vs Rule-Specific Allowlists
```toml
# Global allowlist (highest precedence)
[allowlist]
description = "Organization-wide exceptions"
paths = ['''vendor/''', '''node_modules/''']
# Rule-specific allowlist
[[rules]]
id = "generic-api-key"
[rules.allowlist]
description = "Exceptions only for this rule"
regexes = ['''key\s*=\s*EXAMPLE''']
```
## References
- [Gitleaks Official Documentation](https://github.com/gitleaks/gitleaks)
- [OWASP A07:2021 - Identification and Authentication Failures](https://owasp.org/Top10/A07_2021-Identification_and_Authentication_Failures/)
- [CWE-798: Use of Hard-coded Credentials](https://cwe.mitre.org/data/definitions/798.html)
- [CWE-259: Use of Hard-coded Password](https://cwe.mitre.org/data/definitions/259.html)
- [CWE-321: Use of Hard-coded Cryptographic Key](https://cwe.mitre.org/data/definitions/321.html)
- [PCI-DSS Requirements](https://www.pcisecuritystandards.org/)
- [SOC2 Security Criteria](https://www.aicpa.org/interestareas/frc/assuranceadvisoryservices/aicpasoc2report.html)

9
skills/devsecops/secrets-gitleaks/assets/.gitkeep Normal file
View File

@@ -0,0 +1,9 @@
# Assets Directory
Place files that will be used in the output Claude produces:
- Templates
- Configuration files
- Images/logos
- Boilerplate code
These files are NOT loaded into context but copied/modified in output.

81
skills/devsecops/secrets-gitleaks/assets/config-balanced.toml Normal file
View File

@@ -0,0 +1,81 @@
# Gitleaks Balanced Configuration
# Production-ready configuration balancing security and developer experience
# Use for: Most production repositories
title = "Gitleaks Balanced Configuration"
[extend]
# Extend default Gitleaks rules
useDefault = true
[allowlist]
description = "Balanced allowlist for common false positives"
# Standard non-production paths
paths = [
'''test/.*''',
'''tests/.*''',
'''.*/fixtures/.*''',
'''.*/testdata/.*''',
'''spec/.*''',
'''examples?/.*''',
'''docs?/.*''',
'''\.md$''',
'''\.rst$''',
'''\.txt$''',
'''node_modules/.*''',
'''vendor/.*''',
'''third[_-]party/.*''',
'''\.min\.js$''',
'''\.min\.css$''',
'''dist/.*''',
'''build/.*''',
'''target/.*''',
'''.*/mocks?/.*''',
]
# Common placeholder patterns
stopwords = [
"example",
"placeholder",
"your_api_key_here",
"your_key_here",
"your_secret_here",
"replace_me",
"replaceme",
"changeme",
"change_me",
"insert_key_here",
"xxxxxx",
"000000",
"123456",
"abcdef",
"sample",
"dummy",
"fake",
"test_key",
"test_secret",
"test_password",
"test_token",
"mock",
"TODO",
]
# Public non-secrets
regexes = [
'''-----BEGIN CERTIFICATE-----''',
'''-----BEGIN PUBLIC KEY-----''',
'''data:image/[^;]+;base64,''',
'''[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}''', # UUID
]
# Manually verified false positives (add with comments)
commits = []
# Custom rules for organization-specific patterns can be added below
# Example: Allowlist template files
# [[rules]]
# id = "generic-api-key"
# [rules.allowlist]
# paths = ['''config/.*\.template$''', '''config/.*\.example$''']

178
skills/devsecops/secrets-gitleaks/assets/config-custom.toml Normal file
View File

@@ -0,0 +1,178 @@
# Gitleaks Custom Configuration Template
# Use this as a starting point for organization-specific detection rules
title = "Custom Gitleaks Configuration"
[extend]
# Extend default Gitleaks rules with custom rules
useDefault = true
# =============================================================================
# GLOBAL ALLOWLIST
# =============================================================================
# Global allowlists apply to ALL rules and have highest precedence
[allowlist]
description = "Global allowlist for organization-wide exceptions"
# Paths to exclude from scanning
paths = [
# Test and documentation
'''test/.*''',
'''docs?/.*''',
'''examples?/.*''',
# Dependencies
'''node_modules/.*''',
'''vendor/.*''',
# Build artifacts
'''dist/.*''',
'''build/.*''',
]
# Known placeholder values
stopwords = [
"example",
"placeholder",
"your_key_here",
"test",
"mock",
"dummy",
]
# Public non-secrets
regexes = [
'''-----BEGIN CERTIFICATE-----''',
'''-----BEGIN PUBLIC KEY-----''',
]
# Manually verified commits (add with explanatory comments)
commits = []
# =============================================================================
# CUSTOM DETECTION RULES
# =============================================================================
# Add organization-specific secret patterns here
# Example: Custom API Key Pattern
[[rules]]
id = "acme-corp-api-key"
description = "ACME Corp Internal API Key"
# Regex pattern to match your organization's API key format
# Use triple-quoted strings for complex patterns
regex = '''(?i)acme[_-]?api[_-]?key[\s]*[=:][\s]*['"]?([a-zA-Z0-9]{40})['"]?'''
# Capture group containing the actual secret (for entropy analysis)
secretGroup = 1
# Tags for categorization and filtering
tags = ["api-key", "acme-internal"]
# Optional: Rule-specific allowlist (lower precedence than global)
#[rules.allowlist]
#paths = ['''config/defaults\.yaml''']
#stopwords = ["DEFAULT_KEY"]
# Example: Custom Database Password Pattern
[[rules]]
id = "acme-corp-db-password"
description = "ACME Corp Database Password Format"
# Matches company-specific password format
regex = '''(?i)(db_pass|database_password)[\s]*[=:][\s]*['"]([A-Z][a-z0-9@#$%]{15,})['"]'''
secretGroup = 2
tags = ["password", "database", "acme-internal"]
# Example: High-Entropy Detection with Custom Threshold
[[rules]]
id = "high-entropy-string"
description = "High entropy string (potential secret)"
# Match strings of 32+ alphanumeric characters
regex = '''[a-zA-Z0-9+/]{32,}'''
# Shannon entropy threshold (0.0 - 8.0, higher = more random)
entropy = 4.5
# Which capture group to analyze (0 = entire match)
secretGroup = 0
tags = ["entropy", "generic"]
[rules.allowlist]
# Allowlist base64-encoded images
regexes = ['''data:image/[^;]+;base64,''']
# Example: Custom Service Account Key
[[rules]]
id = "acme-corp-service-account"
description = "ACME Corp Service Account JSON Key"
# Detect JSON structure with specific fields
regex = '''"type":\s*"acme_service_account"'''
tags = ["service-account", "acme-internal"]
# Example: Custom OAuth Token Format
[[rules]]
id = "acme-corp-oauth-token"
description = "ACME Corp OAuth Token"
# Custom token format: acme_oauth_v1_<40 hex chars>
regex = '''acme_oauth_v1_[a-f0-9]{40}'''
tags = ["oauth", "token", "acme-internal"]
# =============================================================================
# TESTING CUSTOM RULES
# =============================================================================
# Test your custom rules with:
# gitleaks detect --config config-custom.toml -v
#
# Test against specific file:
# gitleaks detect --config config-custom.toml --source path/to/file --no-git
#
# Test regex pattern online:
# https://regex101.com/ (select Golang flavor)
#
# =============================================================================
# =============================================================================
# ENTROPY ANALYSIS GUIDE
# =============================================================================
# Entropy values (Shannon entropy):
# 0.0 - 2.5: Very low (repeated characters, simple patterns)
# 2.5 - 3.5: Low (common words, simple sequences)
# 3.5 - 4.5: Medium (mixed case, some randomness)
# 4.5 - 5.5: High (strong randomness, likely secret)
# 5.5 - 8.0: Very high (cryptographic randomness)
#
# Recommended thresholds:
# - API keys: 4.5+
# - Passwords: 3.5+
# - Tokens: 4.5+
# - Generic secrets: 5.0+
# =============================================================================
# =============================================================================
# REGEX CAPTURE GROUPS
# =============================================================================
# Use capture groups to extract the actual secret from surrounding text:
#
# regex = '''api_key\s*=\s*"([a-zA-Z0-9]+)"'''
# ^^^^^^^^^
# Group 1
#
# secretGroup = 1 # Analyze only the key value, not 'api_key = ""'
#
# This improves entropy analysis accuracy and reduces false positives.
# =============================================================================
# =============================================================================
# COMPOSITE RULES (Advanced)
# =============================================================================
# Gitleaks v8.28.0+ supports composite rules for context-aware detection
# Useful for secrets that require nearby context (multi-line patterns)
#[[rules]]
#id = "composite-api-key"
#description = "API key with usage context"
#regex = '''api_key\s*='''
#
#[[rules.composite]]
#pattern = '''initialize_client'''
#location = "line" # "line", "fragment", or "commit"
#distance = 5 # Within 5 lines
#
# This detects api_key = "..." only when "initialize_client" appears within 5 lines
# =============================================================================

48
skills/devsecops/secrets-gitleaks/assets/config-strict.toml Normal file
View File

@@ -0,0 +1,48 @@
# Gitleaks Strict Configuration
# High-sensitivity detection with minimal allowlisting
# Use for: Security-critical repositories, financial services, healthcare
title = "Gitleaks Strict Configuration"
[extend]
# Use all default Gitleaks rules
useDefault = true
[allowlist]
description = "Minimal allowlist - only proven false positives"
# Only allow in build artifacts and dependencies
paths = [
'''node_modules/.*''',
'''vendor/.*''',
'''\.min\.js$''',
'''\.min\.css$''',
]
# Only obvious non-secret patterns
stopwords = [
"EXAMPLE_DO_NOT_USE",
"PLACEHOLDER_REPLACE_ME",
]
# All commits must be manually verified before allowlisting
commits = []
# Additional strict rules for high-value targets
[[rules]]
id = "strict-env-file"
description = "Detect any .env files (should not be in repo)"
regex = '''.*'''
path = '''\.env$'''
tags = ["env-file", "strict"]
[[rules]]
id = "strict-config-secrets"
description = "Config files with potential secrets"
regex = '''(?i)(password|secret|key|token|credential)[\s]*[=:][\s]*['"]?([a-zA-Z0-9!@#$%^&*()_+\-=\[\]{};':"\\|,.<>\/?]{8,})['"]?'''
secretGroup = 2
tags = ["config", "strict"]
[rules.allowlist]
paths = ['''test/.*''']
stopwords = ["EXAMPLE"]

181
skills/devsecops/secrets-gitleaks/assets/github-action.yml Normal file
View File

@@ -0,0 +1,181 @@
# GitHub Actions Workflow for Gitleaks Secret Scanning
# Save as: .github/workflows/gitleaks.yml
name: Secret Scanning with Gitleaks
on:
push:
branches:
- main
- develop
- 'release/**'
pull_request:
branches:
- main
- develop
schedule:
# Run daily at 2 AM UTC
- cron: '0 2 * * *'
workflow_dispatch: # Allow manual triggers
# Cancel in-progress runs when new commit pushed
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
gitleaks-scan:
name: Scan for Secrets
runs-on: ubuntu-latest
permissions:
# Required for uploading SARIF results to GitHub Security tab
security-events: write
# Required for checking out private repos
contents: read
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
# Fetch full history for comprehensive scanning
fetch-depth: 0
- name: Run Gitleaks Scan
id: gitleaks
uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Optional: Use custom configuration
# GITLEAKS_CONFIG: .gitleaks.toml
# Optional: Generate JSON report for further processing
- name: Generate JSON Report
if: always() # Run even if secrets found
run: |
docker run --rm -v ${{ github.workspace }}:/repo \
zricethezav/gitleaks:latest \
detect --source /repo \
--report-path /repo/gitleaks-report.json \
--report-format json \
--exit-code 0 || true
# Optional: Upload JSON report as artifact
- name: Upload Scan Report
if: always()
uses: actions/upload-artifact@v4
with:
name: gitleaks-report
path: gitleaks-report.json
retention-days: 30
# Optional: Generate SARIF report for GitHub Security tab
- name: Generate SARIF Report
if: always()
run: |
docker run --rm -v ${{ github.workspace }}:/repo \
zricethezav/gitleaks:latest \
detect --source /repo \
--report-path /repo/gitleaks.sarif \
--report-format sarif \
--exit-code 0 || true
# Optional: Upload SARIF report to GitHub Security
- name: Upload SARIF to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: gitleaks.sarif
category: gitleaks
# Optional: Comment on PR with findings
- name: Comment PR with Findings
if: failure() && github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
try {
const report = JSON.parse(fs.readFileSync('gitleaks-report.json', 'utf8'));
const findings = report.length;
const comment = `## 🔒 Secret Scanning Results
⚠️ **${findings} potential secret(s) detected!**
Please review the findings and take immediate action:
1. **Do not merge** this PR until secrets are removed
2. Rotate any exposed credentials immediately
3. Remove secrets from code and use environment variables
4. Review the security tab for detailed findings
See [Secret Scanning Guide](https://github.com/${{ github.repository }}/blob/main/docs/secret-scanning.md) for remediation steps.`;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
});
} catch (error) {
console.log('No report file or error reading it:', error.message);
}
# Optional: Post to Slack on failure
- name: Notify Slack on Failure
if: failure()
uses: slackapi/slack-github-action@v1
with:
payload: |
{
"text": "🚨 Secrets detected in ${{ github.repository }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Secret Scanning Alert*\n\nSecrets detected in repository: `${{ github.repository }}`\nBranch: `${{ github.ref_name }}`\nCommit: `${{ github.sha }}`\n\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Details>"
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK
# Optional: Baseline scanning for incremental detection
baseline-scan:
name: Incremental Scan Against Baseline
runs-on: ubuntu-latest
if: github.event_name == 'push'
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Download Existing Baseline
continue-on-error: true
run: |
# Download baseline from artifact storage or S3
# Example: aws s3 cp s3://bucket/.gitleaks-baseline.json .
echo "Baseline download would go here"
- name: Run Incremental Scan
run: |
docker run --rm -v ${{ github.workspace }}:/repo \
zricethezav/gitleaks:latest \
detect --source /repo \
--baseline-path /repo/.gitleaks-baseline.json \
--report-path /repo/new-findings.json \
--report-format json \
--exit-code 1 || true
- name: Upload New Findings
if: always()
uses: actions/upload-artifact@v4
with:
name: new-findings
path: new-findings.json
retention-days: 90

257
skills/devsecops/secrets-gitleaks/assets/gitlab-ci.yml Normal file
View File

@@ -0,0 +1,257 @@
# GitLab CI Pipeline for Gitleaks Secret Scanning
# Save as: .gitlab-ci.yml or include in existing pipeline
# Define stages
stages:
- security
- report
# Default Docker image for security jobs
image: docker:latest
services:
- docker:dind
variables:
# Gitleaks Docker image
GITLEAKS_IMAGE: zricethezav/gitleaks:latest
# Report output path
REPORT_PATH: gitleaks-report.json
# SARIF output for GitLab Security Dashboard
SARIF_PATH: gl-secret-detection-report.json
# Secret scanning job
gitleaks-scan:
stage: security
image: $GITLEAKS_IMAGE
# Run on all branches and merge requests
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
- if: '$CI_COMMIT_BRANCH =~ /^(develop|release)/'
script:
# Run Gitleaks scan
- echo "Running Gitleaks secret detection..."
- |
gitleaks detect \
--source . \
--report-path $REPORT_PATH \
--report-format json \
--verbose || true
# Convert to GitLab SARIF format for Security Dashboard
- |
gitleaks detect \
--source . \
--report-path $SARIF_PATH \
--report-format sarif \
--verbose || true
# Check if secrets were found
- |
if [ -s "$REPORT_PATH" ] && [ "$(cat $REPORT_PATH)" != "null" ]; then
echo "⚠️ Secrets detected! Review findings below."
cat $REPORT_PATH | jq -r '.[] | "File: \(.File)\nLine: \(.StartLine)\nRule: \(.RuleID)\n"'
exit 1
else
echo "✅ No secrets detected"
fi
artifacts:
paths:
- $REPORT_PATH
- $SARIF_PATH
reports:
# GitLab Security Dashboard integration
secret_detection: $SARIF_PATH
when: always
expire_in: 30 days
# Allow failure for initial rollout, then set to false
allow_failure: false
# Optional: Incremental scanning with baseline
gitleaks-incremental:
stage: security
image: $GITLEAKS_IMAGE
# Only run on merge requests
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
script:
# Download baseline from artifacts or storage
- echo "Downloading baseline..."
- |
if [ -f ".gitleaks-baseline.json" ]; then
echo "Using baseline from repository"
else
echo "No baseline found, running full scan"
fi
# Run incremental scan
- |
if [ -f ".gitleaks-baseline.json" ]; then
gitleaks detect \
--source . \
--baseline-path .gitleaks-baseline.json \
--report-path new-findings.json \
--report-format json \
--exit-code 1 || true
if [ -s "new-findings.json" ] && [ "$(cat new-findings.json)" != "null" ]; then
echo "⚠️ New secrets detected since baseline!"
cat new-findings.json | jq .
exit 1
fi
fi
artifacts:
paths:
- new-findings.json
when: always
expire_in: 7 days
# Optional: Create baseline on main branch
create-baseline:
stage: security
image: $GITLEAKS_IMAGE
# Only run on main/master branch
rules:
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
when: manual # Manual trigger to avoid overwriting
script:
- echo "Creating new baseline..."
- |
gitleaks detect \
--source . \
--report-path .gitleaks-baseline.json \
--report-format json \
--exit-code 0 || true
artifacts:
paths:
- .gitleaks-baseline.json
expire_in: 365 days
# Optional: Generate human-readable report
generate-report:
stage: report
image: python:3.11-slim
dependencies:
- gitleaks-scan
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
script:
- pip install jinja2
- |
python3 << 'EOF'
import json
import sys
from datetime import datetime
try:
with open('gitleaks-report.json', 'r') as f:
findings = json.load(f)
if not findings:
print("✅ No secrets detected")
sys.exit(0)
print("# Gitleaks Secret Detection Report")
print(f"\n**Generated**: {datetime.now().isoformat()}")
print(f"**Total Findings**: {len(findings)}\n")
for idx, finding in enumerate(findings, 1):
print(f"\n## Finding {idx}")
print(f"- **File**: {finding.get('File', 'unknown')}")
print(f"- **Line**: {finding.get('StartLine', 'unknown')}")
print(f"- **Rule**: {finding.get('RuleID', 'unknown')}")
print(f"- **Description**: {finding.get('Description', 'unknown')}")
print(f"- **Commit**: {finding.get('Commit', 'N/A')}\n")
except FileNotFoundError:
print("No report file found")
except json.JSONDecodeError:
print("No findings in report")
EOF
artifacts:
paths:
- gitleaks-report.json
# Optional: Comment on merge request
comment-mr:
stage: report
image: alpine:latest
dependencies:
- gitleaks-scan
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
before_script:
- apk add --no-cache curl jq
script:
- |
if [ -s "$REPORT_PATH" ] && [ "$(cat $REPORT_PATH)" != "null" ]; then
FINDING_COUNT=$(cat $REPORT_PATH | jq '. | length')
COMMENT="## 🔒 Secret Scanning Results\n\n"
COMMENT="${COMMENT}⚠️ **${FINDING_COUNT} potential secret(s) detected!**\n\n"
COMMENT="${COMMENT}Please review the findings and take immediate action:\n"
COMMENT="${COMMENT}1. **Do not merge** this MR until secrets are removed\n"
COMMENT="${COMMENT}2. Rotate any exposed credentials immediately\n"
COMMENT="${COMMENT}3. Remove secrets from code and use CI/CD variables\n\n"
COMMENT="${COMMENT}See pipeline artifacts for detailed findings."
# Post comment to merge request
curl --request POST \
--header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
--data-urlencode "body=$COMMENT" \
"$CI_API_V4_URL/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
fi
allow_failure: true
# Optional: Scheduled nightly scan
nightly-scan:
stage: security
image: $GITLEAKS_IMAGE
# Run on schedule only
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
script:
- echo "Running comprehensive nightly secret scan..."
- |
gitleaks detect \
--source . \
--report-path nightly-scan.json \
--report-format json \
--verbose
artifacts:
paths:
- nightly-scan.json
when: always
expire_in: 90 days
# Send notifications on failure
after_script:
- |
if [ $? -ne 0 ]; then
echo "Secrets detected in nightly scan!"
# Add notification logic (email, Slack, etc.)
fi

70
skills/devsecops/secrets-gitleaks/assets/precommit-config.yaml Normal file
View File

@@ -0,0 +1,70 @@
# Pre-commit Framework Configuration for Gitleaks
# Install pre-commit: pip install pre-commit
# Install hooks: pre-commit install
# Run manually: pre-commit run --all-files
#
# More info: https://pre-commit.com/
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.18.0 # Update to latest version: https://github.com/gitleaks/gitleaks/releases
hooks:
- id: gitleaks
name: Gitleaks - Secret Detection
description: Scan staged changes for hardcoded secrets
entry: gitleaks protect --verbose --redact --staged
language: system
pass_filenames: false
# Optional: Custom configuration
# args: ['--config', '.gitleaks.toml']
# Optional: Additional security hooks
# Detect private keys
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: detect-private-key
name: Detect Private Keys
# Check for AWS credentials
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: detect-aws-credentials
name: Detect AWS Credentials
args: ['--allow-missing-credentials']
# Prevent large files (may contain secrets)
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: check-added-large-files
name: Check for Large Files
args: ['--maxkb=1000']
# Check for merge conflicts
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: check-merge-conflict
name: Check for Merge Conflicts
# Ensure files end with newline
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: end-of-file-fixer
name: Fix End of Files
# Trim trailing whitespace
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: trailing-whitespace
name: Trim Trailing Whitespace
# Configuration for pre-commit.ci (optional CI service)
ci:
autofix_prs: false
autoupdate_schedule: monthly

40
skills/devsecops/secrets-gitleaks/references/EXAMPLE.md Normal file
View File

@@ -0,0 +1,40 @@
# Reference Document Template
This file contains detailed reference material that Claude should load only when needed.
## Table of Contents
- [Section 1](#section-1)
- [Section 2](#section-2)
- [Security Standards](#security-standards)
## Section 1
Detailed information, schemas, or examples that are too large for SKILL.md.
## Section 2
Additional reference material.
## Security Standards
### OWASP Top 10
Reference relevant OWASP categories:
- A01: Broken Access Control
- A02: Cryptographic Failures
- etc.
### CWE Mappings
Map to relevant Common Weakness Enumeration categories:
- CWE-79: Cross-site Scripting
- CWE-89: SQL Injection
- etc.
### MITRE ATT&CK
Reference relevant tactics and techniques if applicable:
- TA0001: Initial Access
- T1190: Exploit Public-Facing Application
- etc.

538
skills/devsecops/secrets-gitleaks/references/compliance_mapping.md Normal file
View File

@@ -0,0 +1,538 @@
# Compliance Framework Mapping
Detailed mapping of Gitleaks secret detection to compliance and security frameworks.
## Table of Contents
- [OWASP Top 10](#owasp-top-10)
- [CWE (Common Weakness Enumeration)](#cwe-common-weakness-enumeration)
- [PCI-DSS](#pci-dss)
- [SOC 2](#soc-2)
- [GDPR](#gdpr)
- [NIST Cybersecurity Framework](#nist-cybersecurity-framework)
- [ISO 27001](#iso-27001)
- [HIPAA](#hipaa)
- [Compliance Reporting](#compliance-reporting)
## OWASP Top 10
### A07:2021 – Identification and Authentication Failures
**Relevance**: Hardcoded credentials lead to authentication bypass and unauthorized access.
**Gitleaks Coverage**:
- Detects hardcoded passwords, API keys, tokens
- Identifies database connection strings with embedded credentials
- Finds SSH keys, certificates, and cryptographic secrets
**Control Implementation**:
```yaml
# CI/CD check to prevent authentication failures
name: OWASP A07 - Authentication Control
on: [push, pull_request]
jobs:
secrets-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Scan for hardcoded credentials (OWASP A07)
uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
**Evidence for Auditors**:
- Gitleaks scan reports (JSON/SARIF format)
- CI/CD pipeline logs showing regular scans
- Pre-commit hook installation across developer workstations
- Remediation tracking for detected secrets
### A01:2021 – Broken Access Control
**Relevance**: Exposed API keys and tokens can bypass access control mechanisms.
**Gitleaks Coverage**:
- Cloud provider credentials (AWS, GCP, Azure)
- Service account keys and OAuth tokens
- Administrative API keys
**Control Implementation**:
- Implement secret scanning before deployment
- Rotate credentials when exposure detected
- Review cloud provider audit logs for unauthorized access
### A02:2021 – Cryptographic Failures
**Relevance**: Hardcoded cryptographic keys compromise encryption.
**Gitleaks Coverage**:
- Private keys (RSA, DSA, EC)
- JWT signing secrets
- Encryption keys in configuration files
**Evidence**:
- Detection rules for CWE-321 (Use of Hard-coded Cryptographic Key)
- Remediation procedures for exposed cryptographic material
## CWE (Common Weakness Enumeration)
### CWE-798: Use of Hard-coded Credentials
**Description**: Software contains hard-coded credentials (e.g., password, cryptographic key).
**CVSS Base Score**: Typically 7.5 - 9.8 (High to Critical)
**Gitleaks Detection**:
- All API key rules
- Database connection strings
- Service account credentials
- Generic password patterns
**Remediation Mapping**:
```toml
# Tag all findings with CWE-798
[[rules]]
id = "generic-api-key"
description = "Generic API Key (CWE-798)"
regex = '''(?i)api_key\s*=\s*["']([a-zA-Z0-9]{32,})["']'''
tags = ["api-key", "CWE-798"]
```
### CWE-259: Use of Hard-coded Password
**Description**: Software contains hard-coded password.
**Gitleaks Detection**:
- Password variables in code
- Database connection strings with passwords
- Configuration files with password fields
**Example Finding**:
```json
{
"RuleID": "generic-password",
"Description": "Hard-coded password detected",
"File": "config/database.py",
"Line": 42,
"CWE": "CWE-259"
}
```
### CWE-321: Use of Hard-coded Cryptographic Key
**Description**: Use of hard-coded cryptographic key in product.
**Gitleaks Detection**:
- Private key files (PEM format)
- JWT signing secrets
- Encryption keys in source code
### CWE-522: Insufficiently Protected Credentials
**Description**: Product transmits or stores authentication credentials in insufficiently protected form.
**Gitleaks Coverage**: Detects credentials stored in source code (inadequate protection).
### CWE-257: Storing Passwords in a Recoverable Format
**Description**: Storing passwords in a recoverable format makes them vulnerable.
**Gitleaks Coverage**: Identifies plaintext passwords in configuration and code.
## PCI-DSS
### Requirement 6.5.3: Insecure Cryptographic Storage
**Control Objective**: Protect stored cardholder data.
**Gitleaks Implementation**:
- Scan payment processing code for embedded API keys (Stripe, PayPal, etc.)
- Detect hardcoded encryption keys
- Identify database credentials used for cardholder data access
**Compliance Evidence**:
```bash
# Generate PCI-DSS compliance report
gitleaks detect \
--source ./payment-processing \
--report-format json \
--report-path pci-compliance-scan.json
# Extract payment-related findings
jq '.[] | select(.Tags[] | contains("payment"))' pci-compliance-scan.json
```
### Requirement 8.2.1: Strong Cryptography for Authentication
**Control Objective**: Use strong authentication credentials.
**Gitleaks Implementation**:
- Detect weak/hardcoded authentication tokens
- Identify test credentials in production code paths
### Requirement 10.2: Logging and Monitoring
**Control Objective**: Implement automated audit trails.
**Gitleaks Implementation**:
```python
# Log all secret detection events
import logging
import json
with open('gitleaks-findings.json', 'r') as f:
findings = json.load(f)
for finding in findings:
logging.warning(
f"PCI-DSS Violation: Hardcoded credential detected",
extra={
"rule": finding["RuleID"],
"file": finding["File"],
"line": finding["StartLine"],
"compliance_requirement": "PCI-DSS 6.5.3"
}
)
```
### PCI-DSS Reporting Template
```markdown
# PCI-DSS Requirement 6.5.3 - Secret Scanning Report
**Reporting Period**: Q1 2024
**Scan Date**: 2024-01-15
**Scope**: All repositories handling cardholder data
## Summary
- **Repositories Scanned**: 15
- **Secrets Detected**: 3
- **Remediation Status**: All resolved within 24 hours
## Findings
| Finding ID | Rule | Severity | File | Status | Remediation Date |
|------------|------|----------|------|--------|------------------|
| F001 | stripe-api-key | CRITICAL | payment/config.py | Resolved | 2024-01-15 |
| F002 | database-password | HIGH | db/setup.sql | Resolved | 2024-01-15 |
| F003 | aws-access-key | HIGH | deploy/config.yml | Resolved | 2024-01-16 |
## Control Effectiveness
✅ Automated secret scanning implemented
✅ All findings remediated within SLA
✅ Pre-commit hooks prevent new violations
```
## SOC 2
### CC6.1: Logical and Physical Access Controls
**Control Activity**: Implement controls to prevent unauthorized access to system resources.
**Gitleaks Implementation**:
- Automated detection of exposed credentials
- Pre-commit hooks to prevent credential commits
- CI/CD gates blocking deployments with secrets
**SOC 2 Evidence Package**:
1. Secret scanning policy and procedures
2. Gitleaks configuration file (`.gitleaks.toml`)
3. CI/CD pipeline configurations
4. Scan execution logs (last 12 months)
5. Remediation tracking (issue tickets)
6. Training materials for developers
### CC6.6: Logical Access - Provisioning
**Control Activity**: Provision access based on role, revoke when no longer needed.
**Gitleaks Implementation**:
- Detection of service account keys and tokens
- Audit trail of credential exposure and rotation
- Automated revocation workflows
### CC7.2: System Monitoring
**Control Activity**: Monitor system for security events and anomalies.
**Gitleaks Implementation**:
```yaml
# Continuous monitoring workflow
name: SOC2 CC7.2 - Security Monitoring
on:
schedule:
- cron: '0 2 * * *' # Daily at 2 AM
jobs:
security-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Secret Detection Scan
uses: gitleaks/gitleaks-action@v2
- name: Report to SIEM
run: |
curl -X POST https://siem.company.com/api/events \
-H "Content-Type: application/json" \
-d @gitleaks-report.json
```
### SOC 2 Audit Response Template
```markdown
# SOC 2 Control CC6.1 - Secret Scanning Control
**Control Description**: Automated secret scanning prevents unauthorized access through exposed credentials.
**Control Design**:
1. Pre-commit hooks block credential commits at developer workstation
2. CI/CD pipeline scans all pull requests before merge
3. Nightly scans of all production repositories
4. Automated alerting to security team for violations
**Control Operating Effectiveness**:
- **Frequency**: Continuous (pre-commit) + Daily (scheduled scans)
- **Population**: 247 repositories, 85 developers
- **Sample Period**: January 1 - December 31, 2024
- **Samples Tested**: 52 weekly scan reports
- **Exceptions**: 0
**Evidence of Operation**:
- Attached: gitleaks-audit-log-2024.json
- Attached: remediation-tracking.csv
- Attached: developer-training-records.pdf
```
## GDPR
### Article 32: Security of Processing
**Requirement**: Implement appropriate technical measures to ensure security of personal data.
**Gitleaks Implementation**:
- Detect API keys for services processing personal data
- Identify database credentials for systems storing personal data
- Scan for OAuth tokens with user data access scopes
**GDPR Compliance Mapping**:
| GDPR Requirement | Gitleaks Control | Evidence |
|------------------|------------------|----------|
| Art. 32(1)(a) - Pseudonymization | Detect database credentials protecting personal data | Scan reports |
| Art. 32(1)(b) - Confidentiality | Prevent credential exposure in source code | Pre-commit hooks |
| Art. 32(2) - Risk Assessment | Regular security scanning | Scan schedules |
| Art. 33 - Breach Notification | Detection triggers incident response | Alert logs |
### Data Breach Notification
If Gitleaks detects exposed credentials accessing personal data:
```bash
#!/bin/bash
# gdpr-incident-response.sh
# Assess if personal data is at risk
echo "1. Identify data accessed by exposed credential"
echo "2. Determine if data is personal data under GDPR"
echo "3. Assess likelihood of unauthorized access"
# 72-hour notification requirement
echo "If personal data breach confirmed:"
echo "- Notify supervisory authority within 72 hours"
echo "- Document: nature of breach, data categories affected, likely consequences, measures taken"
```
## NIST Cybersecurity Framework
### Identify (ID.AM): Asset Management
**Subcategory**: ID.AM-2 - Software platforms and applications are inventoried.
**Gitleaks Implementation**: Catalog all repositories with secret scanning coverage.
### Protect (PR.AC): Access Control
**Subcategory**: PR.AC-1 - Identities and credentials are managed.
**Gitleaks Implementation**:
- Automated detection of exposed credentials
- Credential lifecycle management (rotation after exposure)
### Detect (DE.CM): Security Continuous Monitoring
**Subcategory**: DE.CM-4 - Malicious code is detected.
**Gitleaks Implementation**: Secrets considered "malicious" when hardcoded.
### Respond (RS.AN): Analysis
**Subcategory**: RS.AN-1 - Notifications are investigated.
**Gitleaks Implementation**: Alert triage and investigation procedures.
### Recover (RC.RP): Recovery Planning
**Subcategory**: RC.RP-1 - Recovery plan is executed during or after an event.
**Gitleaks Implementation**: Credential rotation and git history cleanup procedures.
## ISO 27001
### A.9.2.4: Management of Secret Authentication Information
**Control**: Allocation of secret authentication information shall be controlled through a formal management process.
**Gitleaks Implementation**:
- Detect deviations from secret management process (hardcoded secrets)
- Enforce secret management policy through pre-commit hooks
### A.9.4.3: Password Management System
**Control**: Password management systems shall be interactive and ensure quality passwords.
**Gitleaks Implementation**: Prevent password hardcoding in source code.
### A.12.6.1: Management of Technical Vulnerabilities
**Control**: Obtain information about technical vulnerabilities and take appropriate measures.
**Gitleaks Implementation**: Continuous vulnerability scanning for credential exposure.
## HIPAA
### § 164.312(a)(1): Access Control
**Standard**: Implement technical policies to allow only authorized access to ePHI.
**Gitleaks Implementation**:
- Detect credentials for systems accessing ePHI
- Prevent unauthorized access through exposed credentials
### § 164.308(a)(1)(ii)(D): Information System Activity Review
**Standard**: Implement procedures to regularly review records of information system activity.
**Gitleaks Implementation**:
```bash
# Weekly HIPAA compliance review
gitleaks detect \
--source ./healthcare-systems \
--report-format json \
> hipaa-weekly-scan.json
# Review findings for ePHI access credentials
jq '.[] | select(.Tags[] | contains("database") or contains("api-key"))' \
hipaa-weekly-scan.json
```
### § 164.312(b): Audit Controls
**Standard**: Implement hardware, software, procedures to record and examine system activity.
**Gitleaks Implementation**: Audit trail of secret detection events.
## Compliance Reporting
### Automated Compliance Report Generation
```python
#!/usr/bin/env python3
"""Generate compliance report from Gitleaks findings."""
import json
import sys
from datetime import datetime
# Compliance framework mappings
COMPLIANCE_MAPPINGS = {
"CWE-798": ["OWASP-A07", "PCI-DSS-6.5.3", "SOC2-CC6.1", "ISO27001-A.9.2.4"],
"CWE-259": ["OWASP-A07", "PCI-DSS-8.2.1", "SOC2-CC6.1", "ISO27001-A.9.4.3"],
"CWE-321": ["OWASP-A02", "PCI-DSS-6.5.3", "ISO27001-A.12.3.1"],
}
def generate_compliance_report(findings_file, framework):
"""Generate compliance-specific report."""
with open(findings_file, 'r') as f:
findings = json.load(f)
# Filter findings relevant to framework
relevant_findings = []
for finding in findings:
cwe = finding.get("CWE", "")
if framework in COMPLIANCE_MAPPINGS.get(cwe, []):
relevant_findings.append(finding)
# Generate report
report = {
"framework": framework,
"generated": datetime.now().isoformat(),
"total_findings": len(relevant_findings),
"findings": relevant_findings,
"compliance_status": "NON-COMPLIANT" if relevant_findings else "COMPLIANT"
}
return report
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: compliance_report.py <findings.json> <framework>")
print("Frameworks: OWASP, PCI-DSS, SOC2, ISO27001, GDPR, HIPAA")
sys.exit(1)
report = generate_compliance_report(sys.argv[1], sys.argv[2])
print(json.dumps(report, indent=2))
```
### Usage
```bash
# Generate PCI-DSS specific report
./compliance_report.py gitleaks-findings.json PCI-DSS > pci-dss-report.json
# Generate SOC2 specific report
./compliance_report.py gitleaks-findings.json SOC2 > soc2-report.json
```
### Compliance Dashboard Metrics
Track these KPIs for compliance reporting:
```yaml
metrics:
- name: "Secret Detection Coverage"
description: "Percentage of repositories with secret scanning enabled"
target: 100%
- name: "Mean Time to Remediation (MTTR)"
description: "Average time from detection to credential rotation"
target: < 4 hours
- name: "False Positive Rate"
description: "Percentage of findings classified as false positives"
target: < 10%
- name: "Pre-commit Hook Adoption"
description: "Percentage of developers with hooks installed"
target: > 95%
- name: "Scan Frequency"
description: "Scans per repository per month"
target: > 30 (daily)
```
## Audit Preparation Checklist
- [ ] Configure Gitleaks across all in-scope repositories
- [ ] Implement CI/CD secret scanning gates
- [ ] Deploy pre-commit hooks to developer workstations
- [ ] Establish remediation procedures and SLAs
- [ ] Create audit trail (scan logs, remediation tickets)
- [ ] Generate compliance-specific reports
- [ ] Document control design and operating effectiveness
- [ ] Prepare evidence package for auditors
- [ ] Train team on secret management policies
- [ ] Schedule regular compliance reviews

276
skills/devsecops/secrets-gitleaks/references/detection_rules.md Normal file
View File

@@ -0,0 +1,276 @@
# Gitleaks Detection Rules Reference
Comprehensive reference of built-in Gitleaks detection rules with CWE mappings and remediation guidance.
## Table of Contents
- [Cloud Provider Credentials](#cloud-provider-credentials)
- [Version Control Systems](#version-control-systems)
- [API Keys and Tokens](#api-keys-and-tokens)
- [Database Credentials](#database-credentials)
- [Private Keys](#private-keys)
- [Generic Patterns](#generic-patterns)
## Cloud Provider Credentials
### AWS Access Key ID
- **Rule ID**: `aws-access-token`
- **Pattern**: `AKIA[0-9A-Z]{16}`
- **CWE**: CWE-798 (Use of Hard-coded Credentials)
- **Severity**: HIGH
- **Description**: AWS Access Key ID for programmatic access
- **Remediation**: Rotate via AWS IAM console, use AWS Secrets Manager or IAM roles
### AWS Secret Access Key
- **Rule ID**: `aws-secret-key`
- **Pattern**: `(?i)aws(.{0,20})?[\'\"][0-9a-zA-Z\/+]{40}[\'\"]`
- **CWE**: CWE-798
- **Severity**: CRITICAL
- **Description**: AWS Secret Access Key paired with Access Key ID
- **Remediation**: Immediate rotation required, review CloudTrail logs for unauthorized access
### GCP API Key
- **Rule ID**: `gcp-api-key`
- **Pattern**: `AIza[0-9A-Za-z\\-_]{35}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: Google Cloud Platform API key
- **Remediation**: Delete and regenerate in GCP Console, review API usage logs
### GCP Service Account
- **Rule ID**: `gcp-service-account`
- **Pattern**: `\"type\": \"service_account\"`
- **CWE**: CWE-798
- **Severity**: CRITICAL
- **Description**: GCP service account JSON key file
- **Remediation**: Delete service account key, use Workload Identity where possible
### Azure Storage Account Key
- **Rule ID**: `azure-storage-key`
- **Pattern**: `(?i)azure.*[\'\"][0-9a-zA-Z\/+]{88}[\'\"]`
- **CWE**: CWE-798
- **Severity**: CRITICAL
- **Description**: Azure Storage Account access key
- **Remediation**: Regenerate keys in Azure Portal, use Azure Key Vault
### Digital Ocean Token
- **Rule ID**: `digitalocean-token`
- **Pattern**: `dop_v1_[a-f0-9]{64}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: Digital Ocean personal access token
- **Remediation**: Revoke token in Digital Ocean console, create new token
## Version Control Systems
### GitHub Personal Access Token
- **Rule ID**: `github-pat`
- **Pattern**: `ghp_[0-9a-zA-Z]{36}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: GitHub personal access token (classic)
- **Remediation**: Revoke in GitHub Settings > Developer settings, review audit log
### GitHub OAuth Token
- **Rule ID**: `github-oauth`
- **Pattern**: `gho_[0-9a-zA-Z]{36}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: GitHub OAuth access token
- **Remediation**: Revoke OAuth app authorization, regenerate token
### GitHub Fine-Grained Token
- **Rule ID**: `github-fine-grained-pat`
- **Pattern**: `github_pat_[0-9a-zA-Z]{22}_[0-9a-zA-Z]{59}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: GitHub fine-grained personal access token
- **Remediation**: Revoke in GitHub Settings, review resource access scope
### GitLab Personal Access Token
- **Rule ID**: `gitlab-pat`
- **Pattern**: `glpat-[0-9a-zA-Z\\-_]{20}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: GitLab personal access token
- **Remediation**: Revoke in GitLab User Settings > Access Tokens
### Bitbucket App Password
- **Rule ID**: `bitbucket-app-password`
- **Pattern**: `(?i)bitbucket.*[\'\"][0-9a-zA-Z]{16}[\'\"]`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: Bitbucket app-specific password
- **Remediation**: Revoke in Bitbucket Personal Settings > App passwords
## API Keys and Tokens
### Stripe API Key
- **Rule ID**: `stripe-api-key`
- **Pattern**: `(?i)(sk|pk)_(test|live)_[0-9a-zA-Z]{24,}`
- **CWE**: CWE-798
- **Severity**: CRITICAL (live), HIGH (test)
- **Description**: Stripe API secret or publishable key
- **Remediation**: Roll keys in Stripe Dashboard, review payment transactions
### Twilio API Key
- **Rule ID**: `twilio-api-key`
- **Pattern**: `SK[0-9a-fA-F]{32}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: Twilio API key
- **Remediation**: Delete key in Twilio Console, create new key
### SendGrid API Key
- **Rule ID**: `sendgrid-api-key`
- **Pattern**: `SG\\.[0-9A-Za-z\\-_]{22}\\.[0-9A-Za-z\\-_]{43}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: SendGrid API key
- **Remediation**: Delete in SendGrid Settings > API Keys, update applications
### Slack Token
- **Rule ID**: `slack-token`
- **Pattern**: `xox[baprs]-[0-9]{10,13}-[0-9]{10,13}-[a-zA-Z0-9]{24,}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: Slack bot, app, or user token
- **Remediation**: Regenerate in Slack App Settings, rotate token
### Slack Webhook
- **Rule ID**: `slack-webhook`
- **Pattern**: `https://hooks\\.slack\\.com/services/T[a-zA-Z0-9_]+/B[a-zA-Z0-9_]+/[a-zA-Z0-9_]+`
- **CWE**: CWE-798
- **Severity**: MEDIUM
- **Description**: Slack incoming webhook URL
- **Remediation**: Regenerate webhook in Slack App Settings
### npm Token
- **Rule ID**: `npm-access-token`
- **Pattern**: `npm_[0-9a-zA-Z]{36}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: npm access token
- **Remediation**: Revoke in npm Account Settings, check package publish history
### PyPI Token
- **Rule ID**: `pypi-upload-token`
- **Pattern**: `pypi-AgEIcHlwaS5vcmc[0-9A-Za-z\\-_]{50,}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: PyPI upload token
- **Remediation**: Delete token in PyPI Account Settings, verify package uploads
## Database Credentials
### PostgreSQL Connection String
- **Rule ID**: `postgres-connection-string`
- **Pattern**: `postgres(ql)?://[a-zA-Z0-9]+:[a-zA-Z0-9]+@[a-zA-Z0-9.-]+:[0-9]+/[a-zA-Z0-9_-]+`
- **CWE**: CWE-798
- **Severity**: CRITICAL
- **Description**: PostgreSQL database connection string with embedded credentials
- **Remediation**: Change database password, use connection string from environment variables
### MySQL Connection String
- **Rule ID**: `mysql-connection-string`
- **Pattern**: `mysql://[a-zA-Z0-9]+:[a-zA-Z0-9]+@[a-zA-Z0-9.-]+:[0-9]+/[a-zA-Z0-9_-]+`
- **CWE**: CWE-259
- **Severity**: CRITICAL
- **Description**: MySQL database connection string with embedded credentials
- **Remediation**: Rotate database password immediately, review access logs
### MongoDB Connection String
- **Rule ID**: `mongodb-connection-string`
- **Pattern**: `mongodb(\+srv)?://[a-zA-Z0-9]+:[a-zA-Z0-9]+@[a-zA-Z0-9.-]+`
- **CWE**: CWE-798
- **Severity**: CRITICAL
- **Description**: MongoDB connection string with credentials
- **Remediation**: Change MongoDB user password, enable IP whitelisting
### Redis URL
- **Rule ID**: `redis-url`
- **Pattern**: `redis://:[a-zA-Z0-9]+@[a-zA-Z0-9.-]+:[0-9]+`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: Redis connection URL with password
- **Remediation**: Change Redis password via CONFIG SET, use ACLs
## Private Keys
### RSA Private Key
- **Rule ID**: `rsa-private-key`
- **Pattern**: `-----BEGIN RSA PRIVATE KEY-----`
- **CWE**: CWE-321 (Use of Hard-coded Cryptographic Key)
- **Severity**: CRITICAL
- **Description**: RSA private key in PEM format
- **Remediation**: Generate new key pair, revoke associated certificates, audit access
### SSH Private Key
- **Rule ID**: `ssh-private-key`
- **Pattern**: `-----BEGIN (EC|DSA|OPENSSH) PRIVATE KEY-----`
- **CWE**: CWE-321
- **Severity**: CRITICAL
- **Description**: SSH private key
- **Remediation**: Remove from authorized_keys on all servers, generate new key
### PGP Private Key
- **Rule ID**: `pgp-private-key`
- **Pattern**: `-----BEGIN PGP PRIVATE KEY BLOCK-----`
- **CWE**: CWE-321
- **Severity**: CRITICAL
- **Description**: PGP/GPG private key
- **Remediation**: Revoke key on keyservers, generate new key pair
### JWT Token
- **Rule ID**: `jwt`
- **Pattern**: `eyJ[A-Za-z0-9_-]{10,}\\.[A-Za-z0-9_-]{10,}\\.[A-Za-z0-9_-]{10,}`
- **CWE**: CWE-798
- **Severity**: HIGH
- **Description**: JSON Web Token (may contain sensitive claims)
- **Remediation**: Invalidate token, check token expiration, rotate signing secret
## Generic Patterns
### Generic API Key
- **Rule ID**: `generic-api-key`
- **Pattern**: `(?i)(api_key|apikey|api-key)[\s]*[=:][\s]*[\'\"]?[a-zA-Z0-9]{32,}[\'\"]?`
- **CWE**: CWE-798
- **Severity**: MEDIUM
- **Description**: Generic API key pattern
- **Remediation**: Rotate credential based on service documentation
### Generic Secret
- **Rule ID**: `generic-secret`
- **Pattern**: `(?i)(secret|password|passwd|pwd)[\s]*[=:][\s]*[\'\"]?[a-zA-Z0-9!@#$%^&*]{16,}[\'\"]?`
- **CWE**: CWE-259
- **Severity**: MEDIUM
- **Description**: Generic secret or password pattern
- **Remediation**: Move to environment variable or secret management system
### High Entropy String
- **Rule ID**: `high-entropy`
- **Pattern**: `[a-zA-Z0-9]{32,}`
- **Entropy**: 4.5+
- **CWE**: CWE-798
- **Severity**: LOW (requires validation)
- **Description**: High-entropy string that may be a credential
- **Remediation**: Validate if actual secret, rotate if necessary
## Usage in Configuration
Add these rule IDs to your `.gitleaks.toml` allowlist if needed:
```toml
[allowlist]
description = "Allow specific rules in test files"
paths = ['''test/''']
rules = ["generic-api-key", "generic-secret"]
```
## CWE Reference
- **CWE-798**: Use of Hard-coded Credentials
- **CWE-259**: Use of Hard-coded Password
- **CWE-321**: Use of Hard-coded Cryptographic Key
- **CWE-522**: Insufficiently Protected Credentials
- **CWE-257**: Storing Passwords in a Recoverable Format

598
skills/devsecops/secrets-gitleaks/references/false_positives.md Normal file
View File

@@ -0,0 +1,598 @@
# False Positives Management
Strategies for managing false positives in Gitleaks secret detection.
## Table of Contents
- [Understanding False Positives](#understanding-false-positives)
- [Allowlist Strategies](#allowlist-strategies)
- [Common False Positive Patterns](#common-false-positive-patterns)
- [Configuration Examples](#configuration-examples)
- [Best Practices](#best-practices)
## Understanding False Positives
False positives occur when legitimate code patterns match secret detection rules.
### Categories of False Positives
1. **Example/Placeholder Values**: Documentation and examples using fake credentials
2. **Test Fixtures**: Test data with credential-like patterns
3. **Non-Secret Constants**: Configuration values that match patterns but aren't sensitive
4. **Generated Code**: Auto-generated code with high-entropy strings
5. **Comments and Documentation**: Explanatory text matching patterns
### Impact Assessment
Before allowlisting, verify it's truly a false positive:
```bash
# Extract the flagged value
echo "api_key_here" | base64 # Check if valid encoding
curl -H "Authorization: Bearer <token>" https://api.service.com/test # Test if active
# Check git history for when added
git log -p --all -S "flagged_value"
# Review context around detection
git show <commit-sha>:<file-path>
```
## Allowlist Strategies
### 1. Path-Based Allowlisting
Exclude entire directories or file patterns:
```toml
[allowlist]
description = "Exclude test and documentation files"
paths = [
'''test/.*''', # All test directories
'''tests/.*''', # Alternative test directory name
'''.*/fixtures/.*''', # Test fixtures anywhere
'''examples/.*''', # Example code
'''docs/.*''', # Documentation
'''.*\.md$''', # Markdown files
'''.*\.rst$''', # ReStructuredText files
'''.*_test\.go$''', # Go test files
'''.*\.test\.js$''', # JavaScript test files
'''.*\.spec\.ts$''', # TypeScript spec files
]
```
### 2. Stopword Allowlisting
Filter out known placeholder values:
```toml
[allowlist]
description = "Common placeholder values"
stopwords = [
"example",
"placeholder",
"your_api_key_here",
"your_secret_here",
"REPLACEME",
"CHANGEME",
"xxxxxx",
"000000",
"123456",
"abcdef",
"sample",
"dummy",
"fake",
"test_key",
"mock_token",
]
```
### 3. Commit-Based Allowlisting
Allowlist specific commits after manual verification:
```toml
[allowlist]
description = "Verified false positives"
commits = [
"a1b2c3d4e5f6", # Initial test fixtures - verified 2024-01-15
"f6e5d4c3b2a1", # Documentation examples - verified 2024-01-16
]
```
Add comment explaining why each commit is allowlisted.
### 4. Regex Allowlisting
Allowlist specific patterns:
```toml
[allowlist]
description = "Pattern-based allowlist"
regexes = [
'''example_api_key_[0-9]+''', # Example keys with numeric suffix
'''key\s*=\s*["']EXAMPLE["']''', # Explicitly marked examples
'''(?i)test_?password_?[0-9]*''', # Test passwords
'''(?i)dummy.*secret''', # Dummy secrets
]
```
### 5. Rule-Specific Allowlisting
Create exceptions for specific rules only:
```toml
[[rules]]
id = "generic-api-key"
description = "Generic API Key"
regex = '''(?i)api_key\s*=\s*["']([a-zA-Z0-9]{32})["']'''
[rules.allowlist]
description = "Allow generic API key pattern in specific contexts"
paths = ['''config/defaults\.yaml''']
regexes = ['''api_key\s*=\s*["']example''']
```
### 6. Global vs Rule Allowlists
Global allowlists override rule-specific ones:
```toml
# Global allowlist - highest precedence
[allowlist]
description = "Organization-wide exceptions"
paths = ['''vendor/''', '''node_modules/''']
# Rule-specific allowlist
[[rules]]
id = "custom-secret"
[rules.allowlist]
description = "Exceptions only for this rule"
paths = ['''config/template\.yml''']
```
## Common False Positive Patterns
### 1. Documentation Examples
**Problem**: README and documentation contain example credentials.
**Solution**:
```toml
[allowlist]
paths = [
'''README\.md$''',
'''CONTRIBUTING\.md$''',
'''docs/.*\.md$''',
'''.*\.example$''', # .env.example files
'''.*\.template$''', # Template files
'''.*\.sample$''', # Sample configurations
]
stopwords = [
"example.com",
"user@example.org",
"YOUR_API_KEY",
]
```
### 2. Test Fixtures
**Problem**: Test data contains credential-like strings for testing credential handling.
**Solution**:
```toml
[allowlist]
paths = [
'''test/fixtures/.*''',
'''spec/fixtures/.*''',
'''.*/testdata/.*''', # Go convention
'''.*/mocks/.*''',
'''cypress/fixtures/.*''', # Cypress test data
]
# Or use inline comments in code
# password = "test_password_123" # gitleaks:allow
```
### 3. Generated Code
**Problem**: Code generators produce high-entropy identifiers.
**Solution**:
```toml
[allowlist]
description = "Generated code"
paths = [
'''.*\.pb\.go$''', # Protocol buffer generated code
'''.*_generated\..*''', # Generated file marker
'''node_modules/.*''', # Dependencies
'''vendor/.*''', # Vendored dependencies
'''dist/.*''', # Build output
'''build/.*''',
]
```
### 4. Configuration Templates
**Problem**: Config templates with placeholder values match patterns.
**Solution**:
```toml
[allowlist]
paths = [
'''config/.*\.template''',
'''templates/.*''',
'''.*\.tpl$''',
'''.*\.tmpl$''',
]
stopwords = [
"REPLACE_WITH_YOUR",
"CONFIGURE_ME",
"SET_THIS_VALUE",
]
```
### 5. Base64 Encoded Strings
**Problem**: Non-secret base64 data flagged due to high entropy.
**Solution**:
```toml
# Increase entropy threshold to reduce false positives
[[rules]]
id = "high-entropy-base64"
regex = '''[a-zA-Z0-9+/]{40,}={0,2}'''
entropy = 5.5 # Increase from default 4.5
```
Or allowlist specific patterns:
```toml
[allowlist]
regexes = [
'''data:image/[^;]+;base64,''', # Base64 encoded images
'''-----BEGIN CERTIFICATE-----''', # Public certificates (not private keys)
]
```
### 6. Public Keys and Certificates
**Problem**: Public keys detected (which are not secrets).
**Solution**:
```toml
[allowlist]
regexes = [
'''-----BEGIN PUBLIC KEY-----''',
'''-----BEGIN CERTIFICATE-----''',
'''-----BEGIN X509 CERTIFICATE-----''',
]
# But DO NOT allowlist:
# -----BEGIN PRIVATE KEY-----
# -----BEGIN RSA PRIVATE KEY-----
```
### 7. UUIDs and Identifiers
**Problem**: UUIDs match high-entropy patterns.
**Solution**:
```toml
[allowlist]
regexes = [
'''[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}''', # UUID
'''[0-9a-f]{24}''', # MongoDB ObjectId
]
```
Or adjust entropy detection:
```toml
[[rules]]
id = "generic-high-entropy"
entropy = 6.0 # Only flag very high entropy
```
## Configuration Examples
### Minimal Configuration
Start with broad allowlists, refine over time:
```toml
title = "Minimal Gitleaks Configuration"
[extend]
useDefault = true # Use all built-in rules
[allowlist]
description = "Broad allowlist for initial rollout"
paths = [
'''test/.*''',
'''.*\.md$''',
'''vendor/.*''',
'''node_modules/.*''',
]
stopwords = [
"example",
"test",
"mock",
"dummy",
]
```
### Strict Configuration
Minimize false positives with targeted allowlists:
```toml
title = "Strict Gitleaks Configuration"
[extend]
useDefault = true
[allowlist]
description = "Minimal allowlist - verify all exceptions"
# Only allow specific known false positives
paths = [
'''docs/api-examples\.md''', # API documentation with examples
'''test/fixtures/auth\.json''', # Authentication test fixtures
]
# Specific known placeholder values
stopwords = [
"YOUR_API_KEY_HERE",
"sk_test_example_key_123456789",
]
# Manually verified commits
commits = [
"abc123def456", # Test fixtures added - verified 2024-01-15 by security@company.com
]
```
### Balanced Configuration
Balance detection sensitivity with operational overhead:
```toml
title = "Balanced Gitleaks Configuration"
[extend]
useDefault = true
[allowlist]
description = "Balanced allowlist"
# Common non-secret paths
paths = [
'''test/fixtures/.*''',
'''spec/fixtures/.*''',
'''.*\.md$''',
'''docs/.*''',
'''examples/.*''',
'''vendor/.*''',
'''node_modules/.*''',
]
# Common placeholders
stopwords = [
"example",
"placeholder",
"your_key_here",
"replace_me",
"changeme",
"test",
"dummy",
"mock",
]
# Public non-secrets
regexes = [
'''-----BEGIN CERTIFICATE-----''',
'''-----BEGIN PUBLIC KEY-----''',
'''data:image/[^;]+;base64,''',
]
```
## Best Practices
### 1. Document Allowlist Decisions
Always add comments explaining why patterns are allowlisted:
```toml
[allowlist]
description = "Verified false positives - reviewed 2024-01-15"
# Test fixtures created during initial test suite development
# Contains only example credentials for testing credential validation
paths = ['''test/fixtures/credentials\.json''']
# Documentation examples using clearly fake values
# All examples prefixed with "example_" or "test_"
stopwords = ["example_", "test_"]
```
### 2. Regular Allowlist Review
Schedule periodic reviews:
```bash
#!/bin/bash
# review-allowlist.sh
echo "Gitleaks Allowlist Review"
echo "========================="
echo ""
# Show allowlist paths
echo "Allowlisted paths:"
grep -A 10 "^\[allowlist\]" .gitleaks.toml | grep "paths = "
# Show allowlisted commits
echo ""
echo "Allowlisted commits:"
grep -A 10 "^\[allowlist\]" .gitleaks.toml | grep "commits = "
# Check if commits still exist
# (May have been removed in history rewrite)
git rev-parse --verify abc123def456 2>/dev/null || echo "WARNING: Commit abc123def456 not found"
```
### 3. Use Inline Annotations Sparingly
For one-off false positives, use inline comments:
```python
# This is a test password for unit tests only
# gitleaks:allow
TEST_PASSWORD = "test_password_123"
```
**Warning**: Overuse of inline annotations indicates poorly tuned configuration.
### 4. Version Control Your Configuration
Track changes to `.gitleaks.toml`:
```bash
git log -p .gitleaks.toml
# See who allowlisted what and when
git blame .gitleaks.toml
```
### 5. Test Allowlist Changes
Before committing allowlist changes:
```bash
# Test configuration
gitleaks detect --config .gitleaks.toml -v
# Verify specific file is now allowed
gitleaks detect --config .gitleaks.toml --source test/fixtures/credentials.json
# Verify secret is still caught in production code
echo 'api_key = "sk_live_actual_key"' > /tmp/test_detection.py
gitleaks detect --config .gitleaks.toml --source /tmp/test_detection.py --no-git
```
### 6. Separate Allowlists by Environment
Use different configurations for different contexts:
```bash
# Strict config for production code
gitleaks detect --config .gitleaks.strict.toml --source src/
# Lenient config for test code
gitleaks detect --config .gitleaks.lenient.toml --source test/
```
### 7. Monitor False Positive Rate
Track metrics over time:
```bash
# Total findings
TOTAL=$(gitleaks detect --report-format json 2>/dev/null | jq '. | length')
# Run with allowlist
AFTER_FILTER=$(gitleaks detect --config .gitleaks.toml --report-format json 2>/dev/null | jq '. | length')
# Calculate reduction
echo "False positive reduction: $(($TOTAL - $AFTER_FILTER)) / $TOTAL"
```
**Target**: < 10% false positive rate for good developer experience.
### 8. Security Review for New Allowlists
Require security team approval for:
- New allowlisted paths in `src/` or production code
- New allowlisted commits (verify manually first)
- Changes to rule-specific allowlists
- New stopwords that could mask real secrets
### 9. Avoid Overly Broad Patterns
**Bad** (too broad):
```toml
[allowlist]
paths = ['''.*'''] # Disables all detection!
stopwords = ["key", "secret"] # Matches too many real secrets
```
**Good** (specific):
```toml
[allowlist]
paths = ['''test/unit/.*\.test\.js$'''] # Specific test directory
stopwords = ["example_key", "test_secret"] # Specific placeholders
```
### 10. Escape Special Characters
When using regex patterns, escape properly:
```toml
[allowlist]
regexes = [
'''api\.example\.com''', # Literal dot
'''config\[\'key\'\]''', # Literal brackets and quotes
]
```
## Troubleshooting False Positives
### Issue: Can't Identify Source of False Positive
```bash
# Run with verbose output
gitleaks detect -v | grep "RuleID"
# Get detailed finding information
gitleaks detect --report-format json | jq '.[] | {file: .File, line: .StartLine, rule: .RuleID}'
# View context around detection
gitleaks detect --report-format json | jq -r '.[0] | .File, .StartLine' | xargs -I {} sh -c 'sed -n "{}-5,{}+5p" {}'
```
### Issue: Allowlist Not Working
```bash
# Verify config is loaded
gitleaks detect --config .gitleaks.toml -v 2>&1 | grep "config"
# Check regex syntax
echo "test_string" | grep -E 'your_regex_pattern'
# Test path matching
echo "test/fixtures/file.json" | grep -E 'test/fixtures/.*'
```
### Issue: Too Many False Positives
1. **Export findings**: `gitleaks detect --report-format json > findings.json`
2. **Analyze patterns**: `jq -r '.[].File' findings.json | sort | uniq -c | sort -rn`
3. **Group by rule**: `jq -r '.[].RuleID' findings.json | sort | uniq -c | sort -rn`
4. **Create targeted allowlists** based on analysis
## False Positive vs Real Secret
When unsure, err on the side of caution:
| Indicator | False Positive | Real Secret |
|-----------|----------------|-------------|
| Location | Test/docs/examples | Production code |
| Pattern | "example", "test", "mock" | No such indicators |
| Entropy | Low/medium | High |
| Format | Incomplete/truncated | Complete/valid |
| Context | Educational comments | Functional code |
| Git history | Added in test commits | Added furtively |
**When in doubt**: Treat as real secret and investigate.

530
skills/devsecops/secrets-gitleaks/references/remediation_guide.md Normal file
View File

@@ -0,0 +1,530 @@
# Secret Remediation Guide
Comprehensive procedures for remediating exposed secrets detected by Gitleaks.
## Table of Contents
- [Immediate Response](#immediate-response)
- [Remediation Workflow](#remediation-workflow)
- [Git History Cleanup](#git-history-cleanup)
- [Cloud Provider Specific](#cloud-provider-specific)
- [Database Credentials](#database-credentials)
- [API Keys and Tokens](#api-keys-and-tokens)
- [Post-Remediation](#post-remediation)
## Immediate Response
When secrets are detected, follow this priority order:
### 1. Assess Exposure (0-15 minutes)
**Questions to answer immediately:**
- Is the repository public or private?
- Has the commit been pushed to remote?
- How long has the secret been exposed?
- What systems does this credential access?
**Actions:**
```bash
# Check if commit is pushed
git log origin/main..HEAD # If output, not yet pushed
# Check repository visibility
gh repo view --json visibility
# Check commit age
git log -1 --format="%ar" <commit-sha>
```
### 2. Rotate Credentials (0-30 minutes)
**CRITICAL**: Rotate the exposed credential immediately, regardless of exposure duration.
Priority order:
1. **Production credentials** - Immediate rotation
2. **Payment/financial systems** - Immediate rotation
3. **Customer data access** - Immediate rotation
4. **Development/test credentials** - Rotate within 24 hours
### 3. Review Access Logs (30-60 minutes)
Check for unauthorized access:
- Cloud provider audit logs (CloudTrail, Cloud Audit Logs, Activity Log)
- Application logs showing authentication attempts
- Database connection logs
- API usage logs
### 4. Remove from Code (0-24 hours)
Remove secret from current code and optionally from git history.
## Remediation Workflow
### Step 1: Rotate the Credential
**Before removing from code**, rotate the credential to prevent race conditions.
#### Cloud Providers
**AWS**:
```bash
# Deactivate compromised key
aws iam update-access-key \
--access-key-id AKIA... \
--status Inactive \
--user-name username
# Create new key
aws iam create-access-key --user-name username
# Delete old key after updating applications
aws iam delete-access-key \
--access-key-id AKIA... \
--user-name username
```
**GCP**:
```bash
# Delete service account key
gcloud iam service-accounts keys delete KEY_ID \
--iam-account=SERVICE_ACCOUNT_EMAIL
# Create new key
gcloud iam service-accounts keys create new-key.json \
--iam-account=SERVICE_ACCOUNT_EMAIL
```
**Azure**:
```bash
# Regenerate storage account key
az storage account keys renew \
--account-name ACCOUNT_NAME \
--key primary
# List keys to verify
az storage account keys list \
--account-name ACCOUNT_NAME
```
#### API Tokens
**GitHub**:
1. Navigate to Settings > Developer settings > Personal access tokens
2. Find the compromised token (check "Last used" column)
3. Click "Delete"
4. Generate new token with minimal required scopes
**Stripe**:
1. Log into Stripe Dashboard
2. Navigate to Developers > API keys
3. Click "Roll" on the compromised key
4. Update all applications with new key
**Generic API Key**:
1. Access provider's console/dashboard
2. Locate API key management
3. Revoke/delete compromised key
4. Generate new key
5. Update applications
6. Test connectivity
### Step 2: Remove from Current Code
Replace hardcoded secrets with environment variables or secret management:
**Before** (insecure):
```python
API_KEY = "sk_live_51ABC123..."
db_password = "MyP@ssw0rd123"
```
**After** (secure):
```python
import os
API_KEY = os.environ.get("STRIPE_API_KEY")
if not API_KEY:
raise ValueError("STRIPE_API_KEY environment variable not set")
db_password = os.environ.get("DB_PASSWORD")
```
**Using secret management**:
```python
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
client = SecretClient(vault_url="https://myvault.vault.azure.net/", credential=credential)
db_password = client.get_secret("database-password").value
```
### Step 3: Commit the Fix
```bash
# Add changes
git add .
# Commit with clear message
git commit -m "refactor: Move API credentials to environment variables
- Replace hardcoded Stripe API key with environment variable
- Replace database password with AWS Secrets Manager reference
- Add validation for required environment variables
Addresses: Secret exposure detected by Gitleaks scan"
# Push
git push origin main
```
## Git History Cleanup
If secrets are in pushed commits, consider removing from git history.
### Decision Matrix
| Scenario | Action | Reason |
|----------|--------|--------|
| Public repo, secret exposed | **Mandatory** history rewrite | Secret is public knowledge |
| Private repo, < 24 hours, < 5 collaborators | **Recommended** history rewrite | Minimal disruption |
| Private repo, > 1 week, > 10 collaborators | **Optional** - Rotate only | High coordination cost |
| Production repo with CI/CD | **Coordinate carefully** | May break automation |
### Method 1: git-filter-repo (Recommended)
Install:
```bash
pip install git-filter-repo
```
Remove specific file from all history:
```bash
# Backup first
git clone --mirror <repo-url> backup-repo.git
# Remove file
git filter-repo --path config/secrets.yaml --invert-paths
# Force push
git push origin --force --all
```
Remove secrets matching pattern:
```bash
# Use callback for complex filtering
git filter-repo --replace-text <(echo 'regex:sk_live_[a-zA-Z0-9]{24}==>REDACTED')
```
### Method 2: BFG Repo-Cleaner
Download:
```bash
# macOS
brew install bfg
# Or download JAR from https://rtyley.github.io/bfg-repo-cleaner/
```
Remove specific file:
```bash
# Clone mirror
git clone --mirror <repo-url> repo-mirror.git
cd repo-mirror.git
# Remove file
bfg --delete-files secrets.env
# Clean up
git reflog expire --expire=now --all
git gc --prune=now --aggressive
# Force push
git push
```
Remove secrets by pattern:
```bash
# Create replacements.txt
echo "PASSWORD1==>***REMOVED***" > replacements.txt
echo "sk_live_51ABC==>***REMOVED***" >> replacements.txt
# Run BFG
bfg --replace-text replacements.txt repo-mirror.git
```
### Method 3: Interactive Rebase (Small Changes)
For recent commits not yet widely distributed:
```bash
# Rebase last N commits
git rebase -i HEAD~5
# In editor, mark commits to 'edit'
# When stopped at each commit:
git rm config/secrets.yaml
git commit --amend --no-edit
git rebase --continue
# Force push
git push --force-with-lease
```
### Post-Rewrite Coordination
After rewriting history:
1. **Notify team immediately**:
```text
URGENT: Git history rewritten to remove exposed credentials.
Action required for all developers:
1. Commit/stash any local changes
2. Run: git fetch origin && git reset --hard origin/main
3. Delete and re-clone if issues persist
Contact security team with questions.
```
2. **Update CI/CD**:
- Invalidate old caches
- May need to reconfigure webhooks
- Update any hardcoded commit references
3. **Update branch protection**:
- May need to temporarily disable
- Re-enable after force push completes
## Cloud Provider Specific
### AWS
**Check for unauthorized access**:
```bash
# List recent API calls for access key
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=Username,AttributeValue=compromised-user \
--max-results 50 \
--start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S)
```
**Revoke all sessions**:
```bash
# Attach policy to deny all actions
aws iam put-user-policy \
--user-name compromised-user \
--policy-name DenyAll \
--policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*"}]}'
```
### GCP
**Check audit logs**:
```bash
gcloud logging read "protoPayload.authenticationInfo.principalEmail=SERVICE_ACCOUNT_EMAIL" \
--limit 100 \
--format json
```
**Disable service account**:
```bash
gcloud iam service-accounts disable SERVICE_ACCOUNT_EMAIL
```
### Azure
**Review activity logs**:
```bash
az monitor activity-log list \
--start-time 2024-01-01T00:00:00Z \
--resource-id /subscriptions/SUBSCRIPTION_ID
```
**Revoke access**:
```bash
# Regenerate keys
az storage account keys renew \
--account-name STORAGE_ACCOUNT \
--key primary
```
## Database Credentials
### PostgreSQL
```sql
-- Change password
ALTER USER app_user WITH PASSWORD 'new_secure_password';
-- View recent connections
SELECT datname, usename, client_addr, backend_start
FROM pg_stat_activity
WHERE usename = 'app_user'
ORDER BY backend_start DESC;
-- Kill active connections (if suspicious)
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE usename = 'app_user' AND client_addr != 'trusted_ip';
```
### MySQL
```sql
-- Change password
ALTER USER 'app_user'@'%' IDENTIFIED BY 'new_secure_password';
FLUSH PRIVILEGES;
-- View recent connections
SELECT * FROM information_schema.PROCESSLIST
WHERE USER = 'app_user';
-- Kill connections
KILL CONNECTION process_id;
```
### MongoDB
```javascript
// Change password
use admin
db.changeUserPassword("app_user", "new_secure_password")
// View recent operations
db.currentOp({ "active": true })
// Kill operation
db.killOp(opid)
```
## API Keys and Tokens
### GitHub
**Audit unauthorized access**:
```bash
# List recent events for token
gh api /users/{username}/events/public | jq '.[] | {type, repo: .repo.name, created_at}'
```
**Revoke all tokens** (if compromised account):
1. Settings > Developer settings > Personal access tokens
2. Select all tokens
3. Click "Delete"
### Slack
**Check workspace audit logs**:
1. Go to workspace settings (admin required)
2. Navigate to Logs > Audit Logs
3. Filter by token usage
**Regenerate token**:
1. Go to api.slack.com/apps
2. Select your app
3. Navigate to OAuth & Permissions
4. Click "Regenerate" on token
## Post-Remediation
### 1. Implement Prevention
**Pre-commit hooks**:
```bash
# Install Gitleaks pre-commit hook
cd /path/to/repo
cat << 'EOF' > .git/hooks/pre-commit
#!/bin/sh
gitleaks protect --verbose --redact --staged
EOF
chmod +x .git/hooks/pre-commit
```
**CI/CD checks**:
```yaml
# .github/workflows/secrets-scan.yml
name: Secret Scanning
on: [push, pull_request]
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
### 2. Update Secret Management
Migrate to proper secret management:
**Environment variables** (minimal):
```bash
# .env (never commit!)
DATABASE_URL=postgresql://user:pass@host:5432/db
API_KEY=sk_live_...
# .gitignore
.env
.env.local
```
**Secret management services**:
- AWS: Secrets Manager, Systems Manager Parameter Store
- GCP: Secret Manager
- Azure: Key Vault
- HashiCorp: Vault
- Kubernetes: Secrets
### 3. Document Incident
Create incident report including:
- **Timeline**: When secret was committed, detected, remediated
- **Exposure**: Duration, repository visibility, access scope
- **Impact**: Systems accessed, data at risk, unauthorized activity
- **Response**: Rotation completed, logs reviewed, history cleaned
- **Prevention**: Controls implemented to prevent recurrence
### 4. Team Training
Conduct training on:
- Using environment variables and secret management
- Pre-commit hooks and local scanning
- Recognizing secrets in code review
- Incident response procedures
### 5. Compliance Notifications
If required by regulations:
- **GDPR**: Notify supervisory authority within 72 hours if personal data at risk
- **PCI-DSS**: Notify card brands and processor if payment data affected
- **SOC2**: Document in compliance report, may trigger audit
- **HIPAA**: Notify covered entities if PHI exposed
## Prevention Checklist
- [ ] Credential rotated and old credential deactivated
- [ ] Access logs reviewed for unauthorized activity
- [ ] Secret removed from current code
- [ ] Git history cleaned (if applicable)
- [ ] Team notified of credential change
- [ ] Applications updated with new credential
- [ ] Pre-commit hooks installed
- [ ] CI/CD secret scanning enabled
- [ ] Secret management solution implemented
- [ ] Incident documented
- [ ] Compliance notifications sent (if required)
- [ ] Team training scheduled
## Emergency Contacts
Maintain contact list for rapid response:
- **Security Team**: security@company.com
- **DevOps On-Call**: devops-oncall@company.com
- **Cloud Provider Support**: Account-specific
- **Compliance Officer**: compliance@company.com