Initial commit

2025-11-30 08:47:43 +08:00
commit 2e8d89fca3
41 changed files with 14051 additions and 0 deletions
--- a/skills/documentation/policyengine-standards-skill/SKILL.md
+++ b/skills/documentation/policyengine-standards-skill/SKILL.md
@@ -0,0 +1,768 @@
+---
+name: policyengine-standards
+description: PolicyEngine coding standards, formatters, CI requirements, and development best practices
+---
+
+# PolicyEngine Standards Skill
+
+Use this skill to ensure code meets PolicyEngine's development standards and passes CI checks.
+
+## When to Use This Skill
+
+- Before committing code to any PolicyEngine repository
+- When CI checks fail with linting/formatting errors
+- Setting up a new PolicyEngine repository
+- Reviewing PRs for standard compliance
+- When AI tools generate code that needs standardization
+
+## Critical Requirements
+
+### Python Version
+⚠️ **MUST USE Python 3.13** - Do NOT downgrade to older versions
+- Check version: `python --version`
+- Use `pyproject.toml` to specify version requirements
+
+### Command Execution
+⚠️ **ALWAYS use `uv run` for Python commands** - Never use bare `python` or `pytest`
+- ✅ Correct: `uv run python script.py`, `uv run pytest tests/`
+- ❌ Wrong: `python script.py`, `pytest tests/`
+- This ensures correct virtual environment and dependencies
+
+### Documentation (Python Projects)
+⚠️ **MUST USE Jupyter Book 2.0 (MyST-NB)** - NOT Jupyter Book 1.x
+- Build docs: `myst build docs` (NOT `jb build`)
+- Use MyST markdown syntax
+
+## Before Committing - Checklist
+
+1. **Write tests first** (TDD - see below)
+2. **Format code**: `make format` or language-specific formatter
+3. **Run tests**: `make test` to ensure all tests pass
+4. **Check linting**: Ensure no linting errors
+5. **Use config files**: Prefer config files over environment variables
+6. **Reference issues**: Include "Fixes #123" in commit message
+
+## Creating Pull Requests
+
+### The CI Waiting Problem
+
+**Common failure pattern:**
+```
+User: "Create a PR and mark it ready when CI passes"
+Claude: "I've created the PR as draft. CI will take a while, I'll check back later..."
+[Chat ends - Claude never checks back]
+Result: PR stays in draft, user has to manually check CI and mark ready
+```
+
+### Solution: Use /create-pr Command
+
+**When creating PRs, use the /create-pr command:**
+
+```bash
+/create-pr
+```
+
+**This command:**
+- ✅ Creates PR as draft
+- ✅ Actually waits for CI (polls every 15 seconds)
+- ✅ Marks ready when CI passes
+- ✅ Reports failures with details
+- ✅ Handles timeouts gracefully
+
+**Why this works:**
+The command contains explicit polling logic that Claude executes, so it actually waits instead of giving up.
+
+### If /create-pr is Not Available
+
+**If the command isn't installed, implement the pattern directly:**
+
+```bash
+# 1. Create PR as draft
+gh pr create --draft --title "Title" --body "Body"
+PR_NUMBER=$(gh pr view --json number --jq '.number')
+
+# 2. Wait for CI (ACTUALLY WAIT - don't give up!)
+POLL_INTERVAL=15
+ELAPSED=0
+
+while true; do  # No timeout - wait as long as needed
+  CHECKS=$(gh pr checks $PR_NUMBER --json status,conclusion)
+  TOTAL=$(echo "$CHECKS" | jq '. | length')
+  COMPLETED=$(echo "$CHECKS" | jq '[.[] | select(.status == "COMPLETED")] | length')
+
+  echo "[$ELAPSED s] CI: $COMPLETED/$TOTAL completed"
+
+  if [ "$COMPLETED" -eq "$TOTAL" ] && [ "$TOTAL" -gt 0 ]; then
+    FAILED=$(echo "$CHECKS" | jq '[.[] | select(.conclusion == "FAILURE")] | length')
+    if [ "$FAILED" -eq 0 ]; then
+      echo "✅ All CI passed! Marking ready..."
+      gh pr ready $PR_NUMBER
+      break
+    else
+      echo "❌ CI failed. PR remains draft."
+      gh pr checks $PR_NUMBER
+      break
+    fi
+  fi
+
+  sleep $POLL_INTERVAL
+  ELAPSED=$((ELAPSED + POLL_INTERVAL))
+done
+
+# Important: No timeout! Population simulations can take 30+ minutes.
+```
+
+### DO NOT Say "I'll Check Back Later"
+
+**❌ WRONG:**
+```
+"I've created the PR as draft. CI checks will take a few minutes.
+I'll check back later once they complete."
+```
+
+**Why wrong:** You cannot check back later. The chat session ends.
+
+**✅ CORRECT:**
+```
+"I've created the PR as draft. Now polling CI status every 15 seconds..."
+[Actually polls using while loop]
+"CI checks completed. All passed! Marking PR as ready for review."
+```
+
+### When to Create Draft vs Ready
+
+**Always create as draft when:**
+- CI checks are configured
+- User asks to wait for CI
+- Making automated changes
+- Unsure if CI will pass
+
+**Create as ready only when:**
+- User explicitly requests ready PR
+- No CI configured
+- CI already verified locally
+
+### PR Workflow Standards
+
+**Standard flow:**
+```bash
+# 1. Ensure branch is pushed
+git push -u origin feature-branch
+
+# 2. Create PR as draft
+gh pr create --draft --title "..." --body "..."
+
+# 3. Wait for CI (use polling loop - see pattern above)
+
+# 4. If CI passes:
+gh pr ready $PR_NUMBER
+
+# 5. If CI fails:
+echo "CI failed. PR remains draft. Fix issues and push again."
+```
+
+## Test-Driven Development (TDD)
+
+PolicyEngine follows Test-Driven Development practices across all repositories.
+
+### TDD Workflow
+
+**1. Write test first (RED):**
+```python
+# tests/test_new_feature.py
+def test_california_eitc_calculation():
+    """Test California EITC for family with 2 children earning $30,000."""
+    situation = create_family(income=30000, num_children=2, state="CA")
+    sim = Simulation(situation=situation)
+    ca_eitc = sim.calculate("ca_eitc", 2024)[0]
+
+    # Test fails initially (feature not implemented yet)
+    assert ca_eitc == 3000, "CA EITC should be $3,000 for this household"
+```
+
+**2. Implement feature (GREEN):**
+```python
+# policyengine_us/variables/gov/states/ca/tax/income/credits/ca_eitc.py
+class ca_eitc(Variable):
+    value_type = float
+    entity = TaxUnit
+    definition_period = YEAR
+
+    def formula(tax_unit, period, parameters):
+        # Implementation to make test pass
+        federal_eitc = tax_unit("eitc", period)
+        return federal_eitc * parameters(period).gov.states.ca.tax.eitc.match
+```
+
+**3. Refactor (REFACTOR):**
+```python
+# Clean up, optimize, add documentation
+# All while tests continue to pass
+```
+
+### TDD Benefits
+
+**Why PolicyEngine uses TDD:**
+- ✅ **Accuracy** - Tests verify implementation matches regulations
+- ✅ **Documentation** - Tests show expected behavior
+- ✅ **Regression prevention** - Changes don't break existing features
+- ✅ **Confidence** - Safe to refactor
+- ✅ **Isolation** - Multi-agent workflow (test-creator and rules-engineer work separately)
+
+### TDD in Multi-Agent Workflow
+
+**Country model development:**
+1. **@document-collector** gathers regulations
+2. **@test-creator** writes tests from regulations (isolated, no implementation access)
+3. **@rules-engineer** implements from regulations (isolated, no test access)
+4. Both work from same source → tests verify implementation accuracy
+
+**See policyengine-core-skill and country-models agents for details.**
+
+### Test Examples
+
+**Python (pytest):**
+```python
+def test_ctc_for_two_children():
+    """Test CTC calculation for married couple with 2 children."""
+    situation = create_married_couple(
+        income_1=75000,
+        income_2=50000,
+        num_children=2,
+        child_ages=[5, 8]
+    )
+
+    sim = Simulation(situation=situation)
+    ctc = sim.calculate("ctc", 2024)[0]
+
+    assert ctc == 4000, "CTC should be $2,000 per child"
+```
+
+**React (Jest + RTL):**
+```javascript
+import { render, screen } from '@testing-library/react';
+import TaxCalculator from './TaxCalculator';
+
+test('displays calculated tax', () => {
+  render(<TaxCalculator income={50000} />);
+
+  // Test what user sees, not implementation
+  expect(screen.getByText(/\$5,000/)).toBeInTheDocument();
+});
+```
+
+### Test Organization
+
+**Python:**
+```
+tests/
+├── test_variables/
+│   ├── test_income.py
+│   ├── test_deductions.py
+│   └── test_credits.py
+├── test_parameters/
+└── test_simulations/
+```
+
+**React:**
+```
+src/
+├── components/
+│   └── TaxCalculator/
+│       ├── TaxCalculator.jsx
+│       └── TaxCalculator.test.jsx
+```
+
+### Running Tests
+
+**Python:**
+```bash
+# All tests
+make test
+
+# With uv
+uv run pytest tests/ -v
+
+# Specific test
+uv run pytest tests/test_credits.py::test_ctc_for_two_children -v
+
+# With coverage
+uv run pytest tests/ --cov=policyengine_us --cov-report=html
+```
+
+**React:**
+```bash
+# All tests
+make test
+
+# Watch mode
+npm test -- --watch
+
+# Specific test
+npm test -- TaxCalculator.test.jsx
+
+# Coverage
+npm test -- --coverage
+```
+
+### Test Quality Standards
+
+**Good tests:**
+- ✅ Test behavior, not implementation
+- ✅ Clear, descriptive names
+- ✅ Single assertion per test (when possible)
+- ✅ Include documentation (docstrings)
+- ✅ Based on official regulations with citations
+
+**Bad tests:**
+- ❌ Testing private methods
+- ❌ Mocking everything
+- ❌ No assertion messages
+- ❌ Magic numbers without explanation
+
+### Example: TDD for New Feature
+
+```python
+# Step 1: Write test (RED)
+def test_new_york_empire_state_child_credit():
+    """Test NY Empire State Child Credit for family with 1 child.
+
+    Based on NY Tax Law Section 606(c-1).
+    Family earning $50,000 with 1 child under 4 should receive $330.
+    """
+    situation = create_family(
+        income=50000,
+        num_children=1,
+        child_ages=[2],
+        state="NY"
+    )
+
+    sim = Simulation(situation=situation)
+    credit = sim.calculate("ny_empire_state_child_credit", 2024)[0]
+
+    assert credit == 330, "Should receive $330 for child under 4"
+
+# Test fails - feature doesn't exist yet
+
+# Step 2: Implement (GREEN)
+# Create variable in policyengine_us/variables/gov/states/ny/...
+# Test passes
+
+# Step 3: Refactor
+# Optimize, add documentation, maintain passing tests
+```
+
+## Python Standards
+
+### Formatting
+- **Formatter**: Black with 79-character line length
+- **Command**: `make format` or `black . -l 79`
+- **Check without changes**: `black . -l 79 --check`
+
+```bash
+# Format all Python files
+make format
+
+# Check if formatting is needed (CI-style)
+black . -l 79 --check
+```
+
+### Code Style
+```python
+# Imports: Grouped and alphabetized
+import os
+import sys
+from pathlib import Path  # stdlib
+
+import numpy as np
+import pandas as pd  # third-party
+
+from policyengine_us import Simulation  # local
+
+# Naming conventions
+class TaxCalculator:  # CamelCase for classes
+    pass
+
+def calculate_income_tax(income):  # snake_case for functions
+    annual_income = income * 12  # snake_case for variables
+    return annual_income
+
+# Type hints (recommended)
+def calculate_tax(income: float, state: str) -> float:
+    """Calculate state income tax.
+
+    Args:
+        income: Annual income in dollars
+        state: Two-letter state code
+
+    Returns:
+        Tax liability in dollars
+    """
+    pass
+
+# Error handling - catch specific exceptions
+try:
+    result = simulation.calculate("income_tax", 2024)
+except KeyError as e:
+    raise ValueError(f"Invalid variable name: {e}")
+```
+
+### Testing
+```python
+import pytest
+
+def test_ctc_calculation():
+    """Test Child Tax Credit calculation for family with 2 children."""
+    situation = create_family(income=50000, num_children=2)
+    sim = Simulation(situation=situation)
+    ctc = sim.calculate("ctc", 2024)[0]
+
+    assert ctc == 4000, "CTC should be $2000 per child"
+```
+
+**Run tests:**
+```bash
+# All tests
+make test
+
+# Or with uv
+uv run pytest tests/ -v
+
+# Specific test
+uv run pytest tests/test_tax.py::test_ctc_calculation -v
+
+# With coverage
+uv run pytest tests/ --cov=policyengine_us --cov-report=html
+```
+
+## JavaScript/React Standards
+
+### Formatting
+- **Formatters**: Prettier + ESLint
+- **Command**: `npm run lint -- --fix && npx prettier --write .`
+- **CI Check**: `npm run lint -- --max-warnings=0`
+
+```bash
+# Format all files
+make format
+
+# Or manually
+npm run lint -- --fix
+npx prettier --write .
+
+# Check if formatting is needed (CI-style)
+npm run lint -- --max-warnings=0
+```
+
+### Code Style
+```javascript
+// Use functional components only (no class components)
+import { useState, useEffect } from "react";
+
+function TaxCalculator({ income, state }) {
+  const [tax, setTax] = useState(0);
+
+  useEffect(() => {
+    // Calculate tax when inputs change
+    calculateTax(income, state).then(setTax);
+  }, [income, state]);
+
+  return (
+    <div>
+      <p>Tax: ${tax.toLocaleString()}</p>
+    </div>
+  );
+}
+
+// File naming
+// - Components: PascalCase.jsx (TaxCalculator.jsx)
+// - Utilities: camelCase.js (formatCurrency.js)
+
+// Environment config - use config file pattern
+// src/config/environment.js
+const config = {
+  API_URL: process.env.NODE_ENV === 'production'
+    ? 'https://api.policyengine.org'
+    : 'http://localhost:5000'
+};
+export default config;
+```
+
+### React Component Size
+- Keep components under 150 lines after formatting
+- Extract complex logic into custom hooks
+- Split large components into smaller ones
+
+## Version Control Standards
+
+### Changelog Management
+
+**CRITICAL**: For PRs, ONLY modify `changelog_entry.yaml`. NEVER manually update `CHANGELOG.md` or `changelog.yaml`.
+
+**Correct Workflow:**
+1. Create `changelog_entry.yaml` at repository root:
+   ```yaml
+   - bump: patch  # or minor, major
+     changes:
+       added:
+       - Description of new feature
+       fixed:
+       - Description of bug fix
+       changed:
+       - Description of change
+   ```
+
+2. Commit ONLY `changelog_entry.yaml` with your code changes
+
+3. GitHub Actions automatically updates `CHANGELOG.md` and `changelog.yaml` on merge
+
+**DO NOT:**
+- ❌ Run `make changelog` manually during PR creation
+- ❌ Commit `CHANGELOG.md` or `changelog.yaml` in your PR
+- ❌ Modify main changelog files directly
+
+### Git Workflow
+
+1. **Create branches on PolicyEngine repos, NOT forks**
+   - Forks cause CI failures due to missing secrets
+   - Request write access if needed
+
+2. **Branch naming**: `feature-name` or `fix-issue-123`
+
+3. **Commit messages**:
+   ```
+   Add CTC reform analysis for CRFB report
+
+   - Implement household-level calculations
+   - Add state-by-state comparison
+   - Create visualizations
+
+   Fixes #123
+   ```
+
+4. **PR description**: Include "Fixes #123" to auto-close issues
+
+### Common Git Pitfalls
+
+**Never do these:**
+- ❌ Force push to main/master
+- ❌ Commit secrets or `.env` files
+- ❌ Skip hooks with `--no-verify`
+- ❌ Create versioned files (app_v2.py, component_new.jsx)
+
+**Always do:**
+- ✅ Fix original files in place
+- ✅ Run formatters before pushing
+- ✅ Reference issue numbers in commits
+- ✅ Watch CI after filing PR
+
+## Common AI Pitfalls
+
+Since many PRs are AI-generated, watch for these common mistakes:
+
+### 1. File Versioning
+**❌ Wrong:**
+```bash
+# Creating new versions instead of fixing originals
+app_new.py
+app_v2.py
+component_refactored.jsx
+```
+
+**✅ Correct:**
+```bash
+# Always modify the original file
+app.py  # Fixed in place
+```
+
+### 2. Formatter Not Run
+**❌ Wrong:** Committing without formatting (main cause of CI failures)
+
+**✅ Correct:**
+```bash
+# Python
+make format
+black . -l 79
+
+# React
+npm run lint -- --fix
+npx prettier --write .
+```
+
+### 3. Environment Variables
+**❌ Wrong:**
+```javascript
+// React env vars without REACT_APP_ prefix
+const API_URL = process.env.API_URL;  // Won't work!
+```
+
+**✅ Correct:**
+```javascript
+// Use config file pattern instead
+import config from './config/environment';
+const API_URL = config.API_URL;
+```
+
+### 4. Using Wrong Python Version
+**❌ Wrong:** Downgrading to Python 3.10 or older
+
+**✅ Correct:** Use Python 3.13 as specified in project requirements
+
+### 5. Manual Changelog Updates
+**❌ Wrong:** Running `make changelog` and committing `CHANGELOG.md`
+
+**✅ Correct:** Only create `changelog_entry.yaml` in PR
+
+## Repository Setup Patterns
+
+### Python Package Structure
+```
+policyengine-package/
+├── policyengine_package/
+│   ├── __init__.py
+│   ├── core/
+│   ├── calculations/
+│   └── utils/
+├── tests/
+│   ├── test_calculations.py
+│   └── test_core.py
+├── pyproject.toml
+├── Makefile
+├── CLAUDE.md
+├── CHANGELOG.md
+└── README.md
+```
+
+### React App Structure
+```
+policyengine-app/
+├── src/
+│   ├── components/
+│   ├── pages/
+│   ├── config/
+│   │   └── environment.js
+│   └── App.jsx
+├── public/
+├── package.json
+├── .eslintrc.json
+├── .prettierrc
+└── README.md
+```
+
+## Makefile Commands
+
+Standard commands across PolicyEngine repos:
+
+```bash
+make install    # Install dependencies
+make test       # Run tests
+make format     # Format code
+make changelog  # Update changelog (automation only, not manual)
+make debug      # Start dev server (apps)
+make build      # Production build (apps)
+```
+
+## CI Stability
+
+### Common CI Issues
+
+**1. Fork PRs Fail**
+- **Problem**: PRs from forks don't have access to repository secrets
+- **Solution**: Create branches directly on PolicyEngine repos
+
+**2. GitHub API Rate Limits**
+- **Problem**: Smoke tests fail with 403 errors
+- **Solution**: Re-run failed jobs (different runners have different limits)
+
+**3. Linting Failures**
+- **Problem**: Code not formatted before commit
+- **Solution**: Always run `make format` before committing
+
+**4. Test Failures in CI but Pass Locally**
+- **Problem**: Missing `uv run` prefix
+- **Solution**: Use `uv run pytest` instead of `pytest`
+
+## Best Practices Checklist
+
+### Code Quality
+- [ ] Code formatted with Black (Python) or Prettier (JS)
+- [ ] No linting errors
+- [ ] All tests pass
+- [ ] Type hints added (Python, where applicable)
+- [ ] Docstrings for public functions/classes
+- [ ] Error handling with specific exceptions
+
+### Version Control
+- [ ] Only `changelog_entry.yaml` created (not CHANGELOG.md)
+- [ ] Commit message references issue number
+- [ ] Branch created on PolicyEngine repo (not fork)
+- [ ] No secrets or .env files committed
+- [ ] Original files modified (no _v2 or _new files)
+
+### Testing
+- [ ] Tests written for new functionality
+- [ ] Tests pass locally with `make test`
+- [ ] Coverage maintained or improved
+- [ ] Edge cases handled
+
+### Documentation
+- [ ] README updated if needed
+- [ ] Code comments for complex logic
+- [ ] API documentation updated if needed
+- [ ] Examples provided for new features
+
+## Quick Reference
+
+### Format Commands by Language
+
+**Python:**
+```bash
+make format                # Format code
+black . -l 79 --check      # Check formatting
+uv run pytest tests/ -v    # Run tests
+```
+
+**React:**
+```bash
+make format                              # Format code
+npm run lint -- --max-warnings=0         # Check linting
+npm test                                 # Run tests
+```
+
+### Pre-Commit Checklist
+```bash
+# 1. Format
+make format
+
+# 2. Test
+make test
+
+# 3. Check linting
+# Python: black . -l 79 --check
+# React: npm run lint -- --max-warnings=0
+
+# 4. Stage and commit
+git add .
+git commit -m "Description
+
+Fixes #123"
+
+# 5. Push and watch CI
+git push
+```
+
+## Resources
+
+- **Main CLAUDE.md**: `/PolicyEngine/CLAUDE.md`
+- **Python Style**: PEP 8, Black documentation
+- **React Style**: Airbnb React/JSX Style Guide
+- **Testing**: pytest documentation, Jest/RTL documentation
+- **Writing Style**: See policyengine-writing-skill for blog posts, PR descriptions, and documentation
+
+## Examples
+
+See PolicyEngine repositories for examples of standard-compliant code:
+- **policyengine-us**: Python package standards
+- **policyengine-app**: React app standards
+- **givecalc**: Streamlit app standards
+- **crfb-tob-impacts**: Analysis repository standards