Initial commit

This commit is contained in:
Zhongwei Li
2025-11-30 08:47:43 +08:00
commit 2e8d89fca3
41 changed files with 14051 additions and 0 deletions

View File

@@ -0,0 +1,478 @@
---
name: policyengine-api
description: PolicyEngine API - Flask REST service powering policyengine.org and programmatic access
---
# PolicyEngine API
The PolicyEngine API is a Flask-based REST service that provides tax and benefit calculations for the web app and programmatic users.
## For Users 👥
### What is the API?
When you use policyengine.org, the API processes your calculations on our servers.
**API base:** https://api.policyengine.org
**What it does:**
- Runs tax and benefit calculations
- Stores and retrieves policy reforms
- Computes population-wide impacts
- Serves parameter and variable metadata
### Public Access
The API is publicly accessible with rate limits:
- **Unauthenticated:** 100 requests/minute
- **Authenticated:** 1,000 requests/minute
**Try it:**
```bash
curl https://api.policyengine.org/us/policy/2
```
### API Documentation
**OpenAPI spec:** https://api.policyengine.org/docs
**Interactive docs:** Swagger UI at API docs endpoint
## For Analysts 📊
### Using the API
**Option 1: Python client (recommended)**
```python
# Use the policyengine package
# See policyengine-python-client-skill
```
**Option 2: Direct API calls**
```python
import requests
# Calculate household impact
response = requests.post(
"https://api.policyengine.org/us/calculate",
json={
"household": household_situation,
"policy_id": None # or reform_id
}
)
result = response.json()
```
### Key Endpoints
**Household calculations:**
```
POST /us/calculate
POST /uk/calculate
```
**Policy management:**
```
GET /us/policy/{policy_id}
POST /us/policy
```
**Economy impacts:**
```
GET /us/economy/{policy_id}/over/{baseline_policy_id}
```
**Metadata:**
```
GET /us/parameters
GET /us/variables
GET /us/parameter/{parameter_name}
GET /us/variable/{variable_name}
```
### Rate Limits and Performance
**Rate limits:**
- 100 req/min (unauthenticated)
- 1,000 req/min (authenticated - contact team)
**Response times:**
- Household calculation: ~200-500ms
- Population impact: ~5-30 seconds
- Cached results: <100ms
**Optimization:**
- Use the same policy_id for multiple requests (caching)
- Batch calculations when possible
- Use webhooks for long-running jobs (population impacts)
## For Contributors 💻
### Repository
**Location:** PolicyEngine/policyengine-api
**Clone:**
```bash
git clone https://github.com/PolicyEngine/policyengine-api
cd policyengine-api
```
### Current Architecture
**To see current structure:**
```bash
tree policyengine_api/
# Key directories:
ls policyengine_api/
# - endpoints/ - HTTP endpoint handlers
# - routes/ - Route registration
# - services/ - Business logic
# - compute_api/ - Calculation services
# - economy_api/ - Economy impact calculations
# - utils/ - Helpers (caching, validation)
# - data/ - Static data
```
### Current Implementation Patterns
**Reference endpoint (read this first):**
```bash
cat policyengine_api/endpoints/economy.py
```
**This demonstrates:**
- Standard endpoint structure
- Request validation
- Caching pattern
- Error handling
- Response formatting
**To find other endpoints:**
```bash
ls policyengine_api/endpoints/
# - household.py
# - policy.py
# - economy.py
# - metadata.py
# - etc.
```
### Standard Endpoint Pattern (Stable)
```python
from flask import Blueprint, request, jsonify
from policyengine_api.utils import cache
blueprint = Blueprint("my_endpoint", __name__)
@blueprint.route("/us/calculate", methods=["POST"])
def calculate():
"""Standard pattern: validate, cache-check, compute, cache, return."""
try:
# 1. Get and validate input
data = request.json
if not data:
return jsonify({"error": "No data provided"}), 400
# 2. Generate cache key
cache_key = f"calc_{hash(str(data))}"
# 3. Check cache
cached = cache.get(cache_key)
if cached:
return jsonify(cached)
# 4. Compute
result = perform_calculation(data)
# 5. Cache result
cache.set(cache_key, result, expire=3600)
# 6. Return
return jsonify(result)
except Exception as e:
return jsonify({"error": str(e), "status": "error"}), 500
```
**Current implementation details:**
```bash
# See actual endpoint for current pattern
cat policyengine_api/endpoints/household.py
```
### Caching Strategy
**To see current caching implementation:**
```bash
# Redis configuration
cat policyengine_api/utils/cache.py
# Find cache usage
grep -r "cache\." policyengine_api/endpoints/
```
**Pattern:**
- Redis for caching
- Cache keys based on inputs
- TTL varies by endpoint (1 hour to 1 day)
- Clear cache on parameter changes
### Background Jobs
For long-running calculations (population impacts):
**To see current implementation:**
```bash
# RQ (Redis Queue) usage
grep -r "@job" policyengine_api/
# Job patterns
cat policyengine_api/economy_api/
```
**Pattern:**
- Use RQ for jobs > 5 seconds
- Return job_id immediately
- Poll for completion
- Cache results
### Country Integration
**How API loads country packages:**
```bash
cat policyengine_api/country.py
```
**Pattern:**
- Dynamically imports country packages
- Routes by country code (/us/, /uk/)
- Manages multiple model versions
### Service Layer
**Business logic separated from endpoints:**
```bash
ls policyengine_api/services/
```
**Pattern:**
```python
# endpoints/household.py
from policyengine_api.services import household_service
@app.route("/us/calculate", methods=["POST"])
def calculate():
result = household_service.calculate(data)
return jsonify(result)
# services/household_service.py
def calculate(data):
# Business logic here
simulation = create_simulation(data)
return simulation.calculate(...)
```
### Testing
**To see current test patterns:**
```bash
ls tests/
cat tests/test_household.py
```
**Run tests:**
```bash
make test
# Specific test
pytest tests/test_economy.py -v
# With coverage
make test-coverage
```
### Development Server
**Start locally:**
```bash
make debug
```
**Test endpoint:**
```bash
curl http://localhost:5000/us/policy/2
```
### Deployment
**To see deployment configuration:**
```bash
# Google Cloud Platform
cat app.yaml # App Engine config
cat cloudbuild.yaml # Cloud Build config
# Environment variables
cat .env.example
```
**Current deployment:**
- Google App Engine
- Cloud SQL (PostgreSQL)
- Redis (caching)
- Cloud Build (CI/CD)
### API Versions
**To see versioning strategy:**
```bash
grep -r "version" policyengine_api/
```
**Current approach:**
- API version in URLs (may add /v1/ prefix)
- Country package versions independent
- Breaking changes rare (backwards compatible)
## Architecture Diagrams
### Request Flow
```
User/App → API Gateway → Flask App → Country Package → Core Engine
Redis Cache
Background Job (if needed)
PostgreSQL (storage)
```
### Dependencies
```
policyengine-core
policyengine-us, policyengine-uk, etc.
policyengine-api (you are here)
policyengine-app (consumes API)
```
**To understand dependencies:**
- See `policyengine-core-skill` for engine patterns
- See `policyengine-us-skill` for country model usage
- See `policyengine-app-skill` for how app calls API
## Common Development Tasks
### Task 1: Add New Endpoint
1. **Study reference implementation:**
```bash
cat policyengine_api/endpoints/economy.py
```
2. **Create new endpoint file:**
```python
# policyengine_api/endpoints/my_endpoint.py
# Follow the pattern from economy.py
```
3. **Register route:**
```bash
# See route registration
cat policyengine_api/routes/__init__.py
```
4. **Add tests:**
```bash
# Follow test pattern
cat tests/test_economy.py
```
### Task 2: Modify Caching Behavior
**See current caching:**
```bash
cat policyengine_api/utils/cache.py
```
**Common changes:**
- Adjust TTL (time to live)
- Change cache key generation
- Add cache invalidation
### Task 3: Update Country Package Version
**To see how versions are managed:**
```bash
# Requirements
cat requirements.txt | grep policyengine-
# Update and deploy
# See deployment docs in README
```
## Security and Best Practices
### Input Validation
**Always validate:**
- Country code (us, uk, ca)
- Policy ID format
- Household structure
- Parameter values
**See validation examples:**
```bash
grep -r "validate" policyengine_api/endpoints/
```
### Error Handling
**Standard error response:**
```python
return jsonify({
"error": "Error message",
"details": additional_context,
"status": "error"
}), status_code
```
**See error patterns:**
```bash
grep -A 5 "jsonify.*error" policyengine_api/endpoints/
```
### Logging
**To see logging configuration:**
```bash
cat policyengine_api/gcp_logging.py
```
**Pattern:**
- Google Cloud Logging
- Log all errors
- Log slow queries (>1s)
- Don't log sensitive data
## Related Skills
- **policyengine-python-client-skill** - Using the API
- **policyengine-core-skill** - Understanding the engine
- **policyengine-us-skill** - Country model integration
- **policyengine-app-skill** - How app consumes API
- **policyengine-standards-skill** - Code quality
- **policyengine-writing-skill** - API documentation style
## Resources
**Repository:** https://github.com/PolicyEngine/policyengine-api
**Live API:** https://api.policyengine.org
**Documentation:** https://api.policyengine.org/docs
**Status:** https://status.policyengine.org

View File

@@ -0,0 +1,900 @@
---
name: policyengine-app
description: PolicyEngine React web application - the user interface at policyengine.org
---
# PolicyEngine App
The PolicyEngine App is the React-based web application that users interact with at policyengine.org.
## For Users 👥
### What is the App?
The app at policyengine.org provides:
- Interactive household calculator
- Policy reform creator
- Population impact analysis
- Blog and research hub
**Access:** https://policyengine.org
### App Features
**Calculator:**
- Enter household details
- See tax and benefit calculations
- Visualize marginal tax rates
- Compare scenarios
**Policy designer:**
- Browse all parameters
- Create custom reforms
- Share via URL
- Download charts
**Research hub:**
- Read policy analysis
- Explore modeled programs
- Access documentation
## For Analysts 📊
### Understanding App URLs
Reform URLs encode all policy changes in the query string, allowing sharing and reproducibility.
**Example URL:**
```
policyengine.org/us/policy?
focus=policyOutput.policyBreakdown&
reform=67696&
region=enhanced_us&
timePeriod=2025&
baseline=2
```
**Parameters:**
- `focus` - Which section to display
- `reform` - Reform ID from database
- `region` - Geographic scope (enhanced_us, CA, congressional districts)
- `timePeriod` - Year of analysis
- `baseline` - Baseline policy ID
### Embedding PolicyEngine
**iFrame integration:**
```html
<iframe
src="https://policyengine.org/us/household?embedded=true"
width="100%"
height="800">
</iframe>
```
**Parameter:**
- `embedded=true` - Removes navigation, optimizes for embedding
### URL Structure
**Household calculator:**
```
/us/household?household=12345
/uk/household?household=67890
```
**Policy page:**
```
/us/policy?reform=12345
/uk/policy?reform=67890
```
**Research/blog:**
```
/us/research/article-slug
/uk/research/article-slug
```
## For Contributors 💻
### Repository
**Location:** PolicyEngine/policyengine-app
**Clone:**
```bash
git clone https://github.com/PolicyEngine/policyengine-app
cd policyengine-app
```
### Current Architecture
**To see current structure:**
```bash
tree src/ -L 2
# Key directories:
ls src/
# - pages/ - Page components
# - applets/ - Reusable UI modules
# - api/ - API integration
# - controls/ - Form controls
# - layout/ - Layout components
# - posts/ - Blog posts
# - routing/ - Routing configuration
# - hooks/ - Custom React hooks
# - data/ - Static data
```
### Technology Stack
**Current dependencies:**
```bash
# See package.json for versions
cat package.json
# Key dependencies:
# - React 18
# - React Router v6
# - Plotly.js
# - Ant Design
# - axios
```
### React Patterns (Critical)
**✅ Functional components only (no classes):**
```javascript
// CORRECT
import { useState, useEffect } from "react";
export default function TaxCalculator({ income }) {
const [tax, setTax] = useState(0);
useEffect(() => {
calculateTax(income).then(setTax);
}, [income]);
return <div>Tax: ${tax}</div>;
}
```
**❌ Class components forbidden:**
```javascript
// WRONG - Don't use class components
class TaxCalculator extends Component {
// ...
}
```
**To find component examples:**
```bash
# Reference components
ls src/pages/
ls src/applets/
# See a complete page
cat src/pages/HouseholdPage.jsx
```
### State Management
**No global state (Redux, Context) - lift state up:**
```javascript
// Parent manages state
function PolicyPage() {
const [reform, setReform] = useState({});
const [impact, setImpact] = useState(null);
return (
<>
<PolicyEditor reform={reform} onChange={setReform} />
<ImpactDisplay impact={impact} />
</>
);
}
```
**To see state patterns:**
```bash
# Find useState usage
grep -r "useState" src/pages/ | head -20
```
### API Integration
**To see current API patterns:**
```bash
cat src/api/call.js # Base API caller
cat src/api/variables.js # Variable metadata
cat src/api/parameters.js # Parameter metadata
```
**Standard pattern:**
```javascript
import { api call } from "api/call";
// Fetch data
const result = await call(
`/us/calculate`,
{ household: householdData },
"POST"
);
```
### Routing
**To see current routing:**
```bash
cat src/routing/routes.js
# Routes defined with React Router v6
# See examples:
grep -r "useNavigate" src/
grep -r "useSearchParams" src/
```
**URL parameters:**
```javascript
import { useSearchParams } from "react-router-dom";
const [searchParams, setSearchParams] = useSearchParams();
// Read
const reformId = searchParams.get("reform");
// Update
setSearchParams({ ...Object.fromEntries(searchParams), reform: newId });
```
### Custom Hooks
**To see PolicyEngine-specific hooks:**
```bash
ls src/hooks/
# - useCountryId.js - Current country
# - useDisplayCategory.js
# - etc.
```
**Usage:**
```javascript
import { useCountryId } from "hooks/useCountryId";
function Component() {
const [countryId, setCountryId] = useCountryId();
// countryId = "us", "uk", or "ca"
}
```
### Charts and Visualization
**Plotly integration:**
```bash
# See chart components
ls src/pages/policy/output/
# Reference implementation
cat src/pages/policy/output/EconomyOutput.jsx
```
**Standard Plotly pattern:**
```javascript
import Plot from "react-plotly.js";
const layout = {
font: { family: "Roboto Serif" },
plot_bgcolor: "white",
// PolicyEngine branding
};
<Plot
data={traces}
layout={layout}
config={{ displayModeBar: false }}
/>;
```
### Blog Posts
**To see blog post structure:**
```bash
ls src/posts/articles/
# Read a recent post
cat src/posts/articles/harris-eitc.md
```
**Blog posts:**
- Written in Markdown
- Stored in `src/posts/articles/`
- Include metadata (title, date, authors)
- Follow policyengine-writing-skill style
**Adding a post:**
```bash
# Create new file
# src/posts/articles/my-analysis.md
# Add to index (if needed)
# See existing posts for format
```
### Styling
**Current styling approach:**
```bash
# See style configuration
ls src/style/
# Colors
cat src/style/colors.js
# Ant Design theme
cat src/style/theme.js
```
**PolicyEngine colors:**
- Teal: `#39C6C0` (primary accent)
- Blue: `#2C6496` (charts, links)
- Dark gray: `#616161` (text)
### Testing
**To see current tests:**
```bash
ls src/__tests__/
# Run tests
make test
# Test pattern
cat src/__tests__/example.test.js
```
**Testing libraries:**
- Jest (test runner)
- React Testing Library (component testing)
- User-centric testing (not implementation details)
### Development Server
**Start locally:**
```bash
make debug
# Opens http://localhost:3000
```
**Environment:**
```bash
# Environment variables
cat .env.example
# Config
ls src/config/
```
### Building and Deployment
**Build:**
```bash
make build
# Creates optimized production build
```
**Deployment:**
```bash
# See deployment config
cat netlify.toml # or appropriate hosting config
```
## Component Patterns
### Standard Component Structure
**To see well-structured components:**
```bash
# Example page
cat src/pages/HouseholdPage.jsx
# Example applet
cat src/applets/PolicySearch.jsx
```
**Pattern:**
```javascript
import { useState, useEffect } from "react";
import { useSearchParams } from "react-router-dom";
import { useCountryId } from "hooks/useCountryId";
export default function MyComponent({ prop1, prop2 }) {
// 1. Hooks first
const [state, setState] = useState(initialValue);
const [countryId] = useCountryId();
const [searchParams] = useSearchParams();
// 2. Effects
useEffect(() => {
// Side effects
}, [dependencies]);
// 3. Event handlers
const handleClick = () => {
setState(newValue);
};
// 4. Render
return (
<div>
{/* JSX */}
</div>
);
}
```
### Component Size Limit
**Keep components under 150 lines after formatting.**
**If component is too large:**
1. Extract sub-components
2. Move logic to custom hooks
3. Split into multiple files
**To find large components:**
```bash
# Find files >150 lines
find src/ -name "*.jsx" -exec wc -l {} \; | sort -rn | head -20
```
### File Naming
**Components:** PascalCase.jsx
- `HouseholdPage.jsx`
- `PolicySearch.jsx`
- `ImpactChart.jsx`
**Utilities:** camelCase.js
- `formatCurrency.js`
- `apiUtils.js`
- `chartHelpers.js`
**Hooks:** camelCase.js with 'use' prefix
- `useCountryId.js`
- `usePolicy.js`
## Common Development Tasks
### Task 1: Add New Page
1. **See page structure:**
```bash
cat src/pages/HouseholdPage.jsx
```
2. **Create new page:**
```javascript
// src/pages/MyNewPage.jsx
export default function MyNewPage() {
return <div>Content</div>;
}
```
3. **Add route:**
```bash
# See routing
cat src/routing/routes.js
# Add your route following the pattern
```
### Task 2: Add New Chart
1. **See chart examples:**
```bash
ls src/pages/policy/output/
cat src/pages/policy/output/DistributionalImpact.jsx
```
2. **Create chart component:**
```javascript
import Plot from "react-plotly.js";
export default function MyChart({ data }) {
return (
<Plot
data={traces}
layout={{
font: { family: "Roboto Serif" },
plot_bgcolor: "white"
}}
/>
);
}
```
### Task 3: Add Blog Post
1. **See post structure:**
```bash
cat src/posts/articles/harris-eitc.md
```
2. **Create post:**
```bash
# Create markdown file
# src/posts/articles/my-analysis.md
# Follow policyengine-writing-skill for style
```
3. **Images:**
```bash
# Store in public/images/posts/
# Reference in markdown
```
## API Integration Patterns
### Fetching Data
**To see API call patterns:**
```bash
cat src/api/call.js
```
**Standard pattern:**
```javascript
import { call } from "api/call";
const fetchData = async () => {
const result = await call(
`/us/calculate`,
{ household: data },
"POST"
);
return result;
};
```
### Loading States
**Pattern:**
```javascript
const [loading, setLoading] = useState(false);
const [error, setError] = useState(null);
const [data, setData] = useState(null);
useEffect(() => {
setLoading(true);
fetchData()
.then(setData)
.catch(setError)
.finally(() => setLoading(false));
}, [dependencies]);
if (loading) return <Spin />;
if (error) return <Error message={error} />;
return <Data data={data} />;
```
## Performance Patterns
### Code Splitting
**To see code splitting:**
```bash
grep -r "React.lazy" src/
```
**Pattern:**
```javascript
import { lazy, Suspense } from "react";
const HeavyComponent = lazy(() => import("./HeavyComponent"));
function Page() {
return (
<Suspense fallback={<Spin />}>
<HeavyComponent />
</Suspense>
);
}
```
### Memoization
**Use React.memo for expensive components:**
```javascript
import { memo } from "react";
const ExpensiveChart = memo(function ExpensiveChart({ data }) {
// Only re-renders if data changes
return <Plot data={data} />;
});
```
## Accessibility
**Requirements:**
- Semantic HTML elements
- ARIA labels for complex widgets
- Keyboard navigation
- Color contrast (WCAG AA)
**To see accessibility patterns:**
```bash
grep -r "aria-" src/
grep -r "role=" src/
```
## Country-Specific Features
### Country Switching
**To see country switching:**
```bash
cat src/hooks/useCountryId.js
```
**Usage:**
```javascript
import { useCountryId } from "hooks/useCountryId";
function Component() {
const [countryId] = useCountryId(); // "us", "uk", or "ca"
// Load country-specific data
const data = countryId === "us" ? usData : ukData;
}
```
### Country-Specific Content
**Conditional rendering:**
```javascript
{countryId === "us" && <USSpecificComponent />}
{countryId === "uk" && <UKSpecificComponent />}
```
**To find country-specific code:**
```bash
grep -r "countryId ===" src/
```
## Development Workflow
### Local Development
**Start dev server:**
```bash
make debug
# App runs on http://localhost:3000
# Connects to production API by default
```
**Connect to local API:**
```bash
# See environment configuration
cat src/config/environment.js
# Or set environment variable
REACT_APP_API_URL=http://localhost:5000 make debug
```
### Testing
**Run tests:**
```bash
make test
```
**Watch mode:**
```bash
npm test -- --watch
```
**Coverage:**
```bash
npm test -- --coverage
```
### Linting and Formatting
**Format code (critical before committing):**
```bash
make format
# Or manually
npm run lint -- --fix
npx prettier --write .
```
**Check linting (CI check):**
```bash
npm run lint -- --max-warnings=0
```
## Current Implementation Reference
### Component Structure
**To see current page structure:**
```bash
ls src/pages/
# - HouseholdPage.jsx
# - PolicyPage.jsx
# - HomePage.jsx
# - etc.
```
**To see a complete page:**
```bash
cat src/pages/PolicyPage.jsx
```
### API Call Patterns
**To see current API integration:**
```bash
cat src/api/call.js # Base caller
cat src/api/variables.js # Variable metadata fetching
cat src/api/parameters.js # Parameter metadata fetching
```
### Routing Configuration
**To see current routes:**
```bash
cat src/routing/routes.js
```
### Form Controls
**To see PolicyEngine-specific form controls:**
```bash
ls src/controls/
# - InputField.jsx
# - SearchParamControl.jsx
# - etc.
```
### Chart Components
**To see chart implementations:**
```bash
ls src/pages/policy/output/
# - BudgetaryImpact.jsx
# - DistributionalImpact.jsx
# - PovertyImpact.jsx
# - etc.
```
**Reference chart:**
```bash
cat src/pages/policy/output/DistributionalImpact.jsx
```
## Multi-Repository Integration
### How App Relates to Other Repos
```
policyengine-core (engine)
policyengine-us, policyengine-uk (country models)
policyengine-api (backend)
policyengine-app (you are here)
```
**Understanding the stack:**
- See `policyengine-core-skill` for engine concepts
- See `policyengine-us-skill` for what variables/parameters mean
- See `policyengine-api-skill` for API endpoints the app calls
### Blog Posts Reference Country Models
**Blog posts often reference variables:**
```bash
# Posts reference variables like "income_tax", "ctc"
# See policyengine-us-skill for variable definitions
cat src/posts/articles/harris-eitc.md
```
## Common Development Tasks
### Task 1: Add New Parameter to UI
1. **Understand parameter:**
```bash
# See parameter in country model
cd ../policyengine-us
cat policyengine_us/parameters/gov/irs/credits/ctc/amount/base_amount.yaml
```
2. **Find similar parameter in app:**
```bash
cd ../policyengine-app
grep -r "ctc.*amount" src/pages/policy/
```
3. **Add UI control following pattern**
### Task 2: Add New Chart
1. **See existing charts:**
```bash
cat src/pages/policy/output/DistributionalImpact.jsx
```
2. **Create new chart component**
3. **Add to policy output page**
### Task 3: Fix Bug in Calculator
1. **Find relevant component:**
```bash
# Search for the feature
grep -r "keyword" src/pages/
```
2. **Read component code**
3. **Make fix following React patterns**
4. **Test with dev server:**
```bash
make debug
```
## Build and Deployment
**Production build:**
```bash
make build
# Creates optimized bundle in build/
```
**Deployment:**
```bash
# See deployment configuration
cat netlify.toml # or appropriate config
```
**Environment variables:**
```bash
# React env vars must have REACT_APP_ prefix
REACT_APP_API_URL=https://api.policyengine.org
# Or use config file pattern (recommended)
cat src/config/environment.js
```
## Style Guide
**Follow policyengine-standards-skill for:**
- ESLint configuration
- Prettier formatting
- Component size limits
- File organization
**Follow policyengine-writing-skill for:**
- Blog post content
- Documentation
- UI copy
## Resources
**Repository:** https://github.com/PolicyEngine/policyengine-app
**Live app:** https://policyengine.org
**Staging:** https://staging.policyengine.org (if applicable)
**Related skills:**
- **policyengine-api-skill** - Understanding the backend
- **policyengine-us-skill** - Understanding variables/parameters
- **policyengine-writing-skill** - Blog post style
- **policyengine-standards-skill** - Code quality

View File

@@ -0,0 +1,487 @@
---
name: policyengine-core
description: PolicyEngine Core simulation engine - the foundation powering all PolicyEngine calculations
---
# PolicyEngine Core
PolicyEngine Core is the microsimulation engine that powers all PolicyEngine calculations. It's a fork of OpenFisca-Core adapted for PolicyEngine's needs.
## For Users 👥
### What is Core?
When you use policyengine.org to calculate taxes or benefits, PolicyEngine Core is the "calculator" running behind the scenes.
**Core provides:**
- The simulation engine that processes tax rules
- Variable and parameter management
- Entity relationships (person → family → household)
- Period handling (2024, 2025, etc.)
You don't interact with Core directly - you use it through:
- **Web app:** policyengine.org
- **Python packages:** policyengine-us, policyengine-uk
- **API:** api.policyengine.org
### Why Core Matters
Core ensures:
-**Accuracy** - Calculations follow official rules exactly
-**Consistency** - Same rules applied everywhere
-**Transparency** - All rules traceable to legislation
-**Performance** - Vectorized calculations for speed
## For Analysts 📊
### Understanding Core Concepts
When writing PolicyEngine code, you'll encounter Core concepts:
**Variables:**
- Represent quantities (income_tax, ctc, snap, etc.)
- Defined for specific entities (person, household, tax_unit)
- Calculated from formulas or set directly
**Parameters:**
- Policy rules that change over time (tax rates, benefit amounts)
- Organized hierarchically (gov.irs.credits.ctc.amount.base_amount)
- Stored in YAML files
**Entities:**
- Person: Individual
- Family: Family unit
- Tax unit: Tax filing unit
- Household: Physical household
- Marital unit: Marital status grouping
- SPM unit: Supplemental Poverty Measure unit
**Periods:**
- Year: 2024, 2025, etc.
- Month: 2024-01, 2024-02, etc.
- Specific dates: 2024-06-15
### Core in Action
```python
from policyengine_us import Simulation
# When you create a simulation
sim = Simulation(situation=household)
# Core manages:
# - Entity relationships
# - Variable dependencies
# - Parameter lookups
# - Period conversions
# When you calculate
result = sim.calculate("income_tax", 2024)
# Core:
# 1. Checks if already calculated
# 2. Identifies dependencies (income → AGI → taxable income → tax)
# 3. Calculates dependencies first
# 4. Applies formulas
# 5. Returns result
```
### Core vs Country Packages
**Core (policyengine-core):**
- Generic simulation engine
- No specific tax/benefit rules
- Variable and parameter infrastructure
**Country packages (policyengine-us, etc.):**
- Built on Core
- Contain specific tax/benefit rules
- Define variables and parameters for that country
**Relationship:**
```
policyengine-core (engine)
↓ powers
policyengine-us (US rules)
↓ used by
policyengine-api (REST API)
↓ serves
policyengine-app (web interface)
```
## For Contributors 💻
### Repository
**Location:** PolicyEngine/policyengine-core
**Origin:** Fork of OpenFisca-Core
**Clone:**
```bash
git clone https://github.com/PolicyEngine/policyengine-core
```
### Current Architecture
**To see current structure:**
```bash
tree policyengine_core/
# Key directories:
# - variables/ - Variable class and infrastructure
# - parameters/ - Parameter class and infrastructure
# - entities/ - Entity definitions
# - simulations/ - Simulation class
# - periods/ - Period handling
# - reforms/ - Reform application
```
**To understand a specific component:**
```bash
# Variable system
cat policyengine_core/variables/variable.py
# Parameter system
cat policyengine_core/parameters/parameter.py
# Simulation engine
cat policyengine_core/simulations/simulation.py
# Entity system
cat policyengine_core/entities/entity.py
```
### Key Classes
**Variable:**
```python
# To see Variable class implementation
cat policyengine_core/variables/variable.py
# Variables in country packages inherit from this:
from policyengine_core.variables import Variable
class income_tax(Variable):
value_type = float
entity = Person
label = "Income tax"
definition_period = YEAR
def formula(person, period, parameters):
# Vectorized formula
return calculate_tax(...)
```
**Simulation:**
```python
# To see Simulation class implementation
cat policyengine_core/simulations/simulation.py
# Manages calculation graph and caching
sim = Simulation(situation=situation)
sim.calculate("variable", period)
```
**Parameters:**
```python
# To see Parameter handling
cat policyengine_core/parameters/parameter_node.py
# Access in formulas:
parameters(period).gov.irs.credits.ctc.amount.base_amount
```
### Vectorization (Critical!)
Core requires vectorized operations - no if-elif-else with arrays:
**❌ Wrong (scalar logic):**
```python
if age < 18:
eligible = True
else:
eligible = False
```
**✅ Correct (vectorized):**
```python
eligible = age < 18 # NumPy boolean array
```
**Why:** Core processes many households simultaneously for performance.
**To see vectorization examples:**
```bash
# Search for where() usage (vectorized if-then-else)
grep -r "np.where" policyengine_core/
# Find select() usage (vectorized case statements)
grep -r "select" policyengine_core/
```
### Formula Dependencies
Core automatically resolves variable dependencies:
```python
class taxable_income(Variable):
def formula(person, period, parameters):
# Core automatically calculates these first:
agi = person("adjusted_gross_income", period)
deduction = person("standard_deduction", period)
return agi - deduction
class income_tax(Variable):
def formula(person, period, parameters):
# Core knows to calculate taxable_income first
taxable = person("taxable_income", period)
return apply_brackets(taxable, ...)
```
**To see dependency resolution:**
```bash
# Find trace functionality
grep -r "trace" policyengine_core/simulations/
# Enable in your code:
simulation.trace = True
simulation.calculate("income_tax", 2024)
```
### Period Handling
**To see period implementation:**
```bash
cat policyengine_core/periods/period.py
# Period types:
# - YEAR: 2024
# - MONTH: 2024-01
# - ETERNITY: permanent values
```
**Usage in variables:**
```python
# Annual variable
definition_period = YEAR # Called with 2024
# Monthly variable
definition_period = MONTH # Called with "2024-01"
# Convert periods
yearly_value = person("monthly_income", period.this_year) * 12
```
### Testing Core Changes
**To run Core tests:**
```bash
cd policyengine-core
make test
# Specific test
pytest tests/core/test_variables.py -v
```
**To test in country package:**
```bash
# Changes to Core affect all country packages
cd policyengine-us
pip install -e ../policyengine-core # Local development install
make test
```
### Key Differences from OpenFisca
PolicyEngine Core differs from OpenFisca-Core:
**To see PolicyEngine changes:**
```bash
# Compare to OpenFisca
# Core fork diverged to add:
# - Enhanced performance
# - Better error messages
# - PolicyEngine-specific features
# See commit history for PolicyEngine changes
git log --oneline
```
## Core Development Workflow
### Making Changes to Core
1. **Clone repo:**
```bash
git clone https://github.com/PolicyEngine/policyengine-core
```
2. **Install for development:**
```bash
make install
```
3. **Make changes** to variable.py, simulation.py, etc.
4. **Test locally:**
```bash
make test
```
5. **Test in country package:**
```bash
cd ../policyengine-us
pip install -e ../policyengine-core
make test
```
6. **Format and commit:**
```bash
make format
git commit -m "Description"
```
### Understanding Impact
Changes to Core affect:
- ✅ All country packages (US, UK, Canada, IL, NG)
- ✅ The API
- ✅ The web app
- ✅ All analysis tools
**Critical:** Always test in multiple country packages before merging.
## Common Core Patterns
### Pattern 1: Adding a New Variable Type
**Current variable types:**
```bash
# See supported types
grep "value_type" policyengine_core/variables/variable.py
```
**Types:** int, float, bool, str, Enum, date
### Pattern 2: Custom Formulas
**Formula signature:**
```python
def formula(entity, period, parameters):
# entity: Person, TaxUnit, Household, etc.
# period: 2024, "2024-01", etc.
# parameters: Parameter tree for period
return calculated_value
```
**To see formula examples:**
```bash
# Search country packages for formulas
grep -A 10 "def formula" ../policyengine-us/policyengine_us/variables/ | head -50
```
### Pattern 3: Parameter Access
**Accessing parameters in formulas:**
```python
# Navigate parameter tree
param = parameters(period).gov.irs.credits.ctc.amount.base_amount
# Parameters automatically valid for period
# No need to check dates manually
```
**To see parameter structure:**
```bash
# Example from country package
tree ../policyengine-us/policyengine_us/parameters/gov/
```
## Advanced Topics
### Formula Caching
Core caches calculations automatically:
```python
# First call calculates
tax1 = sim.calculate("income_tax", 2024)
# Second call returns cached value
tax2 = sim.calculate("income_tax", 2024) # Instant
```
### Neutralizing Variables
```python
# Set variable to zero in reform
reform = {
"income_tax": {
"2024-01-01.2100-12-31": 0
}
}
```
### Adding Variables
Country packages add variables by inheriting from Core's Variable class.
**See policyengine-us-skill for variable creation patterns.**
## Resources
**Repository:** https://github.com/PolicyEngine/policyengine-core
**Documentation:**
- Core API docs (see README in repo)
- OpenFisca docs (original): https://openfisca.org/doc/
**Related skills:**
- **policyengine-us-skill** - Using Core through country packages
- **policyengine-standards-skill** - Code quality standards
## Troubleshooting
### Common Issues
**Variable not found:**
```python
# Error: Variable 'income_tax' not found
# Solution: Variable is defined in country package, not Core
# Use policyengine-us, not policyengine-core directly
```
**Scalar vs array operations:**
```python
# Error: truth value of array is ambiguous
# Solution: Use np.where() instead of if-else
# See vectorization section above
```
**Period mismatch:**
```python
# Error: Cannot compute variable_name for period 2024-01
# Solution: Check definition_period matches request
# YEAR variables need YEAR periods (2024, not "2024-01")
```
**To debug:**
```python
# Enable tracing
sim.trace = True
sim.calculate("variable", period)
# See calculation dependency tree
```
## Contributing to Core
**Before contributing:**
1. Read Core README
2. Understand OpenFisca architecture
3. Test changes in multiple country packages
4. Follow policyengine-standards-skill
**Development standards:**
- Python 3.10-3.13
- Black formatting (79-char)
- Comprehensive tests
- No breaking changes without discussion

View File

@@ -0,0 +1,356 @@
---
name: policyengine-python-client
description: Using PolicyEngine programmatically via Python client or REST API
---
# PolicyEngine Python Client
This skill covers programmatic access to PolicyEngine for analysts and researchers.
## Installation
```bash
# Install the Python client
pip install policyengine
# Or for local development
pip install policyengine-us # Just the US model (offline)
```
## Quick Start: Python Client
```python
from policyengine import Simulation
# Create a household
household = {
"people": {
"you": {
"age": {"2024": 30},
"employment_income": {"2024": 50000}
}
},
"households": {
"your household": {
"members": ["you"],
"state_name": {"2024": "CA"}
}
}
}
# Run simulation
sim = Simulation(situation=household, country_id="us")
income_tax = sim.calculate("income_tax", "2024")
```
## For Users: Why Use Python?
**Web app limitations:**
- ✅ Great for exploring policies interactively
- ❌ Can't analyze many households at once
- ❌ Can't automate repetitive analyses
- ❌ Limited customization of charts
**Python benefits:**
- ✅ Analyze thousands of households in batch
- ✅ Automate regular policy analysis
- ✅ Create custom visualizations
- ✅ Integrate with other data sources
- ✅ Reproducible research
## For Analysts: Common Workflows
### Workflow 1: Calculate Your Own Taxes
```python
from policyengine import Simulation
# Your household (more complex than web app)
household = {
"people": {
"you": {
"age": {"2024": 35},
"employment_income": {"2024": 75000},
"qualified_dividend_income": {"2024": 5000},
"charitable_cash_donations": {"2024": 3000}
},
"spouse": {
"age": {"2024": 33},
"employment_income": {"2024": 60000}
},
"child1": {"age": {"2024": 8}},
"child2": {"age": {"2024": 5}}
},
# ... entities setup (see policyengine-us-skill)
}
sim = Simulation(situation=household, country_id="us")
# Calculate specific values
federal_income_tax = sim.calculate("income_tax", "2024")
state_income_tax = sim.calculate("state_income_tax", "2024")
ctc = sim.calculate("ctc", "2024")
eitc = sim.calculate("eitc", "2024")
print(f"Federal income tax: ${federal_income_tax:,.0f}")
print(f"State income tax: ${state_income_tax:,.0f}")
print(f"Child Tax Credit: ${ctc:,.0f}")
print(f"EITC: ${eitc:,.0f}")
```
### Workflow 2: Analyze a Policy Reform
```python
from policyengine import Simulation
# Define reform (increase CTC to $5,000)
reform = {
"gov.irs.credits.ctc.amount.base_amount": {
"2024-01-01.2100-12-31": 5000
}
}
# Compare baseline vs reform
household = create_household() # Your household definition
sim_baseline = Simulation(situation=household, country_id="us")
sim_reform = Simulation(situation=household, country_id="us", reform=reform)
ctc_baseline = sim_baseline.calculate("ctc", "2024")
ctc_reform = sim_reform.calculate("ctc", "2024")
print(f"CTC baseline: ${ctc_baseline:,.0f}")
print(f"CTC reform: ${ctc_reform:,.0f}")
print(f"Increase: ${ctc_reform - ctc_baseline:,.0f}")
```
### Workflow 3: Batch Analysis
```python
import pandas as pd
from policyengine import Simulation
# Analyze multiple households
households = [
{"income": 30000, "children": 0},
{"income": 50000, "children": 2},
{"income": 100000, "children": 3},
]
results = []
for h in households:
situation = create_household(income=h["income"], num_children=h["children"])
sim = Simulation(situation=situation, country_id="us")
results.append({
"income": h["income"],
"children": h["children"],
"income_tax": sim.calculate("income_tax", "2024"),
"ctc": sim.calculate("ctc", "2024"),
"eitc": sim.calculate("eitc", "2024")
})
df = pd.DataFrame(results)
print(df)
```
## Using the REST API Directly
### Authentication
**Public access:**
- 100 requests per minute (unauthenticated)
- No API key needed for basic use
**Authenticated access:**
- 1,000 requests per minute
- Contact hello@policyengine.org for API key
### Key Endpoints
**Calculate household impact:**
```python
import requests
url = "https://api.policyengine.org/us/calculate"
payload = {
"household": household_dict,
"policy_id": reform_id # or None for baseline
}
response = requests.post(url, json=payload)
result = response.json()
```
**Get policy details:**
```python
# Get policy metadata
response = requests.get("https://api.policyengine.org/us/policy/12345")
policy = response.json()
```
**Get parameter values:**
```python
# Get current parameter value
response = requests.get(
"https://api.policyengine.org/us/parameter/gov.irs.credits.ctc.amount.base_amount"
)
parameter = response.json()
```
### For Full API Documentation
**OpenAPI spec:** https://api.policyengine.org/docs
**To explore:**
```bash
# View all endpoints
curl https://api.policyengine.org/docs
# Test calculate endpoint
curl -X POST https://api.policyengine.org/us/calculate \
-H "Content-Type: application/json" \
-d '{"household": {...}}'
```
## Limitations and Considerations
### Rate Limits
**Unauthenticated:**
- 100 requests/minute
- Good for exploratory analysis
**Authenticated:**
- 1,000 requests/minute
- Required for production use
### Data Privacy
- PolicyEngine does not store household data
- All calculations happen server-side and are not logged
- Reform URLs are public (don't include personal info in reforms)
### Performance
**API calls:**
- Simple household: ~200-500ms
- Population impact: ~5-30 seconds (varies by reform)
- Use caching for repeated calculations
**Local simulation (policyengine-us):**
- Faster for batch analysis
- No rate limits
- No network dependency
- Limited to one country per package
## Choosing Local vs API
### Use Local (policyengine-us package)
**When:**
- Batch analysis of many households
- Need offline capability
- Analyzing parameter sweeps (axes)
- Development/testing
**Install:**
```bash
pip install policyengine-us # US only
pip install policyengine-uk # UK only
```
**Example:**
```python
from policyengine_us import Simulation
# Works offline
sim = Simulation(situation=household)
```
### Use API (policyengine or requests)
**When:**
- Multi-country analysis
- Using latest model version
- Don't want to manage dependencies
- Integration with web services
**Example:**
```python
import requests
# Requires internet
response = requests.post("https://api.policyengine.org/us/calculate", ...)
```
## For Contributors: Understanding the Client
**Repository:** PolicyEngine/policyengine.py
**To see implementation:**
```bash
# Clone the client
git clone https://github.com/PolicyEngine/policyengine.py
# See the Simulation class
cat policyengine/simulation.py
# See API integration
cat policyengine/api.py
```
**Architecture:**
- `Simulation` class wraps API calls
- `calculate()` method handles caching
- Transparent fallback between API and local
## Advanced: Direct Country Package Usage
For maximum control and performance, use country packages directly:
```python
from policyengine_us import Simulation
# Full control over situation structure
situation = {
# Complete situation dictionary
# See policyengine-us-skill for patterns
}
sim = Simulation(situation=situation)
result = sim.calculate("variable_name", 2024)
```
**Benefits:**
- No API dependency
- Faster (no network)
- Full access to all variables
- Use axes for parameter sweeps
**See policyengine-us-skill for detailed patterns.**
## Examples and Tutorials
**PolicyEngine documentation:**
- US: https://policyengine.org/us/docs
- UK: https://policyengine.org/uk/docs
**Example notebooks:**
- Repository: PolicyEngine/analysis-notebooks
- See policyengine-analysis-skill for analysis patterns
**Community examples:**
- Blog posts: policyengine.org/us/research
- GitHub discussions: github.com/PolicyEngine discussions
## Getting Help
**For usage questions:**
- GitHub Discussions: https://github.com/PolicyEngine/policyengine-us/discussions
**For bugs:**
- File issues in appropriate repo (policyengine-us, policyengine.py, etc.)
**For collaboration:**
- Email: hello@policyengine.org