gh-policyengine-policyengine-claude/commands/review-pr.md

description

description
Review an existing PR and post findings to GitHub (does not make code changes)

Reviewing PR: $ARGUMENTS

READ-ONLY MODE: This command analyzes the PR and posts a comprehensive review to GitHub WITHOUT making any code changes.

Use /fix-pr to apply fixes automatically.

Important: Avoiding Duplicate Reviews

Before posting, check if you already have an active review on this PR. If you do AND there are no replies from others:

• Delete your old comments
• Post a single updated review
• DO NOT create multiple review comments

Only post additional reviews if others have engaged with your previous review.

Determining Which PR to Review

First, determine which PR to review based on the arguments:

# If no arguments provided, use current branch's PR
if [ -z "$ARGUMENTS" ]; then
    CURRENT_BRANCH=$(git branch --show-current)
    PR_NUMBER=$(gh pr list --head "$CURRENT_BRANCH" --json number --jq '.[0].number')
    if [ -z "$PR_NUMBER" ]; then
        echo "No PR found for current branch $CURRENT_BRANCH"
        exit 1
    fi
# If argument is a number, use it directly
elif [[ "$ARGUMENTS" =~ ^[0-9]+$ ]]; then
    PR_NUMBER=$ARGUMENTS
# Otherwise, search for PR by description/title
else
    PR_NUMBER=$(gh pr list --search "$ARGUMENTS" --json number,title --jq '.[0].number')
    if [ -z "$PR_NUMBER" ]; then
        echo "No PR found matching: $ARGUMENTS"
        exit 1
    fi
fi

echo "Reviewing PR #$PR_NUMBER"

Phase 1: PR Analysis

Gather context about the PR:

gh pr view $PR_NUMBER --comments
gh pr checks $PR_NUMBER
gh pr diff $PR_NUMBER

Document findings:

• Current CI status
• Existing review comments
• Files changed
• Type of implementation (new program, bug fix, enhancement)

Phase 2: Comprehensive Validation (Read-Only)

Run validators to COLLECT issues (do not fix):

Step 1: Domain Validation

Invoke implementation-validator to identify:

• Federal/state jurisdiction issues
• Variable naming violations
• Hard-coded values
• Performance optimization opportunities
• Documentation gaps
• PolicyEngine pattern violations

IMPORTANT: Instruct agent to ONLY report issues, not fix them.

Step 2: Reference Validation

Invoke reference-validator to check:

• Missing parameter references
• References that don't corroborate values
• Incorrect source types (federal vs state)
• Generic/vague citations

Step 3: Implementation Validation

Invoke implementation-validator to identify:

• Hard-coded values in formulas
• Incomplete implementations (TODOs, stubs)
• Entity usage errors
• Formula pattern violations

Step 4: Test Coverage Analysis

Invoke edge-case-generator to identify:

• Missing boundary tests
• Untested edge cases
• Parameter combinations not tested

Step 5: Documentation Review

Invoke documentation-enricher to find:

• Missing docstrings
• Lack of calculation examples
• Missing regulatory references
• Unclear variable descriptions

Phase 3: Collect and Organize Findings

Aggregate all issues into structured format:

{
  "overall_summary": "Brief summary of PR quality and main concerns",
  "severity": "APPROVE|COMMENT|REQUEST_CHANGES",
  "line_comments": [
    {
      "path": "policyengine_us/variables/gov/states/ma/dta/ccfa/income.py",
      "line": 42,
      "body": "💡 **Hard-coded value detected**\n\nThis value should be parameterized:\n\n```suggestion\nmax_income = parameters(period).gov.states.ma.dta.ccfa.max_income\n```\n\n📚 See: [Parameter Guidelines](https://github.com/PolicyEngine/policyengine-us/blob/master/CLAUDE.md#parameter-structure-best-practices)"
    },
    {
      "path": "policyengine_us/tests/variables/gov/states/ma/dta/ccfa/test_eligibility.yaml",
      "line": 15,
      "body": "🧪 **Missing edge case test**\n\nConsider adding a test for income exactly at the threshold:\n\n```yaml\n- name: At threshold\n  period: 2024-01\n  input:\n    household_income: 75_000\n  output:\n    ma_ccfa_eligible: true\n```"
    }
  ],
  "general_comments": [
    "Consider adding integration tests for multi-child households",
    "Documentation could benefit from real-world calculation examples"
  ]
}

Phase 4: Check for Existing Reviews

IMPORTANT: Before posting a new review, check if you already have an active review:

# Check for existing reviews from you (Claude)
EXISTING_REVIEWS=$(gh api "/repos/{owner}/{repo}/pulls/$PR_NUMBER/reviews" \
  --jq '[.[] | select(.user.login == "MaxGhenis") | {id: .id, state: .state, submitted_at: .submitted_at}]')

# Check if there are any comments/replies from others since your last review
LATEST_REVIEW_TIME=$(echo "$EXISTING_REVIEWS" | jq -r '.[0].submitted_at // empty')

if [ -n "$LATEST_REVIEW_TIME" ]; then
  # Check for comments after your review
  COMMENTS_AFTER=$(gh api "/repos/{owner}/{repo}/issues/$PR_NUMBER/comments" \
    --jq "[.[] | select(.created_at > \"$LATEST_REVIEW_TIME\" and .user.login != \"MaxGhenis\")]")

  if [ -z "$COMMENTS_AFTER" ] || [ "$COMMENTS_AFTER" == "[]" ]; then
    echo "Existing review found with no replies from others"
    echo "STRATEGY: Delete old comments and post single updated review"

    # Delete old comment-only reviews/comments
    gh api "/repos/{owner}/{repo}/issues/$PR_NUMBER/comments" \
      --jq '.[] | select(.user.login == "MaxGhenis") | .id' | \
      while read comment_id; do
        gh api --method DELETE "/repos/{owner}/{repo}/issues/comments/$comment_id"
      done
  else
    echo "Others have replied to your review - will add new review"
  fi
fi

Phase 5: Post GitHub Review

Post the review using GitHub CLI:

# Create review comments JSON
cat > /tmp/review_comments.json <<'EOF'
[FORMATTED_COMMENTS_ARRAY]
EOF

# Post the review with inline comments
gh api \
  --method POST \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  "/repos/{owner}/{repo}/pulls/$PR_NUMBER/reviews" \
  -f body="$OVERALL_SUMMARY" \
  -f event="$SEVERITY" \
  -F comments=@/tmp/review_comments.json

Severity Guidelines

• APPROVE: Minor suggestions only, no critical issues
• COMMENT: Has issues but not blocking (educational feedback)
• REQUEST_CHANGES: Has critical issues that must be fixed:
  • Hard-coded values
  • Missing tests for core functionality
  • Incorrect implementations
  • Missing required documentation

Phase 6: Post Summary Comment

Add a comprehensive summary as a regular comment:

gh pr comment $PR_NUMBER --body "## 📋 Review Summary

### ✅ Strengths
- [List positive aspects]
- Well-structured variable organization
- Clear parameter names

### 🔍 Issues Found

#### Critical (Must Fix)
- [ ] 3 hard-coded values in income calculations
- [ ] Missing edge case tests for boundary conditions
- [ ] Parameter references not corroborated by sources

#### Suggestions (Optional)
- [ ] Consider vectorization in \`calculate_benefit()\`
- [ ] Add calculation walkthrough to documentation
- [ ] Extract shared logic into helper variable

### 📊 Validation Results
- **Domain Validation**: X issues
- **Reference Validation**: X issues
- **Implementation Validation**: X issues
- **Test Coverage**: X gaps identified
- **Documentation**: X improvements suggested

### 🚀 Next Steps

To apply these fixes automatically, run:
\`\`\`bash
/fix-pr $PR_NUMBER
\`\`\`

Or address manually and re-request review when ready.

---
💡 **Tip**: Use \`/fix-pr\` to automatically apply all suggested fixes."

Phase 6: CI Status Check

If CI is failing, include CI failure summary:

gh pr checks $PR_NUMBER --json name,status,conclusion \
  --jq '.[] | select(.conclusion == "failure") | "❌ " + .name'

Add to review:

### ⚠️ CI Failures
- ❌ lint: Black formatting issues
- ❌ test: 3 test failures in test_income.py

Run `/fix-pr` to automatically fix these issues.

Command Options

Usage Examples

• /review-pr - Review PR for current branch
• /review-pr 6390 - Review PR #6390
• /review-pr "Massachusetts CCFA" - Search for and review PR by title

Validation Modes

Standard Review (Default)

• All validators run
• Comprehensive analysis
• Balanced severity

Quick Review

/review-pr [PR] --quick

• Skip deep validation
• Focus on CI failures and critical issues
• Faster turnaround

Strict Review

/review-pr [PR] --strict

• Maximum scrutiny
• Flag even minor style issues
• Request changes for any violations

Output Format

The review will include:

1. Overall Summary: High-level assessment
2. Inline Comments: Specific issues at exact lines
3. Code Suggestions: GitHub suggestion blocks where applicable
4. Summary Comment: Checklist of all issues
5. Next Steps: How to address findings

Review Categories

🔴 Critical Issues (Always flag)

• Hard-coded values in formulas
• Missing parameter references
• Incorrect formula implementations
• Missing tests for core functionality
• CI failures

🟡 Suggestions (Optional improvements)

• Performance optimizations
• Documentation enhancements
• Code style improvements
• Additional test coverage
• Refactoring opportunities

🟢 Positive Feedback

• Well-implemented patterns
• Good test coverage
• Clear documentation
• Proper parameterization

Success Criteria

A good review should:

• ✅ Identify all hard-coded values
• ✅ Flag missing/incorrect references
• ✅ Suggest specific improvements with examples
• ✅ Provide actionable next steps
• ✅ Be respectful and constructive
• ✅ Include code suggestion blocks for easy application

Common Review Patterns

Hard-coded Value

💡 **Hard-coded value detected**

This value should be parameterized. Create a parameter:

\`\`\`yaml
# policyengine_us/parameters/gov/states/ma/program/threshold.yaml
values:
  2024-01-01: 75000
metadata:
  unit: currency-USD
  period: year
  reference: [Link to source]
\`\`\`

Then use in formula:
\`\`\`suggestion
threshold = parameters(period).gov.states.ma.program.threshold
\`\`\`

Missing Test

🧪 **Missing edge case test**

Add test for boundary condition:

\`\`\`yaml
- name: At exact threshold
  period: 2024-01
  input:
    income: 75_000
  output:
    eligible: false  # Just above threshold
\`\`\`

Missing Documentation

📚 **Documentation enhancement**

Add docstring with example:

\`\`\`suggestion
def formula(person, period, parameters):
    """
    Calculate benefit amount.

    Example:
        Income $50,000 → Benefit $2,400
        Income $75,000 → Benefit $1,200

    Reference: MA DTA Regulation 106 CMR 702.310
    """
\`\`\`

Performance Issue

⚡ **Vectorization opportunity**

Replace loop with vectorized operation:

\`\`\`suggestion
return where(
    household_size == 1,
    base_amount,
    base_amount * household_size * 0.7
)
\`\`\`

Error Handling

If validation agents fail:

1. Run basic validation manually
2. Document what couldn't be checked
3. Note limitations in review
4. Still post whatever findings were collected

Pre-Flight Checklist

Before starting, confirm:

• I will NOT make any code changes
• I will ONLY analyze and post reviews
• I will collect findings from all validators
• I will format findings as GitHub review comments
• I will provide actionable suggestions
• I will be constructive and respectful

Start by determining which PR to review, then proceed to Phase 1: Analyze the PR.