65 lines
3.1 KiB
Markdown
65 lines
3.1 KiB
Markdown
# Baseline Test Analysis (RED Phase)
|
|
|
|
## Identified Failure Patterns
|
|
|
|
Without the skill, agents exhibit these consistent problems:
|
|
|
|
### 1. Doesn't Know GitHub CLI Commands for Checks
|
|
- **Symptom**: Asks user for more context instead of using available commands
|
|
- **Example**: "Could you share the specific error messages?" instead of `gh pr checks`
|
|
- **Root Cause**: Doesn't know `gh pr checks` or `gh api check-runs` commands exist
|
|
|
|
### 2. Incorrect Konflux Check Pattern Matching
|
|
- **Symptom**: Suggests wrong prefixes like `konflux/`, `build/`, `ci/`
|
|
- **Correct Pattern**: Check NAME contains "konflux" (case-insensitive) or URL contains "konflux"
|
|
- **Examples of Correct Names**: "Red Hat Konflux / ...", "Konflux Fedora / ...", "Integration Service / ..."
|
|
|
|
### 3. Suggests Manual Browser-Based Workflows
|
|
- **Symptom**: Recommends `gh pr view --web` to open browser
|
|
- **Better Approach**: Use `gh pr view --json` or `gh api` for programmatic access
|
|
- **Impact**: Can't automate or script the workflow
|
|
|
|
### 4. Confuses GitHub Actions with Konflux Checks
|
|
- **Symptom**: Suggests looking at "Actions" tab or `gh run list`
|
|
- **Correct**: Konflux uses GitHub Checks API (check-runs), not Actions workflows
|
|
- **Impact**: Sends user to wrong place entirely
|
|
|
|
### 5. Doesn't Know How to Extract URLs from Check Output
|
|
- **Symptom**: Suggests checking cluster with kubectl instead of getting URL from GitHub first
|
|
- **Missing Knowledge**: Integration test checks have PipelineRun URLs in `output.text` field
|
|
- **Command Needed**: `gh api repos/.../commits/.../check-runs --jq '.check_runs[].output.text'`
|
|
|
|
### 6. Good at Manual URL Parsing (Positive Finding!)
|
|
- **Works Well**: When given a URL directly, can extract cluster/namespace/pipelinerun
|
|
- **Keep**: Agents understand URL structure, just need to know how to GET the URLs
|
|
|
|
### 7. Doesn't Distinguish Build vs Integration Test Checks
|
|
- **Missing**: Knowledge that `-on-pull-request` and `-on-push` are build checks with direct URLs
|
|
- **Missing**: Integration test checks need output parsing
|
|
|
|
## Key Gaps to Address in Skill
|
|
|
|
1. **Teach `gh pr checks` and `gh api` commands**
|
|
2. **Correct Konflux check identification pattern** (case-insensitive "konflux" in name/URL)
|
|
3. **Distinguish build checks (direct URL) from integration tests (parse output)**
|
|
4. **Show `-on-pull-request` vs `-on-push` naming**
|
|
5. **Emphasize programmatic API access over browser workflows**
|
|
6. **Clarify Checks API vs Actions workflows**
|
|
|
|
## Rationalizations Observed
|
|
|
|
- "I don't have direct access" → Actually does via gh CLI
|
|
- "I cannot use WebSearch" → Doesn't need to, can use gh API
|
|
- "I recommend you check manually" → Should provide commands
|
|
- "Would you like me to help you with..." → Should just DO it
|
|
|
|
## Success Criteria for GREEN Phase
|
|
|
|
Agent should:
|
|
- ✅ Use `gh pr checks` or `gh api` immediately (no asking for context)
|
|
- ✅ Filter for "konflux" (case-insensitive) in check names
|
|
- ✅ Extract PipelineRun URLs programmatically
|
|
- ✅ Distinguish -on-pull-request vs -on-push
|
|
- ✅ Know build checks have direct URLs, integration tests need parsing
|
|
- ✅ Provide complete kubectl-ready information (cluster, namespace, pipelinerun)
|