Initial commit
This commit is contained in:
Zhongwei Li
190
commands/list-unstable-tests.md
Normal file
190
commands/list-unstable-tests.md
Normal file
@@ -0,0 +1,190 @@
|
||||
---
|
||||
description: List unstable tests with pass rate below 95%
|
||||
argument-hint: <version> <keywords> [sippy-url]
|
||||
---
|
||||
|
||||
## Name
|
||||
ci:list-unstable-tests
|
||||
|
||||
## Synopsis
|
||||
```
|
||||
/ci:list-unstable-tests <version> <keywords> [sippy-url]
|
||||
```
|
||||
|
||||
## Description
|
||||
The `ci:list-unstable-tests` command queries OpenShift CI test results from Sippy and lists all tests matching the keywords that have a pass rate below 95%. This is useful for quickly identifying unstable tests that need attention.
|
||||
|
||||
By default, it queries the production Sippy instance at `sippy.dptools.openshift.org`. You can optionally specify a different Sippy instance URL to query alternative environments (e.g., QE component readiness).
|
||||
|
||||
This command is useful for:
|
||||
- Identifying unstable tests with inconsistent pass rates
|
||||
- Finding regression candidates for investigation
|
||||
- Generating reports of unstable test cases
|
||||
- Prioritizing test stabilization efforts
|
||||
- Quality gate checks before releases
|
||||
|
||||
## Arguments
|
||||
- `$1` (version): OpenShift version to query (e.g., "4.21", "4.20", "4.19")
|
||||
- `$2` (keywords): Keywords to search in test names (e.g., "olmv1", "sig-storage", "operator")
|
||||
- `$3` (sippy-url) [optional]: Sippy instance base URL. Defaults to "sippy.dptools.openshift.org" if not provided. Examples: "qe-component-readiness.dptools.openshift.org"
|
||||
|
||||
## Implementation
|
||||
|
||||
1. **Parse Arguments**
|
||||
- Extract version from `$1` (e.g., "4.20")
|
||||
- Extract keywords from `$2` (e.g., "olmv1")
|
||||
- Extract Sippy URL from `$3` if provided, otherwise use default "sippy.dptools.openshift.org"
|
||||
- Normalize URL to extract base domain for API endpoint (strip "/sippy-ng/" suffix if present)
|
||||
- Add "https://" prefix if not already present
|
||||
|
||||
2. **Build Sippy API Request**
|
||||
- Construct filter JSON for the `/api/tests` endpoint:
|
||||
```python
|
||||
filters = {
|
||||
"items": [
|
||||
{
|
||||
"columnField": "name",
|
||||
"not": False,
|
||||
"operatorValue": "contains",
|
||||
"value": keywords
|
||||
}
|
||||
],
|
||||
"linkOperator": "and"
|
||||
}
|
||||
```
|
||||
- Set query parameters:
|
||||
- `release`: The OpenShift version
|
||||
- `filter`: JSON-encoded filter object
|
||||
- `sort`: "asc"
|
||||
- `sortField`: "current_pass_percentage"
|
||||
|
||||
3. **Query Test Statistics**
|
||||
- Construct API endpoint from provided URL: `https://{base_url}/api/tests`
|
||||
- Make GET request to the constructed endpoint
|
||||
- Parse response to extract test data
|
||||
|
||||
4. **Filter Tests Below 95% Pass Rate**
|
||||
- Iterate through all returned tests
|
||||
- Filter tests where `current_pass_percentage < 95`
|
||||
- Sort filtered results by pass percentage (ascending, worst first)
|
||||
- Collect the following data for each unstable test:
|
||||
- Test name
|
||||
- Current pass percentage
|
||||
- Total runs
|
||||
- Passes
|
||||
- Failures
|
||||
- Net improvement (trend indicator)
|
||||
|
||||
5. **Format and Display Results**
|
||||
- Display summary header with:
|
||||
- Total number of tests matching keywords
|
||||
- Number of tests below 95% pass rate
|
||||
- Percentage of unstable tests
|
||||
- List each unstable test with:
|
||||
- Test name
|
||||
- Pass rate percentage
|
||||
- Run statistics (runs/passes/failures)
|
||||
- Trend indicator (net improvement)
|
||||
- Sort by pass percentage (worst tests first)
|
||||
- If no tests are below 95%, display success message indicating all tests are stable
|
||||
|
||||
## Return Value
|
||||
|
||||
**Format**: Formatted text output with:
|
||||
|
||||
**Summary Section:**
|
||||
- Total tests matching keywords
|
||||
- Tests below 95% pass rate (unstable tests)
|
||||
- Overall stability percentage
|
||||
|
||||
**Unstable Tests List:**
|
||||
For each test with pass rate < 95%:
|
||||
- Test name
|
||||
- Pass rate percentage
|
||||
- Total runs
|
||||
- Passes
|
||||
- Failures
|
||||
- Net improvement
|
||||
|
||||
If all tests pass at 95% or above, display a success message indicating all tests are stable.
|
||||
|
||||
## Examples
|
||||
|
||||
1. **List unstable OLMv1 tests from QE Sippy**:
|
||||
```
|
||||
/ci:list-unstable-tests 4.20 olmv1 qe-component-readiness.dptools.openshift.org
|
||||
```
|
||||
|
||||
Lists all OLMv1-related tests in version 4.20 from QE Sippy that have a pass rate below 95%.
|
||||
|
||||
2. **List unstable storage tests (using default Sippy)**:
|
||||
```
|
||||
/ci:list-unstable-tests 4.21 sig-storage
|
||||
```
|
||||
|
||||
Lists all storage-related tests in version 4.21 from production Sippy with pass rate below 95%.
|
||||
|
||||
3. **List unstable operator tests**:
|
||||
```
|
||||
/ci:list-unstable-tests 4.19 operator
|
||||
```
|
||||
|
||||
Lists all operator-related tests in version 4.19 with pass rate below 95%.
|
||||
|
||||
4. **Check specific component stability**:
|
||||
```
|
||||
/ci:list-unstable-tests 4.20 sig-network qe-component-readiness.dptools.openshift.org
|
||||
```
|
||||
|
||||
Lists all network-related unstable tests from QE Sippy.
|
||||
|
||||
## Notes
|
||||
|
||||
- **Pass Rate Threshold**: Fixed at 95% - tests with pass rate >= 95% are considered stable
|
||||
- **Default Sippy URL**: If no Sippy URL is provided, the command uses `sippy.dptools.openshift.org` by default
|
||||
- The command queries data from the last 7 days by default
|
||||
- Ensure you can access the Sippy API endpoints
|
||||
- Results are sorted by pass percentage (ascending) to show most unstable tests first
|
||||
- The net improvement metric shows if the test is getting worse (negative) or better (positive)
|
||||
- If no tests match the keywords, an appropriate message will be displayed
|
||||
- If all matching tests have pass rate >= 95%, a success message will be shown indicating all tests are stable
|
||||
|
||||
## Output Example
|
||||
|
||||
```
|
||||
================================================================================
|
||||
Unstable Tests Report - 4.20 olmv1
|
||||
================================================================================
|
||||
Sippy Instance: qe-component-readiness.dptools.openshift.org
|
||||
Pass Rate Threshold: < 95%
|
||||
|
||||
Summary:
|
||||
Total Tests Matching 'olmv1': 45
|
||||
Unstable Tests (< 95%): 8 (17.8%)
|
||||
Stable Tests (>= 95%): 37 (82.2%)
|
||||
|
||||
================================================================================
|
||||
Tests Below 95% Pass Rate (sorted by worst first):
|
||||
================================================================================
|
||||
|
||||
1. Test: [sig-olmv1] clusterextension install should fail validation
|
||||
Pass Rate: 23.5%
|
||||
Runs: 17 | Passes: 4 | Failures: 13
|
||||
Net Improvement: -45.2
|
||||
|
||||
2. Test: [sig-olmv1] clusterextension upgrade from v1 to v2
|
||||
Pass Rate: 67.8%
|
||||
Runs: 28 | Passes: 19 | Failures: 9
|
||||
Net Improvement: -12.3
|
||||
|
||||
[... additional tests ...]
|
||||
|
||||
================================================================================
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- `/ci:query-test-result` - Query detailed results for a specific test
|
||||
- Sippy UI (Production): https://sippy.dptools.openshift.org/sippy-ng/
|
||||
- Sippy UI (QE): https://qe-component-readiness.dptools.openshift.org
|
||||
- Sippy API Documentation: https://github.com/openshift/sippy
|
||||
Reference in New Issue
Block a user