Initial commit

skills/buildkite-status/references/buildkite-states.md

@@ -0,0 +1,107 @@
+# Buildkite Build and Job States
+
+Understanding Buildkite states is critical for correctly interpreting build status. Some states are misleading or require additional context.
+
+## Build States
+
+### Terminal States
+
+- **`passed`** - All jobs completed successfully
+- **`failed`** - One or more jobs failed
+- **`canceled`** - Build was manually canceled
+- **`skipped`** - Build was skipped (e.g., due to branch filters)
+- **`blocked`** - Build is waiting for manual approval via block step
+
+### Active States
+
+- **`running`** - Build is currently executing
+- **`scheduled`** - Build is queued and waiting to start
+- **`creating`** - Build is being created
+
+## Job States
+
+### Terminal States
+
+- **`passed`** - Job completed successfully
+- **`failed`** - Job failed with non-zero exit code
+- **`canceled`** - Job was canceled
+- **`skipped`** - Job was skipped
+- **`timed_out`** - Job exceeded time limit
+
+### Special States (Often Misleading)
+
+- **`broken`** - This is the most misleading state. It can mean:
+
+  - Job was skipped because an earlier job in the pipeline failed
+  - Job was skipped due to dependency conditions not being met
+  - Job was skipped due to conditional logic in the pipeline config
+  - **NOT necessarily a failure of this specific job**
+
+  Example: In the zen-payroll pipeline, many jobs show as "broken" but are actually skipped because their dependencies indicated they weren't needed (e.g., no relevant file changes).
+
+- **`soft_failed`** - Job failed but was marked as "soft fail" (doesn't block pipeline)
+  - Shows as failed but doesn't cause overall build failure
+  - Often used for optional checks or flaky tests
+
+### Active States
+
+- **`waiting`** - Job is waiting for dependencies
+- **`waiting_failed`** - Job was waiting but its dependency failed
+- **`assigned`** - Job has been assigned to an agent
+- **`accepted`** - Agent has accepted the job
+- **`running`** - Job is currently executing
+- **`blocked`** - Job is a block step waiting for manual unblock
+
+## Interpreting Build Status
+
+### Progressive Disclosure Pattern
+
+When checking build status, follow this pattern:
+
+1. **Start with overall state**: `passed`, `failed`, `canceled`, `blocked`
+2. **If failed, check job summary**: How many jobs failed vs broken vs passed?
+3. **Examine failed jobs specifically**: Don't assume "broken" means the job itself failed
+4. **Check annotations**: Some projects surface important failures in annotations
+5. **Inspect logs**: For actual failures, read the job logs
+
+### Common Pitfalls
+
+1. **Treating "broken" as "failed"**: A "broken" job is often just skipped due to pipeline logic, not an actual failure.
+
+2. **Ignoring soft fails**: Jobs marked as `soft_failed` may contain important information even though they don't block the build.
+
+3. **Missing blocked builds**: A `blocked` build is waiting for approval and won't progress without manual intervention.
+
+4. **Overlooking job dependencies**: Jobs may be skipped (`broken`) because their dependencies weren't met, which is expected behavior.
+
+## Project-Specific Patterns
+
+### zen-payroll Pipeline
+
+- **Heavy use of conditional execution**: Many jobs are conditionally skipped based on file changes
+- **"broken" is normal**: A build with many "broken" jobs may still be perfectly healthy
+- **Check annotations**: Important test failures are often surfaced in build annotations
+- **Multiple test suites**: Different test types (unit, integration, system) have different failure patterns
+
+### Smaller Pipelines (e.g., gusto-karafka)
+
+- **Fewer conditional jobs**: Most jobs are expected to run
+- **"broken" usually indicates a problem**: Less conditional logic means broken jobs are more likely to be actual issues
+- **Simpler job graphs**: Easier to trace why a job didn't run
+- **May not use annotations**: Failures are usually just in job logs
+
+## When to Investigate
+
+Investigate a build when:
+
+1. Overall build state is `failed`
+2. Jobs show `failed` state (not just `broken`)
+3. Build is `blocked` and you need to unblock it
+4. Annotations contain error messages
+5. Job logs show actual errors (red output, stack traces, test failures)
+
+Don't automatically investigate when:
+
+1. Build is `passed` (even if some jobs are `broken`)
+2. Jobs are `soft_failed` unless specifically requested
+3. Jobs are `broken` due to conditional execution (check pipeline config)