519 lines
15 KiB
Markdown
519 lines
15 KiB
Markdown
---
|
|
name: ml-pipeline-orchestrator
|
|
description: |
|
|
Orchestrates complete machine learning pipelines within SpecWeave increments. Activates when users request "ML pipeline", "train model", "build ML system", "end-to-end ML", "ML workflow", "model training pipeline", or similar. Guides users through data preprocessing, feature engineering, model training, evaluation, and deployment using SpecWeave's spec-driven approach. Integrates with increment lifecycle for reproducible ML development.
|
|
---
|
|
|
|
# ML Pipeline Orchestrator
|
|
|
|
## Overview
|
|
|
|
This skill transforms ML development into a SpecWeave increment-based workflow, ensuring every ML project follows the same disciplined approach: spec → plan → tasks → implement → validate. It orchestrates the complete ML lifecycle from data exploration to model deployment, with full traceability and living documentation.
|
|
|
|
## Core Philosophy
|
|
|
|
**SpecWeave + ML = Disciplined Data Science**
|
|
|
|
Traditional ML development often lacks structure:
|
|
- ❌ Jupyter notebooks with no version control
|
|
- ❌ Experiments without documentation
|
|
- ❌ Models deployed with no reproducibility
|
|
- ❌ Team knowledge trapped in individual notebooks
|
|
|
|
SpecWeave brings discipline:
|
|
- ✅ Every ML feature is an increment (with spec, plan, tasks)
|
|
- ✅ Experiments tracked and documented automatically
|
|
- ✅ Model versions tied to increments
|
|
- ✅ Living docs capture learnings and decisions
|
|
|
|
## How It Works
|
|
|
|
### Phase 1: ML Increment Planning
|
|
|
|
When you request "build a recommendation model", the skill:
|
|
|
|
1. **Creates ML increment structure**:
|
|
```
|
|
.specweave/increments/0042-recommendation-model/
|
|
├── spec.md # ML requirements, success metrics
|
|
├── plan.md # Pipeline architecture
|
|
├── tasks.md # Implementation tasks
|
|
├── tests.md # Evaluation criteria
|
|
├── experiments/ # Experiment tracking
|
|
│ ├── exp-001-baseline/
|
|
│ ├── exp-002-xgboost/
|
|
│ └── exp-003-neural-net/
|
|
├── data/ # Data samples, schemas
|
|
│ ├── schema.yaml
|
|
│ └── sample.csv
|
|
├── models/ # Trained models
|
|
│ ├── model-v1.pkl
|
|
│ └── model-v2.pkl
|
|
└── notebooks/ # Exploratory notebooks
|
|
├── 01-eda.ipynb
|
|
└── 02-feature-engineering.ipynb
|
|
```
|
|
|
|
2. **Generates ML-specific spec** (spec.md):
|
|
```markdown
|
|
## ML Problem Definition
|
|
- Problem type: Recommendation (collaborative filtering)
|
|
- Input: User behavior history
|
|
- Output: Top-N product recommendations
|
|
- Success metrics: Precision@10 > 0.25, Recall@10 > 0.15
|
|
|
|
## Data Requirements
|
|
- Training data: 6 months user interactions
|
|
- Validation: Last month
|
|
- Features: User profile, product attributes, interaction history
|
|
|
|
## Model Requirements
|
|
- Latency: <100ms inference
|
|
- Throughput: 1000 req/sec
|
|
- Accuracy: Better than random baseline by 3x
|
|
- Explainability: Must explain top-3 recommendations
|
|
```
|
|
|
|
3. **Creates ML-specific tasks** (tasks.md):
|
|
```markdown
|
|
- [ ] T-001: Data exploration and quality analysis
|
|
- [ ] T-002: Feature engineering pipeline
|
|
- [ ] T-003: Train baseline model (random/popularity)
|
|
- [ ] T-004: Train candidate models (3 algorithms)
|
|
- [ ] T-005: Hyperparameter tuning (best model)
|
|
- [ ] T-006: Model evaluation (all metrics)
|
|
- [ ] T-007: Model explainability (SHAP/LIME)
|
|
- [ ] T-008: Production deployment preparation
|
|
- [ ] T-009: A/B test plan
|
|
```
|
|
|
|
### Phase 2: Pipeline Execution
|
|
|
|
The skill guides through each task with best practices:
|
|
|
|
#### Task 1: Data Exploration
|
|
```python
|
|
# Generated template with SpecWeave integration
|
|
import pandas as pd
|
|
import mlflow
|
|
from specweave import track_experiment
|
|
|
|
# Auto-logs to .specweave/increments/0042.../experiments/
|
|
with track_experiment("exp-001-eda") as exp:
|
|
df = pd.read_csv("data/interactions.csv")
|
|
|
|
# EDA
|
|
exp.log_param("dataset_size", len(df))
|
|
exp.log_metric("missing_values", df.isnull().sum().sum())
|
|
|
|
# Auto-generates report in increment folder
|
|
exp.save_report("eda-summary.md")
|
|
```
|
|
|
|
#### Task 3: Train Baseline
|
|
```python
|
|
from sklearn.dummy import DummyClassifier
|
|
from specweave import track_model
|
|
|
|
with track_model("baseline-random", increment="0042") as model:
|
|
clf = DummyClassifier(strategy="uniform")
|
|
clf.fit(X_train, y_train)
|
|
|
|
# Automatically logged to increment
|
|
model.log_metrics({
|
|
"accuracy": 0.12,
|
|
"precision@10": 0.08
|
|
})
|
|
model.save_artifact(clf, "baseline.pkl")
|
|
```
|
|
|
|
#### Task 4: Train Candidate Models
|
|
```python
|
|
from xgboost import XGBClassifier
|
|
from specweave import ModelExperiment
|
|
|
|
# Parallel experiments with auto-tracking
|
|
experiments = [
|
|
ModelExperiment("xgboost", XGBClassifier, params_xgb),
|
|
ModelExperiment("lightgbm", LGBMClassifier, params_lgbm),
|
|
ModelExperiment("neural-net", KerasModel, params_nn)
|
|
]
|
|
|
|
results = run_experiments(
|
|
experiments,
|
|
increment="0042",
|
|
save_to="experiments/"
|
|
)
|
|
|
|
# Auto-generates comparison table in increment docs
|
|
```
|
|
|
|
### Phase 3: Increment Completion
|
|
|
|
When `/specweave:done 0042` runs:
|
|
|
|
1. **Validates ML-specific criteria**:
|
|
- ✅ All experiments logged
|
|
- ✅ Best model saved
|
|
- ✅ Evaluation metrics documented
|
|
- ✅ Model explainability artifacts present
|
|
|
|
2. **Generates completion summary**:
|
|
```markdown
|
|
## Recommendation Model - COMPLETE
|
|
|
|
### Experiments Run: 7
|
|
1. exp-001-baseline (random): precision@10=0.08
|
|
2. exp-002-popularity: precision@10=0.18
|
|
3. exp-003-xgboost: precision@10=0.26 ✅ BEST
|
|
4. exp-004-lightgbm: precision@10=0.24
|
|
5. exp-005-neural-net: precision@10=0.22
|
|
...
|
|
|
|
### Best Model
|
|
- Algorithm: XGBoost
|
|
- Version: model-v3.pkl
|
|
- Metrics: precision@10=0.26, recall@10=0.16
|
|
- Training time: 45 min
|
|
- Model size: 12 MB
|
|
|
|
### Deployment Ready
|
|
- ✅ Inference latency: 35ms (target: <100ms)
|
|
- ✅ Explainability: SHAP values computed
|
|
- ✅ A/B test plan documented
|
|
```
|
|
|
|
3. **Syncs living docs** (via `/specweave:sync-docs`):
|
|
- Updates architecture docs with model design
|
|
- Adds ADR for algorithm selection
|
|
- Documents learnings in runbooks
|
|
|
|
## When to Use This Skill
|
|
|
|
Activate this skill when you need to:
|
|
|
|
- **Build ML features end-to-end** - From idea to deployed model
|
|
- **Ensure reproducibility** - Every experiment tracked and documented
|
|
- **Follow ML best practices** - Baseline comparison, proper validation, explainability
|
|
- **Integrate ML with software engineering** - ML as increments, not isolated notebooks
|
|
- **Maintain team knowledge** - Living docs capture why decisions were made
|
|
|
|
## ML Pipeline Stages
|
|
|
|
### 1. Data Stage
|
|
- Data exploration (EDA)
|
|
- Data quality assessment
|
|
- Schema validation
|
|
- Sample data documentation
|
|
|
|
### 2. Feature Stage
|
|
- Feature engineering
|
|
- Feature selection
|
|
- Feature importance analysis
|
|
- Feature store integration (optional)
|
|
|
|
### 3. Training Stage
|
|
- Baseline model (random, rule-based)
|
|
- Candidate models (3+ algorithms)
|
|
- Hyperparameter tuning
|
|
- Cross-validation
|
|
|
|
### 4. Evaluation Stage
|
|
- Comprehensive metrics (accuracy, precision, recall, F1, AUC)
|
|
- Business metrics (latency, throughput)
|
|
- Model comparison (vs baseline, vs previous version)
|
|
- Error analysis
|
|
|
|
### 5. Explainability Stage
|
|
- Feature importance
|
|
- SHAP values
|
|
- LIME explanations
|
|
- Example predictions with rationale
|
|
|
|
### 6. Deployment Stage
|
|
- Model packaging
|
|
- Inference pipeline
|
|
- A/B test plan
|
|
- Monitoring setup
|
|
|
|
## Integration with SpecWeave Workflow
|
|
|
|
### With Experiment Tracking
|
|
```bash
|
|
# Start ML increment
|
|
/specweave:inc "0042-recommendation-model"
|
|
|
|
# Automatically integrates experiment tracking
|
|
# All MLflow/W&B logs saved to increment folder
|
|
```
|
|
|
|
### With Living Docs
|
|
```bash
|
|
# After training best model
|
|
/specweave:sync-docs update
|
|
|
|
# Automatically:
|
|
# - Updates architecture/ml-models.md
|
|
# - Adds ADR for algorithm choice
|
|
# - Documents hyperparameters in runbooks
|
|
```
|
|
|
|
### With GitHub Sync
|
|
```bash
|
|
# Create GitHub issue for model retraining
|
|
/specweave:github:create-issue "Retrain recommendation model with new data"
|
|
|
|
# Linked to increment 0042
|
|
# Issue tracks model performance over time
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
### 1. Always Start with Baseline
|
|
```python
|
|
# Before training complex models, establish baseline
|
|
baseline_results = train_baseline_model(
|
|
strategies=["random", "popularity", "rule-based"]
|
|
)
|
|
# Requirement: New model must beat best baseline by 20%+
|
|
```
|
|
|
|
### 2. Use Cross-Validation
|
|
```python
|
|
# Never trust single train/test split
|
|
cv_scores = cross_val_score(model, X, y, cv=5)
|
|
exp.log_metric("cv_mean", cv_scores.mean())
|
|
exp.log_metric("cv_std", cv_scores.std())
|
|
```
|
|
|
|
### 3. Track Everything
|
|
```python
|
|
# Hyperparameters, metrics, artifacts, environment
|
|
exp.log_params(model.get_params())
|
|
exp.log_metrics({"accuracy": acc, "f1": f1})
|
|
exp.log_artifact("model.pkl")
|
|
exp.log_artifact("requirements.txt") # Reproducibility
|
|
```
|
|
|
|
### 4. Document Failures
|
|
```python
|
|
# Failed experiments are valuable learnings
|
|
with track_experiment("exp-006-failed-lstm") as exp:
|
|
# ... training fails ...
|
|
exp.log_note("FAILED: LSTM overfits badly, needs regularization")
|
|
exp.set_status("failed")
|
|
# This documents why LSTM wasn't chosen
|
|
```
|
|
|
|
### 5. Model Versioning
|
|
```python
|
|
# Tie model versions to increments
|
|
model_version = f"0042-v{iteration}"
|
|
mlflow.register_model(
|
|
f"runs:/{run_id}/model",
|
|
f"recommendation-model-{model_version}"
|
|
)
|
|
```
|
|
|
|
## Examples
|
|
|
|
### Example 1: Classification Pipeline
|
|
```bash
|
|
User: "Build a fraud detection model for transactions"
|
|
|
|
Skill creates increment 0051-fraud-detection with:
|
|
- spec.md: Binary classification, 99% precision target
|
|
- plan.md: Imbalanced data handling, threshold tuning
|
|
- tasks.md: 9 tasks from EDA to deployment
|
|
- experiments/: exp-001-baseline, exp-002-xgboost, etc.
|
|
|
|
Guides through:
|
|
1. EDA → identify class imbalance (0.1% fraud)
|
|
2. Baseline → random/majority (terrible results)
|
|
3. Candidates → XGBoost, LightGBM, Neural Net
|
|
4. Threshold tuning → optimize for precision
|
|
5. SHAP → explain high-risk predictions
|
|
6. Deploy → model + threshold + explainer
|
|
```
|
|
|
|
### Example 2: Regression Pipeline
|
|
```bash
|
|
User: "Predict customer lifetime value"
|
|
|
|
Skill creates increment 0063-ltv-prediction with:
|
|
- spec.md: Regression, RMSE < $50 target
|
|
- plan.md: Time-based validation, feature engineering
|
|
- tasks.md: Customer cohort analysis, feature importance
|
|
|
|
Key difference: Regression-specific evaluation (RMSE, MAE, R²)
|
|
```
|
|
|
|
### Example 3: Time Series Forecasting
|
|
```bash
|
|
User: "Forecast weekly sales for next 12 weeks"
|
|
|
|
Skill creates increment 0072-sales-forecasting with:
|
|
- spec.md: Time series, MAPE < 10% target
|
|
- plan.md: Seasonal decomposition, ARIMA vs Prophet
|
|
- tasks.md: Stationarity tests, residual analysis
|
|
|
|
Key difference: Time series validation (no random split)
|
|
```
|
|
|
|
## Framework Support
|
|
|
|
This skill works with all major ML frameworks:
|
|
|
|
### Scikit-Learn
|
|
```python
|
|
from sklearn.ensemble import RandomForestClassifier
|
|
from specweave import track_sklearn_model
|
|
|
|
model = RandomForestClassifier(n_estimators=100)
|
|
with track_sklearn_model(model, increment="0042") as tracked:
|
|
tracked.fit(X_train, y_train)
|
|
tracked.evaluate(X_test, y_test)
|
|
```
|
|
|
|
### PyTorch
|
|
```python
|
|
import torch
|
|
from specweave import track_pytorch_model
|
|
|
|
model = NeuralNet()
|
|
with track_pytorch_model(model, increment="0042") as tracked:
|
|
for epoch in range(epochs):
|
|
tracked.train_epoch(train_loader)
|
|
tracked.log_metric(f"loss_epoch_{epoch}", loss)
|
|
```
|
|
|
|
### TensorFlow/Keras
|
|
```python
|
|
from tensorflow import keras
|
|
from specweave import KerasCallback
|
|
|
|
model = keras.Sequential([...])
|
|
model.fit(
|
|
X_train, y_train,
|
|
callbacks=[KerasCallback(increment="0042")]
|
|
)
|
|
```
|
|
|
|
### XGBoost/LightGBM
|
|
```python
|
|
import xgboost as xgb
|
|
from specweave import track_boosting_model
|
|
|
|
dtrain = xgb.DMatrix(X_train, label=y_train)
|
|
with track_boosting_model("xgboost", increment="0042") as tracked:
|
|
model = xgb.train(params, dtrain, callbacks=[tracked.callback])
|
|
```
|
|
|
|
## Integration Points
|
|
|
|
### With `experiment-tracker` skill
|
|
- Auto-detects MLflow/W&B in project
|
|
- Configures tracking URI to increment folder
|
|
- Syncs experiment metadata to increment docs
|
|
|
|
### With `model-evaluator` skill
|
|
- Generates comprehensive evaluation reports
|
|
- Compares models across experiments
|
|
- Highlights best model with confidence intervals
|
|
|
|
### With `feature-engineer` skill
|
|
- Generates feature engineering pipeline
|
|
- Documents feature importance
|
|
- Creates feature store schemas
|
|
|
|
### With `ml-engineer` agent
|
|
- Delegates complex ML decisions to specialized agent
|
|
- Reviews model architecture
|
|
- Suggests improvements based on results
|
|
|
|
## Skill Outputs
|
|
|
|
After running `/specweave:do` on an ML increment, you get:
|
|
|
|
```
|
|
.specweave/increments/0042-recommendation-model/
|
|
├── spec.md ✅
|
|
├── plan.md ✅
|
|
├── tasks.md ✅ (all completed)
|
|
├── COMPLETION-SUMMARY.md ✅
|
|
├── experiments/
|
|
│ ├── exp-001-baseline/
|
|
│ │ ├── metrics.json
|
|
│ │ ├── params.json
|
|
│ │ └── logs/
|
|
│ ├── exp-002-xgboost/ ✅ BEST
|
|
│ │ ├── metrics.json
|
|
│ │ ├── params.json
|
|
│ │ ├── model.pkl
|
|
│ │ └── shap_values.pkl
|
|
│ └── comparison.md
|
|
├── models/
|
|
│ ├── model-v3.pkl (best)
|
|
│ └── model-v3.metadata.json
|
|
├── data/
|
|
│ ├── schema.yaml
|
|
│ └── sample.parquet
|
|
└── notebooks/
|
|
├── 01-eda.ipynb
|
|
├── 02-feature-engineering.ipynb
|
|
└── 03-model-analysis.ipynb
|
|
```
|
|
|
|
## Commands
|
|
|
|
This skill integrates with SpecWeave commands:
|
|
|
|
```bash
|
|
# Create ML increment
|
|
/specweave:inc "build recommendation model"
|
|
→ Activates ml-pipeline-orchestrator
|
|
→ Creates ML-specific increment structure
|
|
|
|
# Execute ML tasks
|
|
/specweave:do
|
|
→ Guides through data → train → eval workflow
|
|
→ Auto-tracks experiments
|
|
|
|
# Validate ML increment
|
|
/specweave:validate 0042
|
|
→ Checks: experiments logged, model saved, metrics documented
|
|
→ Validates: model meets success criteria
|
|
|
|
# Complete ML increment
|
|
/specweave:done 0042
|
|
→ Generates ML completion summary
|
|
→ Syncs model metadata to living docs
|
|
```
|
|
|
|
## Tips
|
|
|
|
1. **Start simple** - Always begin with baseline, then iterate
|
|
2. **Track failures** - Document why approaches didn't work
|
|
3. **Version data** - Use DVC or similar for data versioning
|
|
4. **Reproducibility** - Log environment (requirements.txt, conda env)
|
|
5. **Incremental improvement** - Each increment improves on previous model
|
|
6. **Team collaboration** - Living docs make ML decisions visible to all
|
|
|
|
## Advanced: Multi-Increment ML Projects
|
|
|
|
For complex ML systems (e.g., recommendation system with multiple models):
|
|
|
|
```
|
|
0042-recommendation-data-pipeline
|
|
0043-recommendation-candidate-generation
|
|
0044-recommendation-ranking-model
|
|
0045-recommendation-reranking
|
|
0046-recommendation-ab-test
|
|
```
|
|
|
|
Each increment:
|
|
- Has its own spec, plan, tasks
|
|
- Builds on previous increments
|
|
- Documents model interactions
|
|
- Maintains system-level living docs
|