48 lines
1.1 KiB
Markdown
48 lines
1.1 KiB
Markdown
---
|
|
model: claude-opus-4-1
|
|
---
|
|
|
|
# Machine Learning Pipeline
|
|
|
|
Design and implement a complete ML pipeline for: $ARGUMENTS
|
|
|
|
Create a production-ready pipeline including:
|
|
|
|
1. **Data Ingestion**:
|
|
- Multiple data source connectors
|
|
- Schema validation with Pydantic
|
|
- Data versioning strategy
|
|
- Incremental loading capabilities
|
|
|
|
2. **Feature Engineering**:
|
|
- Feature transformation pipeline
|
|
- Feature store integration
|
|
- Statistical validation
|
|
- Handling missing data and outliers
|
|
|
|
3. **Model Training**:
|
|
- Experiment tracking (MLflow/W&B)
|
|
- Hyperparameter optimization
|
|
- Cross-validation strategy
|
|
- Model versioning
|
|
|
|
4. **Model Evaluation**:
|
|
- Comprehensive metrics
|
|
- A/B testing framework
|
|
- Bias detection
|
|
- Performance monitoring
|
|
|
|
5. **Deployment**:
|
|
- Model serving API
|
|
- Batch/stream prediction
|
|
- Model registry
|
|
- Rollback capabilities
|
|
|
|
6. **Monitoring**:
|
|
- Data drift detection
|
|
- Model performance tracking
|
|
- Alert system
|
|
- Retraining triggers
|
|
|
|
Include error handling, logging, and make it cloud-agnostic. Use modern tools like DVC, MLflow, or similar. Ensure reproducibility and scalability.
|