516 lines
10 KiB
Markdown
516 lines
10 KiB
Markdown
---
|
|
name: dependency-management
|
|
description: Automatically applies when managing Python dependencies. Ensures proper use of uv/Poetry, lock files, version constraints, conflict resolution, and dependency security.
|
|
category: python
|
|
---
|
|
|
|
# Dependency Management Patterns
|
|
|
|
When managing Python dependencies, follow these patterns for reproducible, secure environments.
|
|
|
|
**Trigger Keywords**: dependencies, uv, poetry, pip, requirements, lock file, dependency conflict, version pinning, pyproject.toml, pip-compile
|
|
|
|
**Agent Integration**: Used by `backend-architect`, `devops-engineer`, `python-engineer`
|
|
|
|
## ✅ Correct Pattern: Using uv
|
|
|
|
```bash
|
|
# Install uv (modern, fast dependency manager)
|
|
curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
|
|
# Initialize project
|
|
uv init myproject
|
|
cd myproject
|
|
|
|
# Add dependencies
|
|
uv add fastapi pydantic sqlalchemy
|
|
|
|
# Add dev dependencies
|
|
uv add --dev pytest pytest-cov black ruff mypy
|
|
|
|
# Add optional dependencies
|
|
uv add --optional docs sphinx
|
|
|
|
# Install all dependencies
|
|
uv sync
|
|
|
|
# Install with optional dependencies
|
|
uv sync --extra docs
|
|
|
|
# Update dependencies
|
|
uv lock --upgrade
|
|
uv sync
|
|
|
|
# Remove dependency
|
|
uv remove package-name
|
|
```
|
|
|
|
## pyproject.toml with uv
|
|
|
|
```toml
|
|
# pyproject.toml
|
|
[project]
|
|
name = "myproject"
|
|
version = "0.1.0"
|
|
description = "My Python project"
|
|
requires-python = ">=3.11"
|
|
|
|
dependencies = [
|
|
"fastapi>=0.109.0,<1.0.0",
|
|
"pydantic>=2.5.0,<3.0.0",
|
|
"sqlalchemy>=2.0.0,<3.0.0",
|
|
"httpx>=0.26.0,<1.0.0",
|
|
]
|
|
|
|
[project.optional-dependencies]
|
|
dev = [
|
|
"pytest>=7.4.0",
|
|
"pytest-cov>=4.1.0",
|
|
"pytest-asyncio>=0.23.0",
|
|
"black>=24.0.0",
|
|
"ruff>=0.1.0",
|
|
"mypy>=1.8.0",
|
|
]
|
|
docs = [
|
|
"sphinx>=7.2.0",
|
|
"sphinx-rtd-theme>=2.0.0",
|
|
]
|
|
|
|
[tool.uv]
|
|
dev-dependencies = [
|
|
"pytest>=7.4.0",
|
|
"black>=24.0.0",
|
|
]
|
|
|
|
# Lock file is automatically managed
|
|
# uv.lock contains exact versions
|
|
```
|
|
|
|
## Using Poetry
|
|
|
|
```bash
|
|
# Install Poetry
|
|
curl -sSL https://install.python-poetry.org | python3 -
|
|
|
|
# Initialize project
|
|
poetry new myproject
|
|
cd myproject
|
|
|
|
# Add dependencies
|
|
poetry add fastapi pydantic sqlalchemy
|
|
|
|
# Add dev dependencies
|
|
poetry add --group dev pytest pytest-cov black ruff mypy
|
|
|
|
# Add optional dependencies
|
|
poetry add --optional sphinx
|
|
poetry add --optional sphinx-rtd-theme
|
|
|
|
# Install dependencies
|
|
poetry install
|
|
|
|
# Install with optional dependencies
|
|
poetry install --extras docs
|
|
|
|
# Update dependencies
|
|
poetry update
|
|
|
|
# Lock dependencies without installing
|
|
poetry lock --no-update
|
|
|
|
# Remove dependency
|
|
poetry remove package-name
|
|
|
|
# Show dependency tree
|
|
poetry show --tree
|
|
```
|
|
|
|
## pyproject.toml with Poetry
|
|
|
|
```toml
|
|
# pyproject.toml
|
|
[tool.poetry]
|
|
name = "myproject"
|
|
version = "0.1.0"
|
|
description = "My Python project"
|
|
authors = ["Your Name <you@example.com>"]
|
|
readme = "README.md"
|
|
packages = [{include = "myproject", from = "src"}]
|
|
|
|
[tool.poetry.dependencies]
|
|
python = "^3.11"
|
|
fastapi = "^0.109.0"
|
|
pydantic = "^2.5.0"
|
|
sqlalchemy = "^2.0.0"
|
|
httpx = "^0.26.0"
|
|
|
|
# Optional dependencies
|
|
sphinx = {version = "^7.2.0", optional = true}
|
|
sphinx-rtd-theme = {version = "^2.0.0", optional = true}
|
|
|
|
[tool.poetry.group.dev.dependencies]
|
|
pytest = "^7.4.0"
|
|
pytest-cov = "^4.1.0"
|
|
pytest-asyncio = "^0.23.0"
|
|
black = "^24.0.0"
|
|
ruff = "^0.1.0"
|
|
mypy = "^1.8.0"
|
|
|
|
[tool.poetry.extras]
|
|
docs = ["sphinx", "sphinx-rtd-theme"]
|
|
|
|
[build-system]
|
|
requires = ["poetry-core"]
|
|
build-backend = "poetry.core.masonry.api"
|
|
```
|
|
|
|
## Version Constraints
|
|
|
|
```toml
|
|
# Caret requirements (^) - recommended for libraries
|
|
# ^1.2.3 means >=1.2.3,<2.0.0
|
|
fastapi = "^0.109.0"
|
|
|
|
# Tilde requirements (~) - for bug fix updates
|
|
# ~1.2.3 means >=1.2.3,<1.3.0
|
|
pytest = "~7.4.0"
|
|
|
|
# Exact version (=)
|
|
black = "24.1.0"
|
|
|
|
# Greater than or equal
|
|
httpx = ">=0.26.0"
|
|
|
|
# Compatible release (~=)
|
|
# ~=1.2.3 is equivalent to >=1.2.3,<1.3.0
|
|
sqlalchemy = "~=2.0.0"
|
|
|
|
# Multiple constraints
|
|
pydantic = ">=2.5.0,<3.0.0"
|
|
|
|
# Wildcard
|
|
requests = "2.*"
|
|
```
|
|
|
|
## Lock Files
|
|
|
|
```python
|
|
# uv.lock (generated by uv)
|
|
# Contains exact versions of all dependencies
|
|
# Commit to version control for reproducibility
|
|
|
|
# poetry.lock (generated by poetry)
|
|
# Contains exact versions and hashes
|
|
# Commit to version control
|
|
|
|
# Benefits of lock files:
|
|
# 1. Reproducible builds
|
|
# 2. Security (verify hashes)
|
|
# 3. Faster installs
|
|
# 4. Conflict detection
|
|
```
|
|
|
|
## Dependency Conflict Resolution
|
|
|
|
```python
|
|
# Check for conflicts
|
|
# uv
|
|
uv lock
|
|
|
|
# Poetry
|
|
poetry lock
|
|
|
|
# If conflicts occur:
|
|
# 1. Check dependency requirements
|
|
poetry show package-name
|
|
|
|
# 2. Update conflicting package
|
|
poetry update package-name
|
|
|
|
# 3. Use version ranges that overlap
|
|
[tool.poetry.dependencies]
|
|
fastapi = "^0.109.0" # Requires pydantic ^2.0
|
|
pydantic = "^2.5.0" # Compatible!
|
|
|
|
# 4. Override dependencies if needed (Poetry)
|
|
[tool.poetry.overrides]
|
|
"problematic-package" = "1.2.3"
|
|
```
|
|
|
|
## Security Scanning
|
|
|
|
```bash
|
|
# Check for security vulnerabilities with uv
|
|
uv pip list --outdated
|
|
|
|
# Use pip-audit for security scanning
|
|
pip install pip-audit
|
|
pip-audit
|
|
|
|
# Use safety
|
|
pip install safety
|
|
safety check
|
|
|
|
# Use Poetry's built-in audit (if available)
|
|
poetry audit
|
|
|
|
# GitHub Dependabot
|
|
# Automatically creates PRs for security updates
|
|
# Enable in .github/dependabot.yml
|
|
```
|
|
|
|
## Dependabot Configuration
|
|
|
|
```yaml
|
|
# .github/dependabot.yml
|
|
version: 2
|
|
updates:
|
|
- package-ecosystem: "pip"
|
|
directory: "/"
|
|
schedule:
|
|
interval: "weekly"
|
|
open-pull-requests-limit: 10
|
|
reviewers:
|
|
- "username"
|
|
labels:
|
|
- "dependencies"
|
|
- "automated"
|
|
|
|
# Group updates
|
|
groups:
|
|
development-dependencies:
|
|
patterns:
|
|
- "pytest*"
|
|
- "black"
|
|
- "ruff"
|
|
- "mypy"
|
|
|
|
production-dependencies:
|
|
patterns:
|
|
- "fastapi"
|
|
- "pydantic"
|
|
- "sqlalchemy"
|
|
|
|
# Version updates
|
|
versioning-strategy: increase
|
|
```
|
|
|
|
## CI/CD Integration
|
|
|
|
```yaml
|
|
# .github/workflows/dependencies.yml
|
|
name: Dependency Check
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
schedule:
|
|
- cron: "0 0 * * 0" # Weekly
|
|
|
|
jobs:
|
|
check-dependencies:
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@v4
|
|
with:
|
|
python-version: "3.11"
|
|
|
|
- name: Install uv
|
|
run: curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
|
|
- name: Install dependencies
|
|
run: uv sync
|
|
|
|
- name: Check for outdated packages
|
|
run: uv pip list --outdated
|
|
|
|
- name: Security audit
|
|
run: |
|
|
pip install pip-audit
|
|
pip-audit
|
|
|
|
- name: Check lock file
|
|
run: |
|
|
uv lock
|
|
git diff --exit-code uv.lock
|
|
```
|
|
|
|
## Virtual Environment Management
|
|
|
|
```bash
|
|
# uv automatically manages virtual environments
|
|
uv venv # Create virtual environment
|
|
source .venv/bin/activate # Activate
|
|
|
|
# Poetry
|
|
poetry shell # Activate Poetry environment
|
|
poetry env info # Show environment info
|
|
poetry env list # List environments
|
|
poetry env remove python3.11 # Remove environment
|
|
|
|
# Manual venv
|
|
python -m venv .venv
|
|
source .venv/bin/activate # Linux/Mac
|
|
.venv\Scripts\activate # Windows
|
|
```
|
|
|
|
## Exporting Dependencies
|
|
|
|
```bash
|
|
# Export to requirements.txt format
|
|
|
|
# uv
|
|
uv pip compile pyproject.toml -o requirements.txt
|
|
|
|
# Poetry
|
|
poetry export -f requirements.txt --output requirements.txt
|
|
poetry export -f requirements.txt --with dev --output requirements-dev.txt
|
|
poetry export -f requirements.txt --extras docs --output requirements-docs.txt
|
|
|
|
# For Docker
|
|
poetry export -f requirements.txt --without-hashes --output requirements.txt
|
|
```
|
|
|
|
## Docker Integration
|
|
|
|
```dockerfile
|
|
# Dockerfile with uv
|
|
FROM python:3.11-slim
|
|
|
|
WORKDIR /app
|
|
|
|
# Install uv
|
|
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
ENV PATH="/root/.cargo/bin:$PATH"
|
|
|
|
# Copy dependency files
|
|
COPY pyproject.toml uv.lock ./
|
|
|
|
# Install dependencies
|
|
RUN uv sync --frozen --no-dev
|
|
|
|
# Copy application
|
|
COPY . .
|
|
|
|
CMD ["python", "-m", "myproject"]
|
|
|
|
|
|
# Dockerfile with Poetry
|
|
FROM python:3.11-slim
|
|
|
|
WORKDIR /app
|
|
|
|
# Install Poetry
|
|
RUN pip install poetry==1.7.1
|
|
|
|
# Configure Poetry
|
|
ENV POETRY_NO_INTERACTION=1 \
|
|
POETRY_VIRTUALENVS_IN_PROJECT=1 \
|
|
POETRY_VIRTUALENVS_CREATE=1 \
|
|
POETRY_CACHE_DIR=/tmp/poetry_cache
|
|
|
|
# Copy dependency files
|
|
COPY pyproject.toml poetry.lock ./
|
|
|
|
# Install dependencies
|
|
RUN poetry install --without dev --no-root && rm -rf $POETRY_CACHE_DIR
|
|
|
|
# Copy application
|
|
COPY . .
|
|
|
|
# Install project
|
|
RUN poetry install --without dev
|
|
|
|
CMD ["poetry", "run", "python", "-m", "myproject"]
|
|
```
|
|
|
|
## ❌ Anti-Patterns
|
|
|
|
```python
|
|
# ❌ No lock file
|
|
# Just pyproject.toml, no uv.lock or poetry.lock
|
|
# Non-reproducible builds!
|
|
|
|
# ✅ Better: Commit lock file
|
|
git add uv.lock # or poetry.lock
|
|
|
|
|
|
# ❌ Unpinned versions in production
|
|
dependencies = ["fastapi"] # Any version!
|
|
|
|
# ✅ Better: Use version constraints
|
|
dependencies = ["fastapi>=0.109.0,<1.0.0"]
|
|
|
|
|
|
# ❌ Using pip freeze without pip-tools
|
|
pip freeze > requirements.txt # Includes transitive deps!
|
|
|
|
# ✅ Better: Use uv or poetry for dependency management
|
|
|
|
|
|
# ❌ Committing virtual environment
|
|
git add .venv/ # Huge, not portable!
|
|
|
|
# ✅ Better: Add to .gitignore
|
|
.venv/
|
|
venv/
|
|
*.pyc
|
|
|
|
|
|
# ❌ Not separating dev dependencies
|
|
dependencies = ["fastapi", "pytest", "black"] # All mixed!
|
|
|
|
# ✅ Better: Use dev dependencies
|
|
[project]
|
|
dependencies = ["fastapi"]
|
|
|
|
[project.optional-dependencies]
|
|
dev = ["pytest", "black"]
|
|
|
|
|
|
# ❌ Ignoring security warnings
|
|
# pip install shows vulnerabilities but not addressed
|
|
|
|
# ✅ Better: Regular security audits
|
|
pip-audit
|
|
poetry audit
|
|
```
|
|
|
|
## Best Practices Checklist
|
|
|
|
- ✅ Use uv or Poetry for dependency management
|
|
- ✅ Commit lock files (uv.lock, poetry.lock)
|
|
- ✅ Use version constraints, not exact pins
|
|
- ✅ Separate dev dependencies from production
|
|
- ✅ Run security audits regularly
|
|
- ✅ Use Dependabot for automatic updates
|
|
- ✅ Don't commit virtual environments
|
|
- ✅ Export requirements.txt for Docker
|
|
- ✅ Pin Python version requirement
|
|
- ✅ Document dependency installation
|
|
- ✅ Test dependency updates before merging
|
|
- ✅ Use dependency groups for organization
|
|
|
|
## Auto-Apply
|
|
|
|
When managing dependencies:
|
|
1. Use uv or Poetry (not plain pip)
|
|
2. Define dependencies in pyproject.toml
|
|
3. Generate and commit lock file
|
|
4. Use version constraints (^, ~, >=)
|
|
5. Separate dev/docs dependencies
|
|
6. Run security audits (pip-audit)
|
|
7. Set up Dependabot
|
|
8. Test updates in CI before merging
|
|
|
|
## Related Skills
|
|
|
|
- `python-packaging` - For package configuration
|
|
- `git-workflow-standards` - For version control
|
|
- `monitoring-alerting` - For dependency monitoring
|