Initial commit
This commit is contained in:
Zhongwei Li
540
skills/uv-debug/references/python-build-cache.md
Normal file
540
skills/uv-debug/references/python-build-cache.md
Normal file
@@ -0,0 +1,540 @@
|
||||
# Python Build Cache Deep Dive
|
||||
|
||||
## Overview
|
||||
|
||||
This document explains in detail how Python's packaging system caches builds, why this causes "code not updating" issues, and the technical mechanisms behind different installation modes.
|
||||
|
||||
## The Build Process
|
||||
|
||||
### Standard Build (Non-Editable)
|
||||
|
||||
When running `uv tool install .` or `pip install .`:
|
||||
|
||||
```
|
||||
Source Code → setup.py/pyproject.toml → Build Backend → Wheel → Installation
|
||||
```
|
||||
|
||||
**Step-by-step breakdown:**
|
||||
|
||||
1. **Parse metadata:**
|
||||
- Read `pyproject.toml` or `setup.py`
|
||||
- Extract: name, version, dependencies, entry points
|
||||
- Determine which files to include
|
||||
|
||||
2. **Collect source files:**
|
||||
- Find all `.py` files in package
|
||||
- Apply MANIFEST.in rules (if exists)
|
||||
- Apply `pyproject.toml` includes/excludes
|
||||
|
||||
3. **Build wheel (.whl):**
|
||||
- Compile C extensions (if any)
|
||||
- Copy Python files
|
||||
- Generate metadata files
|
||||
- Create ZIP archive named `<package>-<version>-py3-none-any.whl`
|
||||
- Store in `dist/` directory
|
||||
|
||||
4. **Generate metadata:**
|
||||
- Create `<package>.egg-info/` directory
|
||||
- Write `SOURCES.txt` (list of source files used)
|
||||
- Write `RECORD` (list of files to install)
|
||||
- Write `entry_points.txt` (console scripts)
|
||||
- Write `requires.txt` (dependencies)
|
||||
|
||||
5. **Install wheel:**
|
||||
- Extract wheel to installation directory
|
||||
- Create entry point executables in `bin/`
|
||||
- Update Python's package registry
|
||||
|
||||
**Key files created:**
|
||||
|
||||
```
|
||||
build/
|
||||
├── lib/
|
||||
│ └── mypackage/
|
||||
│ └── (compiled files)
|
||||
└── bdist.*/
|
||||
└── (platform-specific builds)
|
||||
|
||||
dist/
|
||||
└── mypackage-1.0.0-py3-none-any.whl ← The cached snapshot
|
||||
|
||||
mypackage.egg-info/
|
||||
├── SOURCES.txt ← Source files at build time
|
||||
├── RECORD ← Files to install
|
||||
├── entry_points.txt ← Console scripts
|
||||
└── requires.txt ← Dependencies
|
||||
```
|
||||
|
||||
### Why It's a Snapshot
|
||||
|
||||
**The wheel is a frozen moment in time:**
|
||||
|
||||
```python
|
||||
# At build time (t=0):
|
||||
mypackage/
|
||||
├── __init__.py ← Included in wheel
|
||||
├── cli.py ← Included in wheel
|
||||
└── commands/
|
||||
├── send.py ← Included in wheel
|
||||
└── read.py ← Included in wheel
|
||||
|
||||
# After adding new file (t=1):
|
||||
mypackage/
|
||||
├── __init__.py
|
||||
├── cli.py
|
||||
└── commands/
|
||||
├── send.py
|
||||
├── read.py
|
||||
└── workflows.py ← NOT in wheel! Built at t=0
|
||||
```
|
||||
|
||||
**The wheel still contains only files from t=0:**
|
||||
```bash
|
||||
$ unzip -l dist/mypackage-1.0.0-py3-none-any.whl
|
||||
mypackage/__init__.py
|
||||
mypackage/cli.py
|
||||
mypackage/commands/send.py
|
||||
mypackage/commands/read.py
|
||||
# workflows.py is MISSING
|
||||
```
|
||||
|
||||
**Even `--force` reinstall uses this stale wheel:**
|
||||
```bash
|
||||
uv tool install --force .
|
||||
# Still installs the old wheel from dist/!
|
||||
```
|
||||
|
||||
## Installation Locations
|
||||
|
||||
### UV Tool Install
|
||||
|
||||
```
|
||||
~/.local/share/uv/tools/<package>/
|
||||
├── bin/
|
||||
│ └── <command> ← Executable entry point
|
||||
├── lib/
|
||||
│ └── python3.x/
|
||||
│ └── site-packages/
|
||||
│ ├── <package>/ ← Package code
|
||||
│ └── <package>-<version>.dist-info/
|
||||
│ ├── RECORD
|
||||
│ ├── entry_points.txt
|
||||
│ └── METADATA
|
||||
```
|
||||
|
||||
### Editable Install
|
||||
|
||||
**Instead of copying files, creates pointer:**
|
||||
|
||||
```
|
||||
~/.local/share/uv/tools/<package>/
|
||||
├── bin/
|
||||
│ └── <command>
|
||||
└── lib/
|
||||
└── python3.x/
|
||||
└── site-packages/
|
||||
├── __editables__/
|
||||
│ └── <package>.pth ← Points to source directory
|
||||
└── <package>-<version>.dist-info/
|
||||
```
|
||||
|
||||
**The `.pth` file contains:**
|
||||
```
|
||||
/absolute/path/to/source/directory
|
||||
```
|
||||
|
||||
**Python's import system:**
|
||||
1. Reads `.pth` file
|
||||
2. Adds path to `sys.path`
|
||||
3. Imports directly from source directory
|
||||
4. New files appear immediately (no reinstall)
|
||||
|
||||
### Local Environment (uv run)
|
||||
|
||||
**No global installation at all:**
|
||||
|
||||
```
|
||||
project/
|
||||
├── .venv/
|
||||
│ ├── bin/
|
||||
│ │ └── python ← Local Python interpreter
|
||||
│ └── lib/
|
||||
│ └── python3.x/
|
||||
│ └── site-packages/ ← Dependencies only
|
||||
├── mypackage/ ← Source code (NOT installed)
|
||||
└── pyproject.toml
|
||||
```
|
||||
|
||||
**How `uv run` works:**
|
||||
|
||||
```bash
|
||||
uv run mycommand
|
||||
```
|
||||
|
||||
Internally executes:
|
||||
```bash
|
||||
PYTHONPATH=/path/to/project:$PYTHONPATH \
|
||||
.venv/bin/python -m mypackage.cli
|
||||
```
|
||||
|
||||
**Import resolution:**
|
||||
1. Check `PYTHONPATH` first (finds `mypackage/` in project root)
|
||||
2. Import directly from source
|
||||
3. No build, no cache, always latest
|
||||
|
||||
## Why --force Doesn't Help
|
||||
|
||||
**Common misconception:**
|
||||
```bash
|
||||
uv tool install --force . # "Force should rebuild, right?"
|
||||
```
|
||||
|
||||
**What `--force` actually does:**
|
||||
- Uninstalls existing package
|
||||
- Reinstalls from available sources
|
||||
- **Does NOT** delete `build/` or `dist/`
|
||||
|
||||
**The problem:**
|
||||
|
||||
```bash
|
||||
# 1. First install (builds wheel)
|
||||
uv tool install .
|
||||
# Creates: dist/mypackage-1.0.0-py3-none-any.whl
|
||||
|
||||
# 2. Add new file
|
||||
touch mypackage/commands/workflows.py
|
||||
|
||||
# 3. Force reinstall
|
||||
uv tool install --force .
|
||||
# Finds existing wheel in dist/
|
||||
# Reinstalls OLD wheel (still no workflows.py!)
|
||||
```
|
||||
|
||||
**Why it finds the old wheel:**
|
||||
|
||||
UV's build process:
|
||||
1. Check if wheel exists in `dist/` matching current version
|
||||
2. If yes, use that wheel (fast!)
|
||||
3. If no, build new wheel
|
||||
|
||||
**The version in `pyproject.toml` hasn't changed**, so UV reuses the cached wheel.
|
||||
|
||||
## How to Force Fresh Build
|
||||
|
||||
**Option 1: Clean first**
|
||||
```bash
|
||||
rm -rf build/ dist/ *.egg-info
|
||||
uv tool install --force .
|
||||
```
|
||||
|
||||
**Option 2: Bump version**
|
||||
```toml
|
||||
[project]
|
||||
version = "1.0.1" # Changed from 1.0.0
|
||||
```
|
||||
```bash
|
||||
uv tool install --force .
|
||||
# No matching wheel in dist/, builds fresh
|
||||
```
|
||||
|
||||
**Option 3: Build explicitly**
|
||||
```bash
|
||||
uv build --force
|
||||
uv tool install --force .
|
||||
```
|
||||
|
||||
## Metadata Files Deep Dive
|
||||
|
||||
### RECORD File
|
||||
|
||||
Lists every file installed, with checksums:
|
||||
|
||||
```
|
||||
mypackage/__init__.py,sha256=abc123...,1234
|
||||
mypackage/cli.py,sha256=def456...,5678
|
||||
mypackage/commands/send.py,sha256=ghi789...,9012
|
||||
mypackage/commands/read.py,sha256=jkl012...,3456
|
||||
```
|
||||
|
||||
**New files aren't in RECORD = won't be installed**
|
||||
|
||||
### entry_points.txt
|
||||
|
||||
Defines console scripts:
|
||||
|
||||
```
|
||||
[console_scripts]
|
||||
gmail = mypackage.cli:main
|
||||
```
|
||||
|
||||
**This is read at install time to create executables in `bin/`**
|
||||
|
||||
Changes to entry points require rebuild.
|
||||
|
||||
### SOURCES.txt
|
||||
|
||||
Lists source files used during build:
|
||||
|
||||
```
|
||||
mypackage/__init__.py
|
||||
mypackage/cli.py
|
||||
mypackage/commands/send.py
|
||||
mypackage/commands/read.py
|
||||
setup.py
|
||||
pyproject.toml
|
||||
```
|
||||
|
||||
**Diagnostic use:** If a file is missing here, it wasn't included in the build.
|
||||
|
||||
## Debugging Cache Issues
|
||||
|
||||
### Check if wheel is stale
|
||||
|
||||
```bash
|
||||
# 1. List files in wheel
|
||||
unzip -l dist/*.whl | grep -i workflows
|
||||
# If empty, file not in wheel
|
||||
|
||||
# 2. Check SOURCES.txt
|
||||
cat *.egg-info/SOURCES.txt | grep workflows
|
||||
# If empty, file wasn't included in build
|
||||
|
||||
# 3. Check build timestamp
|
||||
ls -la dist/*.whl
|
||||
# If older than source files, rebuild needed
|
||||
```
|
||||
|
||||
### Compare local vs installed
|
||||
|
||||
```bash
|
||||
# Source files
|
||||
find mypackage -name "*.py" | sort
|
||||
|
||||
# Installed files
|
||||
find ~/.local/share/uv/tools/mypackage -name "*.py" | sort
|
||||
|
||||
# Diff them
|
||||
diff <(find mypackage -name "*.py" | sort) \
|
||||
<(find ~/.local/share/uv/tools/mypackage -name "*.py" | sed 's|.*mypackage|mypackage|' | sort)
|
||||
```
|
||||
|
||||
### Verify import source
|
||||
|
||||
```python
|
||||
import mypackage
|
||||
print(mypackage.__file__)
|
||||
# Should point to installed location, not source
|
||||
```
|
||||
|
||||
## Performance Trade-offs
|
||||
|
||||
### Why Caching Exists
|
||||
|
||||
**Without caching (rebuild every time):**
|
||||
- Slow: Parsing, file collection, wheel building (seconds to minutes)
|
||||
- Wasteful: Rebuilding unchanged code
|
||||
- Inconsistent: Different builds might produce different results
|
||||
|
||||
**With caching (reuse wheel):**
|
||||
- Fast: Just extract and copy (milliseconds)
|
||||
- Efficient: Build once, install many
|
||||
- Reproducible: Same wheel = same result
|
||||
|
||||
**The trade-off:**
|
||||
- Development: Need to rebuild after changes (overhead)
|
||||
- Production: Install is fast and predictable (benefit)
|
||||
|
||||
## Best Practices by Use Case
|
||||
|
||||
### Active Development
|
||||
```bash
|
||||
# Option 1: No install (recommended)
|
||||
uv sync
|
||||
uv run mycommand
|
||||
|
||||
# Option 2: Editable install
|
||||
uv tool install --editable .
|
||||
|
||||
# Option 3: Makefile automation
|
||||
make install # (with clean dependency)
|
||||
```
|
||||
|
||||
### Testing Production Build
|
||||
```bash
|
||||
# Clean environment
|
||||
rm -rf build/ dist/ *.egg-info
|
||||
|
||||
# Fresh build
|
||||
uv tool install --force .
|
||||
|
||||
# Test
|
||||
mycommand --help
|
||||
```
|
||||
|
||||
### Distribution
|
||||
```bash
|
||||
# Build wheel
|
||||
uv build
|
||||
|
||||
# Upload to PyPI
|
||||
uv publish
|
||||
|
||||
# Users install
|
||||
uv tool install mypackage
|
||||
# (Downloads from PyPI, no source needed)
|
||||
```
|
||||
|
||||
## Common Scenarios
|
||||
|
||||
### Scenario 1: Added New Subcommand
|
||||
|
||||
**Problem:**
|
||||
```bash
|
||||
# Added mypackage/commands/workflows.py
|
||||
uv tool install --force .
|
||||
mycommand workflows # Command not found
|
||||
```
|
||||
|
||||
**Why:**
|
||||
- Entry point might need updating in `pyproject.toml`
|
||||
- Or file just not in cached wheel
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# 1. Check entry points
|
||||
grep -A 5 "\[project.scripts\]" pyproject.toml
|
||||
|
||||
# 2. Clean and rebuild
|
||||
rm -rf build/ dist/ *.egg-info
|
||||
uv tool install --force .
|
||||
```
|
||||
|
||||
### Scenario 2: Updated Dependency
|
||||
|
||||
**Problem:**
|
||||
```bash
|
||||
# Updated pyproject.toml dependencies
|
||||
uv tool install --force .
|
||||
# Still using old dependency version
|
||||
```
|
||||
|
||||
**Why:**
|
||||
- Wheel metadata includes dependency list
|
||||
- Cached wheel has old requirements
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
rm -rf build/ dist/ *.egg-info
|
||||
uv tool install --force --reinstall-package <dependency> .
|
||||
```
|
||||
|
||||
### Scenario 3: Moved Files
|
||||
|
||||
**Problem:**
|
||||
```bash
|
||||
# Moved mypackage/utils.py → mypackage/helpers/utils.py
|
||||
uv tool install --force .
|
||||
# Import still finds old location
|
||||
```
|
||||
|
||||
**Why:**
|
||||
- Old wheel still has `mypackage/utils.py`
|
||||
- New file at `mypackage/helpers/utils.py` not in wheel
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
rm -rf build/ dist/ *.egg-info
|
||||
uv tool install --force .
|
||||
```
|
||||
|
||||
## Wheel Internals
|
||||
|
||||
### Wheel Format
|
||||
|
||||
A wheel is a ZIP archive with structure:
|
||||
|
||||
```
|
||||
mypackage-1.0.0-py3-none-any.whl
|
||||
├── mypackage/ ← Package code
|
||||
│ ├── __init__.py
|
||||
│ └── cli.py
|
||||
└── mypackage-1.0.0.dist-info/ ← Metadata
|
||||
├── WHEEL ← Wheel version, tags
|
||||
├── METADATA ← Package info (name, version, deps)
|
||||
├── RECORD ← File checksums
|
||||
└── entry_points.txt ← Console scripts
|
||||
```
|
||||
|
||||
### Wheel Naming Convention
|
||||
|
||||
```
|
||||
{distribution}-{version}-{python}-{abi}-{platform}.whl
|
||||
```
|
||||
|
||||
Example: `mypackage-1.0.0-py3-none-any.whl`
|
||||
- `mypackage` - Distribution name
|
||||
- `1.0.0` - Version
|
||||
- `py3` - Python 3 compatible
|
||||
- `none` - No ABI requirement
|
||||
- `any` - Any platform
|
||||
|
||||
**Pure Python wheels use `py3-none-any`**
|
||||
**Compiled extensions use specific tags (e.g., `cp311-cp311-macosx_11_0_arm64`)**
|
||||
|
||||
## Comparison: Other Package Managers
|
||||
|
||||
### npm (Node.js)
|
||||
|
||||
```bash
|
||||
npm install # Installs to node_modules/
|
||||
npm link # Similar to editable install
|
||||
```
|
||||
|
||||
**npm doesn't cache builds** - packages are just copied
|
||||
**No build cache issues** - source and installed are always in sync
|
||||
|
||||
### cargo (Rust)
|
||||
|
||||
```bash
|
||||
cargo build # Builds to target/
|
||||
cargo install # Installs from crates.io
|
||||
```
|
||||
|
||||
**cargo caches compiled artifacts** but rebuilds on source changes
|
||||
**Incremental compilation** - only rebuilds changed files
|
||||
|
||||
### pip/uv (Python)
|
||||
|
||||
```bash
|
||||
pip install . # Builds wheel, installs
|
||||
pip install -e . # Editable install
|
||||
```
|
||||
|
||||
**Caches wheels** - can cause stale installs
|
||||
**Requires explicit rebuild** for changes to appear
|
||||
|
||||
## Summary
|
||||
|
||||
**Key Takeaways:**
|
||||
|
||||
1. **Wheels are snapshots** - Frozen at build time, don't auto-update
|
||||
2. **Build artifacts cache** - `build/`, `dist/`, `*.egg-info` persist
|
||||
3. **`--force` doesn't clean** - Reinstalls but may reuse cached wheel
|
||||
4. **Editable mode works differently** - Uses symlinks, not copies
|
||||
5. **`uv run` bypasses install** - Runs directly from source
|
||||
6. **Clean before rebuild** - Only way to guarantee fresh build
|
||||
|
||||
**Mental Model:**
|
||||
|
||||
```
|
||||
Source Code ──build──> Wheel (snapshot) ──install──> Installation
|
||||
↓ ↑ ↓
|
||||
Change Cached! Stale!
|
||||
↓ ↑ ↓
|
||||
Must clean cache to force rebuild
|
||||
```
|
||||
|
||||
**When in doubt:**
|
||||
```bash
|
||||
rm -rf build/ dist/ *.egg-info && uv tool install --force .
|
||||
```
|
||||
Reference in New Issue
Block a user