516 lines
12 KiB
Markdown
516 lines
12 KiB
Markdown
# Zarr Python Quick Reference
|
||
|
||
This reference provides a concise overview of commonly used Zarr functions, parameters, and patterns for quick lookup during development.
|
||
|
||
## Array Creation Functions
|
||
|
||
### `zarr.zeros()` / `zarr.ones()` / `zarr.empty()`
|
||
```python
|
||
zarr.zeros(shape, chunks=None, dtype='f8', store=None, compressor='default',
|
||
fill_value=0, order='C', filters=None)
|
||
```
|
||
Create arrays filled with zeros, ones, or empty (uninitialized) values.
|
||
|
||
**Key parameters:**
|
||
- `shape`: Tuple defining array dimensions (e.g., `(1000, 1000)`)
|
||
- `chunks`: Tuple defining chunk dimensions (e.g., `(100, 100)`), or `None` for no chunking
|
||
- `dtype`: NumPy data type (e.g., `'f4'`, `'i8'`, `'bool'`)
|
||
- `store`: Storage location (string path, Store object, or `None` for memory)
|
||
- `compressor`: Compression codec or `None` for no compression
|
||
|
||
### `zarr.create_array()` / `zarr.create()`
|
||
```python
|
||
zarr.create_array(store, shape, chunks, dtype='f8', compressor='default',
|
||
fill_value=0, order='C', filters=None, overwrite=False)
|
||
```
|
||
Create a new array with explicit control over all parameters.
|
||
|
||
### `zarr.array()`
|
||
```python
|
||
zarr.array(data, chunks=None, dtype=None, compressor='default', store=None)
|
||
```
|
||
Create array from existing data (NumPy array, list, etc.).
|
||
|
||
**Example:**
|
||
```python
|
||
import numpy as np
|
||
data = np.random.random((1000, 1000))
|
||
z = zarr.array(data, chunks=(100, 100), store='data.zarr')
|
||
```
|
||
|
||
### `zarr.open_array()` / `zarr.open()`
|
||
```python
|
||
zarr.open_array(store, mode='a', shape=None, chunks=None, dtype=None,
|
||
compressor='default', fill_value=0)
|
||
```
|
||
Open existing array or create new one.
|
||
|
||
**Mode options:**
|
||
- `'r'`: Read-only
|
||
- `'r+'`: Read-write, file must exist
|
||
- `'a'`: Read-write, create if doesn't exist (default)
|
||
- `'w'`: Create new, overwrite if exists
|
||
- `'w-'`: Create new, fail if exists
|
||
|
||
## Storage Classes
|
||
|
||
### LocalStore (Default)
|
||
```python
|
||
from zarr.storage import LocalStore
|
||
|
||
store = LocalStore('path/to/data.zarr')
|
||
z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
|
||
```
|
||
|
||
### MemoryStore
|
||
```python
|
||
from zarr.storage import MemoryStore
|
||
|
||
store = MemoryStore() # Data only in memory
|
||
z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
|
||
```
|
||
|
||
### ZipStore
|
||
```python
|
||
from zarr.storage import ZipStore
|
||
|
||
# Write
|
||
store = ZipStore('data.zip', mode='w')
|
||
z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
|
||
z[:] = data
|
||
store.close() # MUST close
|
||
|
||
# Read
|
||
store = ZipStore('data.zip', mode='r')
|
||
z = zarr.open_array(store=store)
|
||
data = z[:]
|
||
store.close()
|
||
```
|
||
|
||
### Cloud Storage (S3/GCS)
|
||
```python
|
||
# S3
|
||
import s3fs
|
||
s3 = s3fs.S3FileSystem(anon=False)
|
||
store = s3fs.S3Map(root='bucket/path/data.zarr', s3=s3)
|
||
|
||
# GCS
|
||
import gcsfs
|
||
gcs = gcsfs.GCSFileSystem(project='my-project')
|
||
store = gcsfs.GCSMap(root='bucket/path/data.zarr', gcs=gcs)
|
||
```
|
||
|
||
## Compression Codecs
|
||
|
||
### Blosc Codec (Default)
|
||
```python
|
||
from zarr.codecs.blosc import BloscCodec
|
||
|
||
codec = BloscCodec(
|
||
cname='zstd', # Compressor: 'blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd'
|
||
clevel=5, # Compression level: 0-9
|
||
shuffle='shuffle' # Shuffle filter: 'noshuffle', 'shuffle', 'bitshuffle'
|
||
)
|
||
|
||
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
|
||
dtype='f4', codecs=[codec])
|
||
```
|
||
|
||
**Blosc compressor characteristics:**
|
||
- `'lz4'`: Fastest compression, lower ratio
|
||
- `'zstd'`: Balanced (default), good ratio and speed
|
||
- `'zlib'`: Good compatibility, moderate performance
|
||
- `'lz4hc'`: Better ratio than lz4, slower
|
||
- `'snappy'`: Fast, moderate ratio
|
||
- `'blosclz'`: Blosc's default
|
||
|
||
### Other Codecs
|
||
```python
|
||
from zarr.codecs import GzipCodec, ZstdCodec, BytesCodec
|
||
|
||
# Gzip compression (maximum ratio, slower)
|
||
GzipCodec(level=6) # Level 0-9
|
||
|
||
# Zstandard compression
|
||
ZstdCodec(level=3) # Level 1-22
|
||
|
||
# No compression
|
||
BytesCodec()
|
||
```
|
||
|
||
## Array Indexing and Selection
|
||
|
||
### Basic Indexing (NumPy-style)
|
||
```python
|
||
z = zarr.zeros((1000, 1000), chunks=(100, 100))
|
||
|
||
# Read
|
||
row = z[0, :] # Single row
|
||
col = z[:, 0] # Single column
|
||
block = z[10:20, 50:60] # Slice
|
||
element = z[5, 10] # Single element
|
||
|
||
# Write
|
||
z[0, :] = 42
|
||
z[10:20, 50:60] = np.random.random((10, 10))
|
||
```
|
||
|
||
### Advanced Indexing
|
||
```python
|
||
# Coordinate indexing (point selection)
|
||
z.vindex[[0, 5, 10], [2, 8, 15]] # Specific coordinates
|
||
|
||
# Orthogonal indexing (outer product)
|
||
z.oindex[0:10, [5, 10, 15]] # Rows 0-9, columns 5, 10, 15
|
||
|
||
# Block/chunk indexing
|
||
z.blocks[0, 0] # First chunk
|
||
z.blocks[0:2, 0:2] # First four chunks
|
||
```
|
||
|
||
## Groups and Hierarchies
|
||
|
||
### Creating Groups
|
||
```python
|
||
# Create root group
|
||
root = zarr.group(store='data.zarr')
|
||
|
||
# Create nested groups
|
||
grp1 = root.create_group('group1')
|
||
grp2 = grp1.create_group('subgroup')
|
||
|
||
# Create arrays in groups
|
||
arr = grp1.create_array(name='data', shape=(1000, 1000),
|
||
chunks=(100, 100), dtype='f4')
|
||
|
||
# Access by path
|
||
arr2 = root['group1/data']
|
||
```
|
||
|
||
### Group Methods
|
||
```python
|
||
root = zarr.group('data.zarr')
|
||
|
||
# h5py-compatible methods
|
||
dataset = root.create_dataset('data', shape=(1000, 1000), chunks=(100, 100))
|
||
subgrp = root.require_group('subgroup') # Create if doesn't exist
|
||
|
||
# Visualize structure
|
||
print(root.tree())
|
||
|
||
# List contents
|
||
print(list(root.keys()))
|
||
print(list(root.groups()))
|
||
print(list(root.arrays()))
|
||
```
|
||
|
||
## Array Attributes and Metadata
|
||
|
||
### Working with Attributes
|
||
```python
|
||
z = zarr.zeros((1000, 1000), chunks=(100, 100))
|
||
|
||
# Set attributes
|
||
z.attrs['units'] = 'meters'
|
||
z.attrs['description'] = 'Temperature data'
|
||
z.attrs['created'] = '2024-01-15'
|
||
z.attrs['version'] = 1.2
|
||
z.attrs['tags'] = ['climate', 'temperature']
|
||
|
||
# Read attributes
|
||
print(z.attrs['units'])
|
||
print(dict(z.attrs)) # All attributes as dict
|
||
|
||
# Update/delete
|
||
z.attrs['version'] = 2.0
|
||
del z.attrs['tags']
|
||
```
|
||
|
||
**Note:** Attributes must be JSON-serializable.
|
||
|
||
## Array Properties and Methods
|
||
|
||
### Properties
|
||
```python
|
||
z = zarr.zeros((1000, 1000), chunks=(100, 100), dtype='f4')
|
||
|
||
z.shape # (1000, 1000)
|
||
z.chunks # (100, 100)
|
||
z.dtype # dtype('float32')
|
||
z.size # 1000000
|
||
z.nbytes # 4000000 (uncompressed size in bytes)
|
||
z.nbytes_stored # Actual compressed size on disk
|
||
z.nchunks # 100 (number of chunks)
|
||
z.cdata_shape # Shape in terms of chunks: (10, 10)
|
||
```
|
||
|
||
### Methods
|
||
```python
|
||
# Information
|
||
print(z.info) # Detailed information about array
|
||
print(z.info_items()) # Info as list of tuples
|
||
|
||
# Resizing
|
||
z.resize(1500, 1500) # Change dimensions
|
||
|
||
# Appending
|
||
z.append(new_data, axis=0) # Add data along axis
|
||
|
||
# Copying
|
||
z2 = z.copy(store='new_location.zarr')
|
||
```
|
||
|
||
## Chunking Guidelines
|
||
|
||
### Chunk Size Calculation
|
||
```python
|
||
# For float32 (4 bytes per element):
|
||
# 1 MB = 262,144 elements
|
||
# 10 MB = 2,621,440 elements
|
||
|
||
# Examples for 1 MB chunks:
|
||
(512, 512) # For 2D: 512 × 512 × 4 = 1,048,576 bytes
|
||
(128, 128, 128) # For 3D: 128 × 128 × 128 × 4 = 8,388,608 bytes ≈ 8 MB
|
||
(64, 256, 256) # For 3D: 64 × 256 × 256 × 4 = 16,777,216 bytes ≈ 16 MB
|
||
```
|
||
|
||
### Chunking Strategies by Access Pattern
|
||
|
||
**Time series (sequential access along first dimension):**
|
||
```python
|
||
chunks=(1, 720, 1440) # One time step per chunk
|
||
```
|
||
|
||
**Row-wise access:**
|
||
```python
|
||
chunks=(10, 10000) # Small rows, span columns
|
||
```
|
||
|
||
**Column-wise access:**
|
||
```python
|
||
chunks=(10000, 10) # Span rows, small columns
|
||
```
|
||
|
||
**Random access:**
|
||
```python
|
||
chunks=(500, 500) # Balanced square chunks
|
||
```
|
||
|
||
**3D volumetric data:**
|
||
```python
|
||
chunks=(64, 64, 64) # Cubic chunks for isotropic access
|
||
```
|
||
|
||
## Integration APIs
|
||
|
||
### NumPy Integration
|
||
```python
|
||
import numpy as np
|
||
|
||
z = zarr.zeros((1000, 1000), chunks=(100, 100))
|
||
|
||
# Use NumPy functions
|
||
result = np.sum(z, axis=0)
|
||
mean = np.mean(z)
|
||
std = np.std(z)
|
||
|
||
# Convert to NumPy
|
||
arr = z[:] # Loads entire array into memory
|
||
```
|
||
|
||
### Dask Integration
|
||
```python
|
||
import dask.array as da
|
||
|
||
# Load Zarr as Dask array
|
||
dask_array = da.from_zarr('data.zarr')
|
||
|
||
# Compute operations in parallel
|
||
result = dask_array.mean(axis=0).compute()
|
||
|
||
# Write Dask array to Zarr
|
||
large_array = da.random.random((100000, 100000), chunks=(1000, 1000))
|
||
da.to_zarr(large_array, 'output.zarr')
|
||
```
|
||
|
||
### Xarray Integration
|
||
```python
|
||
import xarray as xr
|
||
|
||
# Open Zarr as Xarray Dataset
|
||
ds = xr.open_zarr('data.zarr')
|
||
|
||
# Write Xarray to Zarr
|
||
ds.to_zarr('output.zarr')
|
||
|
||
# Create with coordinates
|
||
ds = xr.Dataset(
|
||
{'temperature': (['time', 'lat', 'lon'], data)},
|
||
coords={
|
||
'time': pd.date_range('2024-01-01', periods=365),
|
||
'lat': np.arange(-90, 91, 1),
|
||
'lon': np.arange(-180, 180, 1)
|
||
}
|
||
)
|
||
ds.to_zarr('climate.zarr')
|
||
```
|
||
|
||
## Parallel Computing
|
||
|
||
### Synchronizers
|
||
```python
|
||
from zarr import ThreadSynchronizer, ProcessSynchronizer
|
||
|
||
# Multi-threaded writes
|
||
sync = ThreadSynchronizer()
|
||
z = zarr.open_array('data.zarr', mode='r+', synchronizer=sync)
|
||
|
||
# Multi-process writes
|
||
sync = ProcessSynchronizer('sync.sync')
|
||
z = zarr.open_array('data.zarr', mode='r+', synchronizer=sync)
|
||
```
|
||
|
||
**Note:** Synchronization only needed for:
|
||
- Concurrent writes that may span chunk boundaries
|
||
- Not needed for reads (always safe)
|
||
- Not needed if each process writes to separate chunks
|
||
|
||
## Metadata Consolidation
|
||
|
||
```python
|
||
# Consolidate metadata (after creating all arrays/groups)
|
||
zarr.consolidate_metadata('data.zarr')
|
||
|
||
# Open with consolidated metadata (faster, especially on cloud)
|
||
root = zarr.open_consolidated('data.zarr')
|
||
```
|
||
|
||
**Benefits:**
|
||
- Reduces I/O from N operations to 1
|
||
- Critical for cloud storage (reduces latency)
|
||
- Speeds up hierarchy traversal
|
||
|
||
**Cautions:**
|
||
- Can become stale if data updates
|
||
- Re-consolidate after modifications
|
||
- Not for frequently-updated datasets
|
||
|
||
## Common Patterns
|
||
|
||
### Time Series with Growing Data
|
||
```python
|
||
# Start with empty first dimension
|
||
z = zarr.open('timeseries.zarr', mode='a',
|
||
shape=(0, 720, 1440),
|
||
chunks=(1, 720, 1440),
|
||
dtype='f4')
|
||
|
||
# Append new time steps
|
||
for new_timestep in data_stream:
|
||
z.append(new_timestep, axis=0)
|
||
```
|
||
|
||
### Processing Large Arrays in Chunks
|
||
```python
|
||
z = zarr.open('large_data.zarr', mode='r')
|
||
|
||
# Process without loading entire array
|
||
for i in range(0, z.shape[0], 1000):
|
||
chunk = z[i:i+1000, :]
|
||
result = process(chunk)
|
||
save(result)
|
||
```
|
||
|
||
### Format Conversion Pipeline
|
||
```python
|
||
# HDF5 → Zarr
|
||
import h5py
|
||
with h5py.File('data.h5', 'r') as h5:
|
||
z = zarr.array(h5['dataset'][:], chunks=(1000, 1000), store='data.zarr')
|
||
|
||
# Zarr → NumPy file
|
||
z = zarr.open('data.zarr', mode='r')
|
||
np.save('data.npy', z[:])
|
||
|
||
# Zarr → NetCDF (via Xarray)
|
||
ds = xr.open_zarr('data.zarr')
|
||
ds.to_netcdf('data.nc')
|
||
```
|
||
|
||
## Performance Optimization Quick Checklist
|
||
|
||
1. **Chunk size**: 1-10 MB per chunk
|
||
2. **Chunk shape**: Align with access pattern
|
||
3. **Compression**:
|
||
- Fast: `BloscCodec(cname='lz4', clevel=1)`
|
||
- Balanced: `BloscCodec(cname='zstd', clevel=5)`
|
||
- Maximum: `GzipCodec(level=9)`
|
||
4. **Cloud storage**:
|
||
- Larger chunks (5-100 MB)
|
||
- Consolidate metadata
|
||
- Consider sharding
|
||
5. **Parallel I/O**: Use Dask for large operations
|
||
6. **Memory**: Process in chunks, don't load entire arrays
|
||
|
||
## Debugging and Profiling
|
||
|
||
```python
|
||
z = zarr.open('data.zarr', mode='r')
|
||
|
||
# Detailed information
|
||
print(z.info)
|
||
|
||
# Size statistics
|
||
print(f"Uncompressed: {z.nbytes / 1e6:.2f} MB")
|
||
print(f"Compressed: {z.nbytes_stored / 1e6:.2f} MB")
|
||
print(f"Ratio: {z.nbytes / z.nbytes_stored:.1f}x")
|
||
|
||
# Chunk information
|
||
print(f"Chunks: {z.chunks}")
|
||
print(f"Number of chunks: {z.nchunks}")
|
||
print(f"Chunk grid: {z.cdata_shape}")
|
||
```
|
||
|
||
## Common Data Types
|
||
|
||
```python
|
||
# Integers
|
||
'i1', 'i2', 'i4', 'i8' # Signed: 8, 16, 32, 64-bit
|
||
'u1', 'u2', 'u4', 'u8' # Unsigned: 8, 16, 32, 64-bit
|
||
|
||
# Floats
|
||
'f2', 'f4', 'f8' # 16, 32, 64-bit (half, single, double precision)
|
||
|
||
# Others
|
||
'bool' # Boolean
|
||
'c8', 'c16' # Complex: 64, 128-bit
|
||
'S10' # Fixed-length string (10 bytes)
|
||
'U10' # Unicode string (10 characters)
|
||
```
|
||
|
||
## Version Compatibility
|
||
|
||
Zarr-Python version 3.x supports both:
|
||
- **Zarr v2 format**: Legacy format, widely compatible
|
||
- **Zarr v3 format**: New format with sharding, improved metadata
|
||
|
||
Check format version:
|
||
```python
|
||
# Zarr automatically detects format version
|
||
z = zarr.open('data.zarr', mode='r')
|
||
# Format info available in metadata
|
||
```
|
||
|
||
## Error Handling
|
||
|
||
```python
|
||
try:
|
||
z = zarr.open_array('data.zarr', mode='r')
|
||
except zarr.errors.PathNotFoundError:
|
||
print("Array does not exist")
|
||
except zarr.errors.ReadOnlyError:
|
||
print("Cannot write to read-only array")
|
||
except Exception as e:
|
||
print(f"Unexpected error: {e}")
|
||
```
|