# Performance and Optimization

This reference covers Vaex's performance features including lazy evaluation, caching, memory management, async operations, and optimization strategies for processing massive datasets.

## Understanding Lazy Evaluation

Lazy evaluation is the foundation of Vaex's performance:

### How Lazy Evaluation Works

```python
import vaex

df = vaex.open('large_file.hdf5')

# No computation happens here - just defines what to compute
df['total'] = df.price * df.quantity
df['log_price'] = df.price.log()
mean_expr = df.total.mean()

# Computation happens here (when result is needed)
result = mean_expr  # Now the mean is actually calculated
```

**Key concepts:**
- **Expressions** are lazy - they define computations without executing them
- **Materialization** happens when you access the result
- **Query optimization** happens automatically before execution

### When Does Evaluation Happen?

```python
# These trigger evaluation:
print(df.x.mean())                    # Accessing value
array = df.x.values                   # Getting NumPy array
pdf = df.to_pandas_df()              # Converting to pandas
df.export_hdf5('output.hdf5')       # Exporting

# These do NOT trigger evaluation:
df['new_col'] = df.x + df.y          # Creating virtual column
expr = df.x.mean()                    # Creating expression
df_filtered = df[df.x > 10]          # Creating filtered view
```

## Batching Operations with delay=True

Execute multiple operations together for better performance:

### Basic Delayed Execution

```python
# Without delay - each operation processes entire dataset
mean_x = df.x.mean()      # Pass 1 through data
std_x = df.x.std()        # Pass 2 through data
max_x = df.x.max()        # Pass 3 through data

# With delay - single pass through dataset
mean_x = df.x.mean(delay=True)
std_x = df.x.std(delay=True)
max_x = df.x.max(delay=True)
results = vaex.execute([mean_x, std_x, max_x])  # Single pass!

print(results[0])  # mean
print(results[1])  # std
print(results[2])  # max
```

### Delayed Execution with Multiple Columns

```python
# Compute statistics for many columns efficiently
stats = {}
delayed_results = []

for column in ['sales', 'quantity', 'profit', 'cost']:
    mean = df[column].mean(delay=True)
    std = df[column].std(delay=True)
    delayed_results.extend([mean, std])

# Execute all at once
results = vaex.execute(delayed_results)

# Process results
for i, column in enumerate(['sales', 'quantity', 'profit', 'cost']):
    stats[column] = {
        'mean': results[i*2],
        'std': results[i*2 + 1]
    }
```

### When to Use delay=True

Use `delay=True` when:
- Computing multiple aggregations
- Computing statistics on many columns
- Building dashboards or reports
- Any scenario requiring multiple passes through data

```python
# Bad: 4 passes through dataset
mean1 = df.col1.mean()
mean2 = df.col2.mean()
mean3 = df.col3.mean()
mean4 = df.col4.mean()

# Good: 1 pass through dataset
results = vaex.execute([
    df.col1.mean(delay=True),
    df.col2.mean(delay=True),
    df.col3.mean(delay=True),
    df.col4.mean(delay=True)
])
```

## Asynchronous Operations

Process data asynchronously using async/await:

### Async with async/await

```python
import vaex
import asyncio

async def compute_statistics(df):
    # Create async tasks
    mean_task = df.x.mean(delay=True)
    std_task = df.x.std(delay=True)

    # Execute asynchronously
    results = await vaex.async_execute([mean_task, std_task])

    return {'mean': results[0], 'std': results[1]}

# Run async function
async def main():
    df = vaex.open('large_file.hdf5')
    stats = await compute_statistics(df)
    print(stats)

asyncio.run(main())
```

### Using Promises/Futures

```python
# Get future object
future = df.x.mean(delay=True)

# Do other work...

# Get result when ready
result = future.get()  # Blocks until complete
```

## Virtual Columns vs Materialized Columns

Understanding the difference is crucial for performance:

### Virtual Columns (Preferred)

```python
# Virtual column - computed on-the-fly, zero memory
df['total'] = df.price * df.quantity
df['log_sales'] = df.sales.log()
df['full_name'] = df.first_name + ' ' + df.last_name

# Check if virtual
print(df.is_local('total'))  # False = virtual

# Benefits:
# - Zero memory overhead
# - Always up-to-date if source data changes
# - Fast to create
```

### Materialized Columns

```python
# Materialize a virtual column
df['total_materialized'] = df['total'].values

# Or use materialize method
df = df.materialize(df['total'], inplace=True)

# Check if materialized
print(df.is_local('total_materialized'))  # True = materialized

# When to materialize:
# - Column computed repeatedly (amortize cost)
# - Complex expression used in many operations
# - Need to export data
```

### Deciding: Virtual vs Materialized

```python
# Virtual is better when:
# - Column is simple (x + y, x * 2, etc.)
# - Column used infrequently
# - Memory is limited

# Materialize when:
# - Complex computation (multiple operations)
# - Used repeatedly in aggregations
# - Slows down other operations

# Example: Complex calculation used many times
df['complex'] = (df.x.log() * df.y.sqrt() + df.z ** 2).values  # Materialize
```

## Caching Strategies

Vaex automatically caches some operations, but you can optimize further:

### Automatic Caching

```python
# First call computes and caches
mean1 = df.x.mean()  # Computes

# Second call uses cache
mean2 = df.x.mean()  # From cache (instant)

# Cache invalidated if DataFrame changes
df['new_col'] = df.x + 1
mean3 = df.x.mean()  # Recomputes
```

### State Management

```python
# Save DataFrame state (includes virtual columns)
df.state_write('state.json')

# Load state later
df_new = vaex.open('data.hdf5')
df_new.state_load('state.json')  # Restores virtual columns, selections
```

### Checkpoint Pattern

```python
# Export intermediate results for complex pipelines
df['processed'] = complex_calculation(df)

# Save checkpoint
df.export_hdf5('checkpoint.hdf5')

# Resume from checkpoint
df = vaex.open('checkpoint.hdf5')
# Continue processing...
```

## Memory Management

Optimize memory usage for very large datasets:

### Memory-Mapped Files

```python
# HDF5 and Arrow are memory-mapped (optimal)
df = vaex.open('data.hdf5')  # No memory used until accessed

# File stays on disk, only accessed portions loaded to RAM
mean = df.x.mean()  # Streams through data, minimal memory
```

### Chunked Processing

```python
# Process large DataFrame in chunks
chunk_size = 1_000_000

for i1, i2, chunk in df.to_pandas_df(chunk_size=chunk_size):
    # Process chunk (careful: defeats Vaex's purpose)
    process_chunk(chunk)

# Better: Use Vaex operations directly (no chunking needed)
result = df.x.mean()  # Handles large data automatically
```

### Monitoring Memory Usage

```python
# Check DataFrame memory footprint
print(df.byte_size())  # Bytes used by materialized columns

# Check column memory
for col in df.get_column_names():
    if df.is_local(col):
        print(f"{col}: {df[col].nbytes / 1e9:.2f} GB")

# Profile operations
import vaex.profiler
with vaex.profiler():
    result = df.x.mean()
```

## Parallel Computation

Vaex automatically parallelizes operations:

### Multithreading

```python
# Vaex uses all CPU cores by default
import vaex

# Check/set thread count
print(vaex.multithreading.thread_count_default)
vaex.multithreading.thread_count_default = 8  # Use 8 threads

# Operations automatically parallelize
mean = df.x.mean()  # Uses all threads
```

### Distributed Computing with Dask

```python
# Convert to Dask for distributed processing
import vaex
import dask.dataframe as dd

# Create Vaex DataFrame
df_vaex = vaex.open('large_file.hdf5')

# Convert to Dask
df_dask = df_vaex.to_dask_dataframe()

# Process with Dask
result = df_dask.groupby('category')['value'].sum().compute()
```

## JIT Compilation

Vaex can use Just-In-Time compilation for custom operations:

### Using Numba

```python
import vaex
import numba

# Define JIT-compiled function
@numba.jit
def custom_calculation(x, y):
    return x ** 2 + y ** 2

# Apply to DataFrame
df['custom'] = df.apply(custom_calculation,
                        arguments=[df.x, df.y],
                        vectorize=True)
```

### Custom Aggregations

```python
@numba.jit
def custom_sum(a):
    total = 0
    for val in a:
        total += val * 2  # Custom logic
    return total

# Use in aggregation
result = df.x.custom_agg(custom_sum)
```

## Optimization Strategies

### Strategy 1: Minimize Materializations

```python
# Bad: Creates many materialized columns
df['a'] = (df.x + df.y).values
df['b'] = (df.a * 2).values
df['c'] = (df.b + df.z).values

# Good: Keep virtual until final export
df['a'] = df.x + df.y
df['b'] = df.a * 2
df['c'] = df.b + df.z
# Only materialize if exporting:
# df.export_hdf5('output.hdf5')
```

### Strategy 2: Use Selections Instead of Filtering

```python
# Less efficient: Creates new DataFrames
df_high = df[df.value > 100]
df_low = df[df.value <= 100]
mean_high = df_high.value.mean()
mean_low = df_low.value.mean()

# More efficient: Use selections
df.select(df.value > 100, name='high')
df.select(df.value <= 100, name='low')
mean_high = df.value.mean(selection='high')
mean_low = df.value.mean(selection='low')
```

### Strategy 3: Batch Aggregations

```python
# Less efficient: Multiple passes
stats = {
    'mean': df.x.mean(),
    'std': df.x.std(),
    'min': df.x.min(),
    'max': df.x.max()
}

# More efficient: Single pass
delayed = [
    df.x.mean(delay=True),
    df.x.std(delay=True),
    df.x.min(delay=True),
    df.x.max(delay=True)
]
results = vaex.execute(delayed)
stats = dict(zip(['mean', 'std', 'min', 'max'], results))
```

### Strategy 4: Choose Optimal File Formats

```python
# Slow: Large CSV
df = vaex.from_csv('huge.csv')  # Can take minutes

# Fast: HDF5 or Arrow
df = vaex.open('huge.hdf5')     # Instant
df = vaex.open('huge.arrow')    # Instant

# One-time conversion
df = vaex.from_csv('huge.csv', convert='huge.hdf5')
# Future loads: vaex.open('huge.hdf5')
```

### Strategy 5: Optimize Expressions

```python
# Less efficient: Repeated calculations
df['result'] = df.x.log() + df.x.log() * 2

# More efficient: Reuse calculations
df['log_x'] = df.x.log()
df['result'] = df.log_x + df.log_x * 2

# Even better: Combine operations
df['result'] = df.x.log() * 3  # Simplified math
```

## Performance Profiling

### Basic Profiling

```python
import time
import vaex

df = vaex.open('large_file.hdf5')

# Time operations
start = time.time()
result = df.x.mean()
elapsed = time.time() - start
print(f"Computed in {elapsed:.2f} seconds")
```

### Detailed Profiling

```python
# Profile with context manager
with vaex.profiler():
    result = df.groupby('category').agg({'value': 'sum'})
# Prints detailed timing information
```

### Benchmarking Patterns

```python
# Compare strategies
def benchmark_operation(operation, name):
    start = time.time()
    result = operation()
    elapsed = time.time() - start
    print(f"{name}: {elapsed:.3f}s")
    return result

# Test different approaches
benchmark_operation(lambda: df.x.mean(), "Direct mean")
benchmark_operation(lambda: df[df.x > 0].x.mean(), "Filtered mean")
benchmark_operation(lambda: df.x.mean(selection='positive'), "Selection mean")
```

## Common Performance Issues and Solutions

### Issue: Slow Aggregations

```python
# Problem: Multiple separate aggregations
for col in df.column_names:
    print(f"{col}: {df[col].mean()}")

# Solution: Batch with delay=True
delayed = [df[col].mean(delay=True) for col in df.column_names]
results = vaex.execute(delayed)
for col, result in zip(df.column_names, results):
    print(f"{col}: {result}")
```

### Issue: High Memory Usage

```python
# Problem: Materializing large virtual columns
df['large_col'] = (complex_expression).values

# Solution: Keep virtual, or materialize and export
df['large_col'] = complex_expression  # Virtual
# Or: df.export_hdf5('with_new_col.hdf5')
```

### Issue: Slow Exports

```python
# Problem: Exporting with many virtual columns
df.export_csv('output.csv')  # Slow if many virtual columns

# Solution: Export to HDF5 or Arrow (faster)
df.export_hdf5('output.hdf5')
df.export_arrow('output.arrow')

# Or materialize first for CSV
df_materialized = df.materialize()
df_materialized.export_csv('output.csv')
```

### Issue: Repeated Complex Calculations

```python
# Problem: Complex virtual column used repeatedly
df['complex'] = df.x.log() * df.y.sqrt() + df.z ** 3
result1 = df.groupby('cat1').agg({'complex': 'mean'})
result2 = df.groupby('cat2').agg({'complex': 'sum'})
result3 = df.complex.std()

# Solution: Materialize once
df['complex'] = (df.x.log() * df.y.sqrt() + df.z ** 3).values
# Or: df = df.materialize('complex')
```

## Performance Best Practices Summary

1. **Use HDF5 or Arrow formats** - Orders of magnitude faster than CSV
2. **Leverage lazy evaluation** - Don't force computation until necessary
3. **Batch operations with delay=True** - Minimize passes through data
4. **Keep columns virtual** - Materialize only when beneficial
5. **Use selections not filters** - More efficient for multiple segments
6. **Profile your code** - Identify bottlenecks before optimizing
7. **Avoid `.values` and `.to_pandas_df()`** - Keep operations in Vaex
8. **Parallelize naturally** - Vaex uses all cores automatically
9. **Export to efficient formats** - Checkpoint complex pipelines
10. **Optimize expressions** - Simplify math and reuse calculations

## Related Resources

- For DataFrame basics: See `core_dataframes.md`
- For data operations: See `data_processing.md`
- For file I/O optimization: See `io_operations.md`