2263 lines
69 KiB
Markdown
2263 lines
69 KiB
Markdown
|
|
# Debugging Simulation Chaos
|
|
|
|
## Description
|
|
Master deterministic simulation, replay systems, debug visualization, chaos identification, and systematic debugging methodologies for game simulations. Build deterministic multiplayer simulations where clients stay perfectly in sync. Implement replay recording, debug visualization tools, and rigorous testing strategies. Understand butterfly effects, floating-point non-determinism, and order-dependence. Apply scientific method to debugging: reproduce, isolate, hypothesize, test, verify.
|
|
|
|
## When to Use This Skill
|
|
Use this skill when debugging or implementing:
|
|
- Multiplayer game simulations (physics, ecosystems, AI) that must stay in sync
|
|
- Replay systems (save game state and inputs for playback)
|
|
- Non-reproducible bugs in simulations (works sometimes, fails other times)
|
|
- Desync issues (clients see different simulation states)
|
|
- "Works on my machine" problems (platform-specific non-determinism)
|
|
- Chaotic systems (small input changes cause massive output differences)
|
|
- Emergent bugs (simulation stable for hours, then suddenly collapses)
|
|
|
|
Do NOT use this skill for:
|
|
- Simple single-player games with no replay requirements (determinism not critical)
|
|
- Purely cosmetic/visual bugs (animation glitches, rendering issues)
|
|
- Network latency/packet loss problems (this is about simulation determinism, not networking)
|
|
- Intentionally random gameplay (roguelikes where variance is desired)
|
|
|
|
|
|
## Quick Start (Emergency Desync Debugging)
|
|
|
|
If you have a critical multiplayer desync bug and need to debug NOW (< 4 hours):
|
|
|
|
**CRITICAL (Do This First)**:
|
|
1. **Build minimal replay system** - Record inputs, replay identical sequence
|
|
2. **Add state checksums** - Hash simulation state every frame, compare between clients
|
|
3. **Identify divergence point** - Binary search to find first frame where states differ
|
|
4. **Check common culprits**:
|
|
- Unseeded `Math.random()` → Use seeded RNG
|
|
- `Map`/`Set` iteration → Sort keys before iterating
|
|
- Floating point accumulation → Use fixed-point or integer math for critical values
|
|
- `Date.now()` / timing → Use simulation ticks, not wall-clock time
|
|
|
|
**IMPORTANT (Within First Hour)**:
|
|
5. Add debug visualization (draw entity positions, show state values)
|
|
6. Write unit test that reproduces bug (given same inputs → same outputs)
|
|
7. Binary search the problem (comment out subsystems until desync disappears)
|
|
8. Add assertions (check invariants every frame, fail fast on divergence)
|
|
|
|
**CAN DEFER** (After Fix):
|
|
- Full replay UI with scrubbing/playback controls
|
|
- Property-based testing for all invariants
|
|
- Comprehensive debug draw system
|
|
- Performance profiling tools
|
|
|
|
**Example - Emergency Desync Fix in 2 Hours**:
|
|
```python
|
|
import hashlib
|
|
import json
|
|
import random
|
|
|
|
# 1. DETERMINISTIC RNG (seed per simulation)
|
|
class SeededRandom:
|
|
def __init__(self, seed):
|
|
self.rng = random.Random(seed)
|
|
|
|
def random(self):
|
|
return self.rng.random()
|
|
|
|
def randint(self, a, b):
|
|
return self.rng.randint(a, b)
|
|
|
|
# 2. STATE CHECKSUM (detect divergence immediately)
|
|
def checksum_state(simulation):
|
|
"""Hash simulation state to detect desyncs"""
|
|
# Sort all dictionaries/sets to ensure consistent order
|
|
state = {
|
|
'deer': sorted([(d.id, d.x, d.y, d.energy) for d in simulation.deer]),
|
|
'wolves': sorted([(w.id, w.x, w.y, w.energy) for w in simulation.wolves]),
|
|
'grass': simulation.grass,
|
|
'tick': simulation.tick
|
|
}
|
|
state_json = json.dumps(state, sort_keys=True)
|
|
return hashlib.sha256(state_json.encode()).hexdigest()
|
|
|
|
# 3. REPLAY SYSTEM (reproduce bug)
|
|
class ReplaySystem:
|
|
def __init__(self):
|
|
self.recorded_inputs = []
|
|
self.is_recording = True
|
|
self.is_replaying = False
|
|
self.replay_index = 0
|
|
|
|
def record_input(self, input_event):
|
|
"""Record input during normal gameplay"""
|
|
if self.is_recording:
|
|
self.recorded_inputs.append({
|
|
'tick': input_event['tick'],
|
|
'action': input_event['action'],
|
|
'data': input_event['data']
|
|
})
|
|
|
|
def start_replay(self):
|
|
"""Start replaying recorded inputs"""
|
|
self.is_replaying = True
|
|
self.is_recording = False
|
|
self.replay_index = 0
|
|
|
|
def get_next_input(self, current_tick):
|
|
"""Get next input event if available"""
|
|
if not self.is_replaying or self.replay_index >= len(self.recorded_inputs):
|
|
return None
|
|
|
|
next_input = self.recorded_inputs[self.replay_index]
|
|
if next_input['tick'] == current_tick:
|
|
self.replay_index += 1
|
|
return next_input
|
|
return None
|
|
|
|
def save_replay(self, filename):
|
|
"""Save replay to file"""
|
|
with open(filename, 'w') as f:
|
|
json.dump(self.recorded_inputs, f)
|
|
|
|
def load_replay(self, filename):
|
|
"""Load replay from file"""
|
|
with open(filename, 'r') as f:
|
|
self.recorded_inputs = json.load(f)
|
|
|
|
# 4. DETERMINISTIC SIMULATION
|
|
class DeterministicEcosystem:
|
|
def __init__(self, seed):
|
|
self.rng = SeededRandom(seed)
|
|
self.tick = 0
|
|
self.deer = []
|
|
self.wolves = []
|
|
self.grass = 1000.0
|
|
self.replay = ReplaySystem()
|
|
|
|
# Initialize with deterministic entity IDs
|
|
for i in range(10):
|
|
self.deer.append(Deer(id=i, x=i*10.0, y=0.0, energy=100.0))
|
|
|
|
def step(self, dt=1.0):
|
|
"""One simulation step - MUST be deterministic"""
|
|
self.tick += 1
|
|
|
|
# Process in deterministic order (sorted by ID)
|
|
self.deer.sort(key=lambda d: d.id)
|
|
self.wolves.sort(key=lambda w: w.id)
|
|
|
|
# Update grass (no randomness)
|
|
self.grass += 5.0 * dt
|
|
self.grass = min(1000.0, self.grass)
|
|
|
|
# Update deer (deterministic)
|
|
for deer in self.deer:
|
|
# Use seeded RNG for any randomness
|
|
if self.rng.random() < 0.1:
|
|
deer.reproduce()
|
|
|
|
# Checksum validation (detect desyncs immediately)
|
|
checksum = checksum_state(self)
|
|
print(f"Tick {self.tick}: Checksum = {checksum[:8]}")
|
|
|
|
return checksum
|
|
|
|
# 5. UNIT TEST (reproducibility)
|
|
def test_determinism():
|
|
"""Verify simulation is deterministic"""
|
|
# Run simulation twice with same seed
|
|
sim1 = DeterministicEcosystem(seed=42)
|
|
sim2 = DeterministicEcosystem(seed=42)
|
|
|
|
for _ in range(100):
|
|
checksum1 = sim1.step()
|
|
checksum2 = sim2.step()
|
|
|
|
assert checksum1 == checksum2, f"DESYNC at tick {sim1.tick}!"
|
|
|
|
print("✓ Determinism test passed - 100 ticks identical")
|
|
|
|
# Run test
|
|
test_determinism()
|
|
```
|
|
|
|
**This gives you:**
|
|
- ✅ Deterministic RNG (seeded per simulation)
|
|
- ✅ State checksums (detect desyncs immediately)
|
|
- ✅ Replay system (reproduce bugs)
|
|
- ✅ Deterministic ordering (sort before iterating)
|
|
- ✅ Unit test (verify determinism)
|
|
|
|
|
|
## Core Concepts
|
|
|
|
### 1. Determinism in Simulations
|
|
|
|
**What:** A deterministic simulation produces the exact same outputs given the same inputs and initial state. No variance, no "close enough" - bit-for-bit identical.
|
|
|
|
**Why Critical:**
|
|
- **Multiplayer sync:** Clients run same simulation, stay in sync
|
|
- **Replay systems:** Record inputs, replay produces identical results
|
|
- **Testing:** Bugs reproducible, not random "sometimes happens"
|
|
- **Debugging:** Can bisect to find exact tick where bug occurs
|
|
|
|
**Sources of Non-Determinism (ALL must be eliminated):**
|
|
|
|
#### 1.1 Random Number Generation
|
|
```python
|
|
# ❌ NON-DETERMINISTIC (uses system entropy)
|
|
import random
|
|
x = random.random() # Different on each client!
|
|
|
|
# ✅ DETERMINISTIC (seeded RNG)
|
|
class SeededRandom:
|
|
def __init__(self, seed):
|
|
self.rng = random.Random(seed)
|
|
|
|
def random(self):
|
|
return self.rng.random()
|
|
|
|
# Usage
|
|
rng = SeededRandom(seed=12345) # Same seed on all clients
|
|
x = rng.random() # Identical on all clients
|
|
```
|
|
|
|
**Key Insight:** Global `random.seed()` is NOT enough - need separate RNG instance per simulation (multiple simulations might run in same process).
|
|
|
|
#### 1.2 Map/Set Iteration Order
|
|
```python
|
|
# ❌ NON-DETERMINISTIC (iteration order undefined)
|
|
entities = {'wolf_1': wolf1, 'deer_2': deer2, 'wolf_3': wolf3}
|
|
for entity_id, entity in entities.items():
|
|
entity.update() # Order varies between runs!
|
|
|
|
# ✅ DETERMINISTIC (sort before iterating)
|
|
for entity_id in sorted(entities.keys()):
|
|
entities[entity_id].update() # Always same order
|
|
```
|
|
|
|
**Languages affected:** Python, JavaScript, Java (HashMap), C++ (unordered_map)
|
|
|
|
#### 1.3 Floating Point Accumulation
|
|
```python
|
|
# ❌ NON-DETERMINISTIC (accumulation errors differ)
|
|
position = 0.0
|
|
for _ in range(1000):
|
|
position += 0.1 # Accumulates rounding errors differently on platforms
|
|
|
|
# ✅ DETERMINISTIC (fixed-point or integer math)
|
|
position_int = 0 # Store as integer (units = 0.01)
|
|
for _ in range(1000):
|
|
position_int += 10 # Add 0.1 as integer
|
|
position = position_int / 100.0 # Convert back
|
|
```
|
|
|
|
**Example:** x86 vs ARM might compute `0.1 + 0.2` with different rounding. After 10,000 operations, divergence is massive.
|
|
|
|
#### 1.4 Timing Dependencies
|
|
```python
|
|
# ❌ NON-DETERMINISTIC (wall-clock time varies)
|
|
import time
|
|
current_time = time.time()
|
|
if current_time % 10 < 5:
|
|
spawn_enemy()
|
|
|
|
# ✅ DETERMINISTIC (simulation ticks)
|
|
self.tick += 1
|
|
if self.tick % 100 == 0:
|
|
spawn_enemy()
|
|
```
|
|
|
|
**Never use:** `Date.now()`, `Time.time()`, `performance.now()`, `clock_gettime()`
|
|
|
|
#### 1.5 Multithreading/Async
|
|
```python
|
|
# ❌ NON-DETERMINISTIC (thread scheduling varies)
|
|
import threading
|
|
|
|
def update_entity(entity):
|
|
entity.position += entity.velocity
|
|
|
|
threads = [threading.Thread(target=update_entity, args=(e,)) for e in entities]
|
|
for t in threads:
|
|
t.start()
|
|
for t in threads:
|
|
t.join()
|
|
|
|
# ✅ DETERMINISTIC (single-threaded or deterministic parallel)
|
|
for entity in entities:
|
|
entity.position += entity.velocity
|
|
```
|
|
|
|
**Rule:** Simulation must be single-threaded OR use deterministic parallel algorithms (map/reduce with associative operations).
|
|
|
|
#### 1.6 Platform-Specific Behavior
|
|
```python
|
|
# ❌ NON-DETERMINISTIC (math functions differ by platform)
|
|
import math
|
|
angle = math.sin(1.5) # x86 vs ARM might differ in last bit
|
|
|
|
# ✅ DETERMINISTIC (use consistent math library)
|
|
# Use fixed-point, or test on all platforms
|
|
# Or use deterministic math library (e.g., fixed-point trigonometry)
|
|
```
|
|
|
|
**Affected:** `sin`, `cos`, `sqrt` (hardware implementations vary)
|
|
|
|
### 2. Replay Systems
|
|
|
|
**What:** Record all inputs to a simulation, then replay them to reproduce exact behavior.
|
|
|
|
**Architecture:**
|
|
```
|
|
Normal Gameplay:
|
|
User Input → Simulation → Game State → Render
|
|
|
|
Replay:
|
|
Recorded Inputs → Simulation → Game State → Render
|
|
(identical to original)
|
|
```
|
|
|
|
**Key Components:**
|
|
|
|
#### 2.1 Input Recording
|
|
```python
|
|
class ReplayRecorder:
|
|
def __init__(self):
|
|
self.events = []
|
|
|
|
def record(self, tick, event_type, event_data):
|
|
"""Record an input event"""
|
|
self.events.append({
|
|
'tick': tick,
|
|
'type': event_type,
|
|
'data': event_data
|
|
})
|
|
|
|
def save(self, filename):
|
|
"""Save replay to file"""
|
|
import json
|
|
with open(filename, 'w') as f:
|
|
json.dump({
|
|
'version': 1,
|
|
'seed': self.initial_seed,
|
|
'events': self.events
|
|
}, f)
|
|
|
|
# Usage
|
|
recorder = ReplayRecorder()
|
|
recorder.record(tick=0, event_type='spawn_deer', data={'x': 10, 'y': 20})
|
|
recorder.record(tick=15, event_type='player_hunt', data={'target_id': 5})
|
|
recorder.save('bug_reproduction.replay')
|
|
```
|
|
|
|
#### 2.2 Replay Playback
|
|
```python
|
|
class ReplayPlayer:
|
|
def __init__(self, replay_data):
|
|
self.events = replay_data['events']
|
|
self.seed = replay_data['seed']
|
|
self.event_index = 0
|
|
|
|
def get_events_for_tick(self, tick):
|
|
"""Get all events that should occur this tick"""
|
|
events_this_tick = []
|
|
while self.event_index < len(self.events):
|
|
event = self.events[self.event_index]
|
|
if event['tick'] == tick:
|
|
events_this_tick.append(event)
|
|
self.event_index += 1
|
|
elif event['tick'] > tick:
|
|
break
|
|
else:
|
|
self.event_index += 1
|
|
return events_this_tick
|
|
|
|
# Usage
|
|
import json
|
|
with open('bug_reproduction.replay', 'r') as f:
|
|
replay_data = json.load(f)
|
|
|
|
player = ReplayPlayer(replay_data)
|
|
sim = Simulation(seed=replay_data['seed'])
|
|
|
|
for tick in range(1000):
|
|
events = player.get_events_for_tick(tick)
|
|
for event in events:
|
|
sim.apply_event(event)
|
|
sim.step()
|
|
```
|
|
|
|
#### 2.3 State Snapshots
|
|
```python
|
|
class SnapshotSystem:
|
|
def __init__(self):
|
|
self.snapshots = {}
|
|
|
|
def save_snapshot(self, tick, simulation):
|
|
"""Save complete simulation state"""
|
|
import copy
|
|
self.snapshots[tick] = copy.deepcopy(simulation.get_state())
|
|
|
|
def load_snapshot(self, tick):
|
|
"""Restore simulation state"""
|
|
return self.snapshots.get(tick)
|
|
|
|
# Usage - save every 100 ticks
|
|
snapshot_system = SnapshotSystem()
|
|
if sim.tick % 100 == 0:
|
|
snapshot_system.save_snapshot(sim.tick, sim)
|
|
|
|
# Can jump to any saved tick for debugging
|
|
sim.set_state(snapshot_system.load_snapshot(500))
|
|
```
|
|
|
|
**Real-World Example:** Rocket League saves input replay (not video) - only 100KB for 5-minute match.
|
|
|
|
### 3. Debug Visualization
|
|
|
|
**What:** Visual tools to inspect simulation state in real-time. Makes invisible bugs visible.
|
|
|
|
**Key Visualizations:**
|
|
|
|
#### 3.1 Debug Draw System
|
|
```python
|
|
class DebugDraw:
|
|
def __init__(self):
|
|
self.shapes = []
|
|
|
|
def draw_circle(self, x, y, radius, color, label=None):
|
|
"""Draw debug circle"""
|
|
self.shapes.append({
|
|
'type': 'circle',
|
|
'x': x, 'y': y,
|
|
'radius': radius,
|
|
'color': color,
|
|
'label': label
|
|
})
|
|
|
|
def draw_line(self, x1, y1, x2, y2, color):
|
|
"""Draw debug line"""
|
|
self.shapes.append({
|
|
'type': 'line',
|
|
'x1': x1, 'y1': y1,
|
|
'x2': x2, 'y2': y2,
|
|
'color': color
|
|
})
|
|
|
|
def draw_text(self, x, y, text, color):
|
|
"""Draw debug text"""
|
|
self.shapes.append({
|
|
'type': 'text',
|
|
'x': x, 'y': y,
|
|
'text': text,
|
|
'color': color
|
|
})
|
|
|
|
def clear(self):
|
|
"""Clear all debug shapes"""
|
|
self.shapes = []
|
|
|
|
# Usage in simulation
|
|
debug = DebugDraw()
|
|
|
|
for deer in simulation.deer:
|
|
# Draw deer position
|
|
debug.draw_circle(deer.x, deer.y, radius=2, color='green')
|
|
|
|
# Draw deer state
|
|
debug.draw_text(deer.x, deer.y + 3, f"E:{deer.energy:.0f}", color='white')
|
|
|
|
# Draw deer seeking food
|
|
if deer.target_grass:
|
|
debug.draw_line(deer.x, deer.y, deer.target_grass.x, deer.target_grass.y, color='yellow')
|
|
|
|
# Render debug shapes on screen
|
|
render_debug(debug.shapes)
|
|
```
|
|
|
|
#### 3.2 Population Graphs
|
|
```python
|
|
import matplotlib.pyplot as plt
|
|
|
|
class PopulationGraph:
|
|
def __init__(self):
|
|
self.history = {
|
|
'ticks': [],
|
|
'deer': [],
|
|
'wolves': [],
|
|
'grass': []
|
|
}
|
|
|
|
def record(self, tick, deer_count, wolf_count, grass_amount):
|
|
"""Record population snapshot"""
|
|
self.history['ticks'].append(tick)
|
|
self.history['deer'].append(deer_count)
|
|
self.history['wolves'].append(wolf_count)
|
|
self.history['grass'].append(grass_amount)
|
|
|
|
def plot(self):
|
|
"""Plot population over time"""
|
|
plt.figure(figsize=(12, 6))
|
|
plt.plot(self.history['ticks'], self.history['deer'], label='Deer', color='green')
|
|
plt.plot(self.history['ticks'], self.history['wolves'], label='Wolves', color='red')
|
|
plt.plot(self.history['ticks'], self.history['grass'], label='Grass', color='brown', alpha=0.5)
|
|
plt.xlabel('Tick')
|
|
plt.ylabel('Population')
|
|
plt.title('Ecosystem Population Over Time')
|
|
plt.legend()
|
|
plt.grid(True)
|
|
plt.show()
|
|
|
|
# Usage
|
|
graph = PopulationGraph()
|
|
for tick in range(1000):
|
|
simulation.step()
|
|
graph.record(tick, len(simulation.deer), len(simulation.wolves), simulation.grass)
|
|
|
|
# Visualize
|
|
graph.plot()
|
|
```
|
|
|
|
**What to Look For:**
|
|
- Divergence: Two clients' graphs should be identical
|
|
- Oscillations: Natural or chaotic?
|
|
- Extinction events: Sudden drops to zero
|
|
- Runaway growth: Exponential explosion
|
|
|
|
#### 3.3 State Comparison View
|
|
```python
|
|
class StateComparator:
|
|
def compare_states(self, state_a, state_b):
|
|
"""Find differences between two simulation states"""
|
|
differences = []
|
|
|
|
# Compare entity counts
|
|
if len(state_a['deer']) != len(state_b['deer']):
|
|
differences.append({
|
|
'type': 'count_mismatch',
|
|
'entity': 'deer',
|
|
'count_a': len(state_a['deer']),
|
|
'count_b': len(state_b['deer'])
|
|
})
|
|
|
|
# Compare entity states
|
|
for entity_id in state_a['deer']:
|
|
if entity_id not in state_b['deer']:
|
|
differences.append({
|
|
'type': 'missing_entity',
|
|
'entity_id': entity_id,
|
|
'present_in': 'A'
|
|
})
|
|
continue
|
|
|
|
deer_a = state_a['deer'][entity_id]
|
|
deer_b = state_b['deer'][entity_id]
|
|
|
|
# Check position
|
|
if deer_a['x'] != deer_b['x'] or deer_a['y'] != deer_b['y']:
|
|
differences.append({
|
|
'type': 'position_mismatch',
|
|
'entity_id': entity_id,
|
|
'position_a': (deer_a['x'], deer_a['y']),
|
|
'position_b': (deer_b['x'], deer_b['y']),
|
|
'distance': math.sqrt((deer_a['x'] - deer_b['x'])**2 +
|
|
(deer_a['y'] - deer_b['y'])**2)
|
|
})
|
|
|
|
# Check energy
|
|
if deer_a['energy'] != deer_b['energy']:
|
|
differences.append({
|
|
'type': 'energy_mismatch',
|
|
'entity_id': entity_id,
|
|
'energy_a': deer_a['energy'],
|
|
'energy_b': deer_b['energy'],
|
|
'delta': deer_a['energy'] - deer_b['energy']
|
|
})
|
|
|
|
return differences
|
|
|
|
# Usage
|
|
comparator = StateComparator()
|
|
differences = comparator.compare_states(client1_state, client2_state)
|
|
|
|
for diff in differences:
|
|
if diff['type'] == 'count_mismatch':
|
|
print(f"⚠️ {diff['entity']} count: {diff['count_a']} vs {diff['count_b']}")
|
|
elif diff['type'] == 'position_mismatch':
|
|
print(f"⚠️ Entity {diff['entity_id']} position off by {diff['distance']:.2f}")
|
|
```
|
|
|
|
### 4. Chaos Theory and Butterfly Effects
|
|
|
|
**What:** Chaotic systems are sensitive to initial conditions. Tiny differences (0.0001) compound into massive divergence.
|
|
|
|
**Example - Butterfly Effect:**
|
|
```python
|
|
def simulate_population(initial_deer):
|
|
deer = initial_deer
|
|
for tick in range(1000):
|
|
deer = deer * 1.1 - deer * deer / 500 # Logistic growth
|
|
return deer
|
|
|
|
# Tiny difference in initial condition
|
|
result1 = simulate_population(100.0000)
|
|
result2 = simulate_population(100.0001)
|
|
|
|
print(f"Result 1: {result1:.2f}")
|
|
print(f"Result 2: {result2:.2f}")
|
|
print(f"Difference: {abs(result1 - result2):.2f}")
|
|
|
|
# Output:
|
|
# Result 1: 450.23
|
|
# Result 2: 478.91
|
|
# Difference: 28.68 (Tiny 0.0001 input → massive 28.68 output!)
|
|
```
|
|
|
|
**Why This Matters:**
|
|
- Floating point errors accumulate
|
|
- "Close enough" is NOT enough
|
|
- Must be bit-for-bit identical
|
|
- Small bugs cause massive desyncs over time
|
|
|
|
**Feedback Loops Amplify Errors:**
|
|
```
|
|
Tick 0: Deer = 100 vs 100.0001 (0.0001% error)
|
|
↓
|
|
Tick 10: Deer = 110 vs 110.001 (0.001% error) - error grew 10x
|
|
↓
|
|
Tick 100: Deer = 250 vs 252 (0.8% error) - error grew 800x
|
|
↓
|
|
Tick 1000: Deer = 450 vs 478 (6% error) - error grew 6000x
|
|
```
|
|
|
|
**Identifying Chaotic Systems:**
|
|
- Oscillations grow over time (not dampen)
|
|
- Small parameter changes cause wildly different outcomes
|
|
- Sensitive to floating point precision
|
|
- Feedback loops (output becomes input)
|
|
|
|
**Testing for Chaos:**
|
|
```python
|
|
def test_sensitivity():
|
|
"""Test if system is sensitive to initial conditions"""
|
|
base_result = simulate(initial_value=100.0)
|
|
|
|
# Perturb initial condition by 0.01%
|
|
perturbed_result = simulate(initial_value=100.01)
|
|
|
|
error = abs(base_result - perturbed_result)
|
|
error_percent = error / base_result * 100
|
|
|
|
if error_percent > 1.0:
|
|
print(f"⚠️ CHAOTIC SYSTEM: 0.01% input → {error_percent:.1f}% output")
|
|
print("Requires exact determinism, no tolerance for error")
|
|
else:
|
|
print(f"✓ Stable system: 0.01% input → {error_percent:.3f}% output")
|
|
```
|
|
|
|
|
|
## Decision Frameworks
|
|
|
|
### Framework 1: When Does Determinism Matter?
|
|
|
|
**Question:** Do I need a fully deterministic simulation?
|
|
|
|
**Decision Tree:**
|
|
```
|
|
Q: Is this multiplayer with client-side prediction?
|
|
├─ YES → DETERMINISM REQUIRED
|
|
│ └─ Clients must stay in sync
|
|
│
|
|
└─ NO → Q: Do you need replay functionality?
|
|
├─ YES → DETERMINISM REQUIRED
|
|
│ └─ Replays must reproduce exactly
|
|
│
|
|
└─ NO → Q: Are bugs reproducible?
|
|
├─ NO → DETERMINISM HELPFUL
|
|
│ └─ Makes debugging much easier
|
|
│
|
|
└─ YES → Determinism optional
|
|
└─ Can tolerate some randomness
|
|
```
|
|
|
|
**Examples:**
|
|
|
|
| Game Type | Determinism | Why |
|
|
|-----------|-------------|-----|
|
|
| Multiplayer RTS (StarCraft) | REQUIRED | Lockstep networking, clients must sync |
|
|
| Fighting game (Street Fighter) | REQUIRED | Rollback netcode requires determinism |
|
|
| Single-player with replays (Rocket League) | REQUIRED | Replays must be exact |
|
|
| Competitive esports (any) | REQUIRED | Replays for analysis, bug reproduction |
|
|
| Single-player action (Assassin's Creed) | Optional | Random enemy spawns OK |
|
|
| Roguelike (Hades) | Optional | Randomness is feature |
|
|
|
|
**Cost of Determinism:**
|
|
- Can't use platform math libraries (must use fixed-point or cross-platform lib)
|
|
- Can't use multithreading easily (need deterministic parallel algorithms)
|
|
- More testing required (verify on all platforms)
|
|
- Development time: +20-30%
|
|
|
|
**Benefit of Determinism:**
|
|
- Multiplayer: Minimal bandwidth (send inputs, not state)
|
|
- Replays: 100KB instead of 1GB video
|
|
- Debugging: Reproducible bugs, not "random" failures
|
|
- Testing: Unit tests always pass/fail consistently
|
|
|
|
### Framework 2: Replay System Complexity Level
|
|
|
|
**Question:** How sophisticated should my replay system be?
|
|
|
|
**Level 1: Minimal (Emergency Bug Reproduction)**
|
|
```python
|
|
# Just record inputs, no UI
|
|
class MinimalReplay:
|
|
def __init__(self):
|
|
self.inputs = []
|
|
|
|
def record(self, tick, action, data):
|
|
self.inputs.append((tick, action, data))
|
|
|
|
def save(self, filename):
|
|
import json
|
|
with open(filename, 'w') as f:
|
|
json.dump(self.inputs, f)
|
|
```
|
|
**Time to implement:** 2-4 hours
|
|
**Use when:** Need to reproduce a bug ASAP
|
|
|
|
**Level 2: Basic (Playback + Validation)**
|
|
```python
|
|
# Add playback, checksums, validation
|
|
class BasicReplay:
|
|
def record_with_checksum(self, tick, action, data, state_hash):
|
|
self.inputs.append({
|
|
'tick': tick,
|
|
'action': action,
|
|
'data': data,
|
|
'checksum': state_hash # Verify replay matches
|
|
})
|
|
```
|
|
**Time to implement:** 1-2 days
|
|
**Use when:** Need replays for testing, regression testing
|
|
|
|
**Level 3: Advanced (UI + Scrubbing + Debugging)**
|
|
```python
|
|
# Add UI, frame-by-frame, jump to tick
|
|
class AdvancedReplay:
|
|
def __init__(self):
|
|
self.snapshots = {} # Save state every N ticks
|
|
self.current_tick = 0
|
|
|
|
def jump_to_tick(self, target_tick):
|
|
# Find nearest snapshot before target
|
|
snapshot_tick = max([t for t in self.snapshots.keys() if t <= target_tick])
|
|
self.restore_snapshot(snapshot_tick)
|
|
|
|
# Simulate forward to exact tick
|
|
while self.current_tick < target_tick:
|
|
self.step()
|
|
|
|
def step_forward(self):
|
|
"""Advance one tick"""
|
|
self.current_tick += 1
|
|
self.simulate_tick()
|
|
|
|
def step_backward(self):
|
|
"""Go back one tick (restore from snapshot + simulate)"""
|
|
self.jump_to_tick(self.current_tick - 1)
|
|
```
|
|
**Time to implement:** 1-2 weeks
|
|
**Use when:** Complex debugging needs, esports replays, player-facing replays
|
|
|
|
**Decision:**
|
|
- Critical bug, time-constrained → Level 1 (minimal)
|
|
- Ongoing development, need testing → Level 2 (basic)
|
|
- Shipped game, player replays → Level 3 (advanced)
|
|
|
|
### Framework 3: Debug Visualization Strategy
|
|
|
|
**Question:** What should I visualize for this bug?
|
|
|
|
**Debugging Workflow:**
|
|
```
|
|
1. Understand symptoms
|
|
├─ Desync? → Visualize divergence points
|
|
├─ Crash/instability? → Visualize state over time
|
|
├─ Wrong behavior? → Visualize entity decisions
|
|
└─ Performance? → Visualize performance metrics
|
|
|
|
2. Choose visualizations
|
|
├─ Spatial bugs → Debug draw (positions, paths, ranges)
|
|
├─ Temporal bugs → Graphs (populations, resources over time)
|
|
├─ Logic bugs → State inspector (entity internal state)
|
|
└─ Comparison → Side-by-side diff (two clients)
|
|
|
|
3. Iterate
|
|
└─ Add more detail to narrow down problem
|
|
```
|
|
|
|
**Visualization Types:**
|
|
|
|
| Bug Type | Visualization | Example |
|
|
|----------|---------------|---------|
|
|
| Desync | State checksums, comparison table | "Tick 500: Client A has 50 deer, Client B has 52 deer" |
|
|
| Population collapse | Population graph | Graph shows deer plummet from 100 to 0 at tick 345 |
|
|
| Pathfinding | Debug draw lines | Show entity path, target destination, obstacles |
|
|
| Energy/resources | Bar charts over entities | Show energy bars above each deer |
|
|
| Collision | Debug circles | Show collision radius, overlap detection |
|
|
| Chaos/butterfly | Sensitivity graph | "0.01% input change → 15% output change" |
|
|
|
|
**Implementation Priority:**
|
|
1. **State checksums** (detect problem exists) - 1 hour
|
|
2. **Population graphs** (see trends over time) - 2 hours
|
|
3. **Debug draw** (see spatial relationships) - 4 hours
|
|
4. **State inspector** (examine individual entities) - 4 hours
|
|
5. **Comparison diff** (find exact differences) - 8 hours
|
|
|
|
|
|
## Implementation Patterns
|
|
|
|
### Pattern 1: Complete Deterministic Simulation (Production-Ready)
|
|
|
|
```python
|
|
import hashlib
|
|
import json
|
|
import random
|
|
import copy
|
|
|
|
class SeededRandom:
|
|
"""Deterministic RNG - same seed → same sequence"""
|
|
def __init__(self, seed):
|
|
self.rng = random.Random(seed)
|
|
|
|
def random(self):
|
|
return self.rng.random()
|
|
|
|
def randint(self, a, b):
|
|
return self.rng.randint(a, b)
|
|
|
|
def choice(self, sequence):
|
|
return self.rng.choice(sequence)
|
|
|
|
class Entity:
|
|
"""Base entity with deterministic ID"""
|
|
_next_id = 0
|
|
|
|
def __init__(self, x, y, energy):
|
|
self.id = Entity._next_id
|
|
Entity._next_id += 1
|
|
self.x = x
|
|
self.y = y
|
|
self.energy = energy
|
|
self.alive = True
|
|
|
|
def to_dict(self):
|
|
"""Serialize for checksum"""
|
|
return {
|
|
'id': self.id,
|
|
'x': self.x,
|
|
'y': self.y,
|
|
'energy': self.energy,
|
|
'alive': self.alive
|
|
}
|
|
|
|
class Deer(Entity):
|
|
def __init__(self, x, y):
|
|
super().__init__(x, y, energy=100.0)
|
|
self.reproduction_cooldown = 0.0
|
|
|
|
def update(self, dt, rng, grass_amount):
|
|
"""Deterministic update"""
|
|
# Deterministic energy consumption
|
|
self.energy -= 3.0 * dt
|
|
self.reproduction_cooldown = max(0, self.reproduction_cooldown - dt)
|
|
|
|
# Deterministic eating
|
|
if grass_amount > 10:
|
|
eat_amount = min(20.0, grass_amount)
|
|
self.energy = min(100.0, self.energy + eat_amount * 0.5)
|
|
grass_consumed = eat_amount
|
|
else:
|
|
grass_consumed = 0
|
|
|
|
# Deterministic reproduction
|
|
can_reproduce = (
|
|
self.energy > 80 and
|
|
self.reproduction_cooldown == 0 and
|
|
rng.random() < 0.1 # Seeded randomness
|
|
)
|
|
|
|
if can_reproduce:
|
|
self.energy -= 30.0
|
|
self.reproduction_cooldown = 20.0
|
|
return Deer(self.x + rng.random() * 4 - 2, self.y + rng.random() * 4 - 2), grass_consumed
|
|
|
|
# Death from starvation
|
|
if self.energy <= 0:
|
|
self.alive = False
|
|
|
|
return None, grass_consumed
|
|
|
|
class DeterministicSimulation:
|
|
def __init__(self, seed):
|
|
"""Initialize with seed for determinism"""
|
|
self.seed = seed
|
|
self.rng = SeededRandom(seed)
|
|
self.tick = 0
|
|
self.grass = 1000.0
|
|
self.deer = []
|
|
self.wolves = []
|
|
|
|
# Deterministic initialization
|
|
for i in range(10):
|
|
self.deer.append(Deer(x=float(i * 10), y=0.0))
|
|
|
|
# State tracking
|
|
self.checksum_history = []
|
|
|
|
def step(self, dt=1.0):
|
|
"""One simulation step - MUST be deterministic"""
|
|
self.tick += 1
|
|
|
|
# 1. Grass growth (deterministic)
|
|
self.grass += 5.0 * dt
|
|
self.grass = min(1000.0, self.grass)
|
|
|
|
# 2. Update entities in deterministic order (sorted by ID)
|
|
self.deer.sort(key=lambda d: d.id)
|
|
|
|
new_deer = []
|
|
total_grass_consumed = 0.0
|
|
|
|
for deer in self.deer[:]: # Copy list to avoid mutation during iteration
|
|
if not deer.alive:
|
|
continue
|
|
|
|
baby, grass_consumed = deer.update(dt, self.rng, self.grass)
|
|
total_grass_consumed += grass_consumed
|
|
|
|
if baby:
|
|
new_deer.append(baby)
|
|
|
|
# Apply grass consumption
|
|
self.grass -= total_grass_consumed
|
|
self.grass = max(0.0, self.grass)
|
|
|
|
# Add babies
|
|
self.deer.extend(new_deer)
|
|
|
|
# Remove dead deer
|
|
self.deer = [d for d in self.deer if d.alive]
|
|
|
|
# 3. Compute checksum
|
|
checksum = self.compute_checksum()
|
|
self.checksum_history.append((self.tick, checksum))
|
|
|
|
return checksum
|
|
|
|
def compute_checksum(self):
|
|
"""Hash simulation state for desync detection"""
|
|
state = {
|
|
'tick': self.tick,
|
|
'grass': self.grass,
|
|
'deer': sorted([d.to_dict() for d in self.deer], key=lambda x: x['id'])
|
|
}
|
|
state_json = json.dumps(state, sort_keys=True)
|
|
return hashlib.sha256(state_json.encode()).hexdigest()
|
|
|
|
def get_state(self):
|
|
"""Get complete simulation state (for snapshots)"""
|
|
return {
|
|
'seed': self.seed,
|
|
'tick': self.tick,
|
|
'grass': self.grass,
|
|
'deer': [copy.deepcopy(d.to_dict()) for d in self.deer],
|
|
'rng_state': self.rng.rng.getstate()
|
|
}
|
|
|
|
def set_state(self, state):
|
|
"""Restore simulation state (for replay/debugging)"""
|
|
self.seed = state['seed']
|
|
self.tick = state['tick']
|
|
self.grass = state['grass']
|
|
|
|
# Restore entities
|
|
Entity._next_id = max([d['id'] for d in state['deer']] + [0]) + 1
|
|
self.deer = []
|
|
for deer_data in state['deer']:
|
|
deer = Deer(deer_data['x'], deer_data['y'])
|
|
deer.id = deer_data['id']
|
|
deer.energy = deer_data['energy']
|
|
deer.alive = deer_data['alive']
|
|
self.deer.append(deer)
|
|
|
|
# Restore RNG state
|
|
self.rng.rng.setstate(state['rng_state'])
|
|
|
|
# TESTING DETERMINISM
|
|
def test_determinism():
|
|
"""Verify simulation is deterministic"""
|
|
print("Testing determinism...")
|
|
|
|
# Run two simulations with same seed
|
|
sim1 = DeterministicSimulation(seed=42)
|
|
sim2 = DeterministicSimulation(seed=42)
|
|
|
|
for i in range(100):
|
|
checksum1 = sim1.step()
|
|
checksum2 = sim2.step()
|
|
|
|
if checksum1 != checksum2:
|
|
print(f"❌ DESYNC at tick {sim1.tick}!")
|
|
print(f" Sim1 checksum: {checksum1[:16]}")
|
|
print(f" Sim2 checksum: {checksum2[:16]}")
|
|
return False
|
|
|
|
print(f"✅ Determinism verified - 100 ticks identical")
|
|
return True
|
|
|
|
# Run test
|
|
test_determinism()
|
|
```
|
|
|
|
**Key Features:**
|
|
- ✅ Seeded RNG (same seed → same random values)
|
|
- ✅ Deterministic entity IDs
|
|
- ✅ Sorted iteration (consistent order)
|
|
- ✅ Fixed-point friendly (uses float but can be replaced)
|
|
- ✅ State checksums (detect desyncs)
|
|
- ✅ State save/load (snapshots for replay)
|
|
- ✅ Unit test (verify determinism)
|
|
|
|
### Pattern 2: Replay System with Validation
|
|
|
|
```python
|
|
class ReplaySystem:
|
|
def __init__(self):
|
|
self.events = []
|
|
self.snapshots = {}
|
|
self.is_recording = True
|
|
self.is_replaying = False
|
|
self.replay_index = 0
|
|
self.initial_state = None
|
|
|
|
def start_recording(self, simulation):
|
|
"""Start recording from current state"""
|
|
self.is_recording = True
|
|
self.is_replaying = False
|
|
self.events = []
|
|
self.initial_state = simulation.get_state()
|
|
|
|
def record_event(self, tick, event_type, event_data, state_checksum=None):
|
|
"""Record an event with optional checksum"""
|
|
if not self.is_recording:
|
|
return
|
|
|
|
self.events.append({
|
|
'tick': tick,
|
|
'type': event_type,
|
|
'data': event_data,
|
|
'checksum': state_checksum
|
|
})
|
|
|
|
def save_snapshot(self, tick, simulation):
|
|
"""Save state snapshot for fast seeking"""
|
|
self.snapshots[tick] = simulation.get_state()
|
|
|
|
def start_replay(self):
|
|
"""Start replaying recorded events"""
|
|
self.is_recording = False
|
|
self.is_replaying = True
|
|
self.replay_index = 0
|
|
|
|
def get_events_for_tick(self, tick):
|
|
"""Get all events for this tick"""
|
|
events = []
|
|
while self.replay_index < len(self.events):
|
|
event = self.events[self.replay_index]
|
|
if event['tick'] == tick:
|
|
events.append(event)
|
|
self.replay_index += 1
|
|
elif event['tick'] > tick:
|
|
break
|
|
else:
|
|
self.replay_index += 1
|
|
return events
|
|
|
|
def validate_replay(self, simulation):
|
|
"""Verify replay matches original"""
|
|
print("Validating replay...")
|
|
|
|
# Restore initial state
|
|
simulation.set_state(self.initial_state)
|
|
|
|
# Replay all events
|
|
for tick in range(max([e['tick'] for e in self.events]) + 1):
|
|
events = self.get_events_for_tick(tick)
|
|
for event in events:
|
|
# Apply event
|
|
if event['type'] == 'player_hunt':
|
|
simulation.hunt_deer(event['data']['deer_id'])
|
|
|
|
simulation.step()
|
|
|
|
# Validate checksum
|
|
current_checksum = simulation.compute_checksum()
|
|
expected_checksum = next((e['checksum'] for e in self.events if e['tick'] == tick and e['checksum']), None)
|
|
|
|
if expected_checksum and current_checksum != expected_checksum:
|
|
print(f"❌ Replay diverged at tick {tick}")
|
|
print(f" Expected: {expected_checksum[:16]}")
|
|
print(f" Got: {current_checksum[:16]}")
|
|
return False
|
|
|
|
print("✅ Replay validated successfully")
|
|
return True
|
|
|
|
def save(self, filename):
|
|
"""Save replay to file"""
|
|
with open(filename, 'w') as f:
|
|
json.dump({
|
|
'version': 1,
|
|
'initial_state': self.initial_state,
|
|
'events': self.events,
|
|
'snapshots': self.snapshots
|
|
}, f, indent=2)
|
|
|
|
def load(self, filename):
|
|
"""Load replay from file"""
|
|
with open(filename, 'r') as f:
|
|
data = json.load(f)
|
|
self.initial_state = data['initial_state']
|
|
self.events = data['events']
|
|
self.snapshots = data.get('snapshots', {})
|
|
|
|
# USAGE EXAMPLE
|
|
replay = ReplaySystem()
|
|
sim = DeterministicSimulation(seed=12345)
|
|
|
|
# Start recording
|
|
replay.start_recording(sim)
|
|
|
|
# Simulate gameplay
|
|
for tick in range(100):
|
|
sim.step()
|
|
checksum = sim.compute_checksum()
|
|
replay.record_event(tick, 'tick', {}, state_checksum=checksum)
|
|
|
|
# Save snapshot every 10 ticks
|
|
if tick % 10 == 0:
|
|
replay.save_snapshot(tick, sim)
|
|
|
|
# Save replay
|
|
replay.save('test_replay.json')
|
|
|
|
# Validate replay
|
|
replay.start_replay()
|
|
sim2 = DeterministicSimulation(seed=12345)
|
|
replay.validate_replay(sim2)
|
|
```
|
|
|
|
### Pattern 3: Debug Visualization Suite
|
|
|
|
```python
|
|
class DebugVisualization:
|
|
def __init__(self):
|
|
self.enabled = True
|
|
self.draw_commands = []
|
|
self.graphs = {}
|
|
|
|
# Debug Draw
|
|
def draw_circle(self, x, y, radius, color, label=None):
|
|
"""Draw debug circle"""
|
|
if not self.enabled:
|
|
return
|
|
self.draw_commands.append({
|
|
'type': 'circle',
|
|
'x': x, 'y': y,
|
|
'radius': radius,
|
|
'color': color,
|
|
'label': label
|
|
})
|
|
|
|
def draw_line(self, x1, y1, x2, y2, color, width=1):
|
|
"""Draw debug line"""
|
|
if not self.enabled:
|
|
return
|
|
self.draw_commands.append({
|
|
'type': 'line',
|
|
'start': (x1, y1),
|
|
'end': (x2, y2),
|
|
'color': color,
|
|
'width': width
|
|
})
|
|
|
|
def draw_text(self, x, y, text, color='white', size=12):
|
|
"""Draw debug text"""
|
|
if not self.enabled:
|
|
return
|
|
self.draw_commands.append({
|
|
'type': 'text',
|
|
'x': x, 'y': y,
|
|
'text': str(text),
|
|
'color': color,
|
|
'size': size
|
|
})
|
|
|
|
# Graphing
|
|
def init_graph(self, graph_name, series_names):
|
|
"""Initialize a time-series graph"""
|
|
self.graphs[graph_name] = {
|
|
'series': {name: [] for name in series_names},
|
|
'ticks': []
|
|
}
|
|
|
|
def record_values(self, graph_name, tick, **values):
|
|
"""Record values for graph"""
|
|
if graph_name not in self.graphs:
|
|
return
|
|
|
|
graph = self.graphs[graph_name]
|
|
graph['ticks'].append(tick)
|
|
|
|
for series_name, value in values.items():
|
|
if series_name in graph['series']:
|
|
graph['series'][series_name].append(value)
|
|
|
|
def plot_graph(self, graph_name):
|
|
"""Plot graph using matplotlib"""
|
|
import matplotlib.pyplot as plt
|
|
|
|
if graph_name not in self.graphs:
|
|
return
|
|
|
|
graph = self.graphs[graph_name]
|
|
|
|
plt.figure(figsize=(12, 6))
|
|
for series_name, values in graph['series'].items():
|
|
plt.plot(graph['ticks'], values, label=series_name)
|
|
|
|
plt.xlabel('Tick')
|
|
plt.ylabel('Value')
|
|
plt.title(f'{graph_name} Over Time')
|
|
plt.legend()
|
|
plt.grid(True)
|
|
plt.show()
|
|
|
|
# State Inspection
|
|
def inspect_entity(self, entity, x, y):
|
|
"""Show entity state on screen"""
|
|
self.draw_circle(entity.x, entity.y, radius=2, color='yellow')
|
|
|
|
state_text = f"ID: {entity.id}\n"
|
|
state_text += f"Energy: {entity.energy:.1f}\n"
|
|
state_text += f"Pos: ({entity.x:.1f}, {entity.y:.1f})"
|
|
|
|
self.draw_text(x, y, state_text, color='white', size=10)
|
|
|
|
# Comparison
|
|
def compare_states(self, state_a, state_b, label_a="Client A", label_b="Client B"):
|
|
"""Compare two simulation states"""
|
|
print(f"\n=== State Comparison: {label_a} vs {label_b} ===")
|
|
|
|
# Compare deer counts
|
|
deer_a = len(state_a['deer'])
|
|
deer_b = len(state_b['deer'])
|
|
|
|
if deer_a != deer_b:
|
|
print(f"❌ Deer count: {deer_a} vs {deer_b} (diff: {abs(deer_a - deer_b)})")
|
|
else:
|
|
print(f"✅ Deer count: {deer_a} (identical)")
|
|
|
|
# Compare grass
|
|
grass_a = state_a['grass']
|
|
grass_b = state_b['grass']
|
|
grass_diff = abs(grass_a - grass_b)
|
|
|
|
if grass_diff > 0.01:
|
|
print(f"❌ Grass: {grass_a:.2f} vs {grass_b:.2f} (diff: {grass_diff:.2f})")
|
|
else:
|
|
print(f"✅ Grass: {grass_a:.2f} (identical)")
|
|
|
|
# Compare individual deer
|
|
deer_ids_a = set(d['id'] for d in state_a['deer'])
|
|
deer_ids_b = set(d['id'] for d in state_b['deer'])
|
|
|
|
missing_in_b = deer_ids_a - deer_ids_b
|
|
missing_in_a = deer_ids_b - deer_ids_a
|
|
|
|
if missing_in_b:
|
|
print(f"❌ Deer only in {label_a}: {missing_in_b}")
|
|
if missing_in_a:
|
|
print(f"❌ Deer only in {label_b}: {missing_in_a}")
|
|
|
|
# Compare common deer
|
|
common_ids = deer_ids_a & deer_ids_b
|
|
for deer_id in common_ids:
|
|
deer_a = next(d for d in state_a['deer'] if d['id'] == deer_id)
|
|
deer_b = next(d for d in state_b['deer'] if d['id'] == deer_id)
|
|
|
|
energy_diff = abs(deer_a['energy'] - deer_b['energy'])
|
|
if energy_diff > 0.01:
|
|
print(f"❌ Deer {deer_id} energy: {deer_a['energy']:.2f} vs {deer_b['energy']:.2f}")
|
|
|
|
def clear(self):
|
|
"""Clear debug draw commands"""
|
|
self.draw_commands = []
|
|
|
|
# USAGE IN SIMULATION
|
|
debug = DebugVisualization()
|
|
|
|
# Initialize graphs
|
|
debug.init_graph('population', ['deer', 'wolves', 'grass'])
|
|
|
|
# During simulation
|
|
for tick in range(1000):
|
|
sim.step()
|
|
|
|
# Record for graph
|
|
debug.record_values('population', tick,
|
|
deer=len(sim.deer),
|
|
wolves=len(sim.wolves),
|
|
grass=sim.grass)
|
|
|
|
# Debug draw
|
|
for deer in sim.deer:
|
|
debug.draw_circle(deer.x, deer.y, radius=2, color='green')
|
|
debug.draw_text(deer.x, deer.y + 3, f"E:{deer.energy:.0f}", color='white')
|
|
|
|
# Clear for next frame
|
|
debug.clear()
|
|
|
|
# Plot graph
|
|
debug.plot_graph('population')
|
|
|
|
# Compare states
|
|
state1 = sim1.get_state()
|
|
state2 = sim2.get_state()
|
|
debug.compare_states(state1, state2, "Client 1", "Client 2")
|
|
```
|
|
|
|
### Pattern 4: Assertion Framework (Fail Fast)
|
|
|
|
```python
|
|
class SimulationAssertions:
|
|
"""Runtime checks to catch bugs immediately"""
|
|
|
|
@staticmethod
|
|
def assert_positive(value, name):
|
|
"""Assert value is positive"""
|
|
assert value >= 0, f"{name} must be >= 0, got {value}"
|
|
|
|
@staticmethod
|
|
def assert_in_range(value, min_val, max_val, name):
|
|
"""Assert value is in range"""
|
|
assert min_val <= value <= max_val, f"{name} must be in [{min_val}, {max_val}], got {value}"
|
|
|
|
@staticmethod
|
|
def assert_energy_valid(entity):
|
|
"""Assert entity energy is valid"""
|
|
assert 0 <= entity.energy <= 100, f"Entity {entity.id} energy invalid: {entity.energy}"
|
|
|
|
@staticmethod
|
|
def assert_population_bounded(simulation, max_deer, max_wolves):
|
|
"""Assert populations don't explode"""
|
|
deer_count = len(simulation.deer)
|
|
wolf_count = len(simulation.wolves)
|
|
|
|
assert deer_count <= max_deer, f"Deer explosion: {deer_count} > {max_deer}"
|
|
assert wolf_count <= max_wolves, f"Wolf explosion: {wolf_count} > {max_wolves}"
|
|
|
|
@staticmethod
|
|
def assert_no_duplicates(entities):
|
|
"""Assert no duplicate entity IDs"""
|
|
ids = [e.id for e in entities]
|
|
assert len(ids) == len(set(ids)), f"Duplicate entity IDs found: {ids}"
|
|
|
|
@staticmethod
|
|
def assert_checksum_matches(simulation, expected_checksum):
|
|
"""Assert state checksum matches expected"""
|
|
actual = simulation.compute_checksum()
|
|
assert actual == expected_checksum, f"Checksum mismatch!\n Expected: {expected_checksum[:16]}\n Got: {actual[:16]}"
|
|
|
|
# USE IN SIMULATION
|
|
class AssertingSimulation(DeterministicSimulation):
|
|
def step(self, dt=1.0):
|
|
"""Step with assertions"""
|
|
|
|
# Pre-conditions
|
|
SimulationAssertions.assert_positive(self.grass, "grass")
|
|
SimulationAssertions.assert_no_duplicates(self.deer)
|
|
|
|
# Run simulation
|
|
result = super().step(dt)
|
|
|
|
# Post-conditions
|
|
SimulationAssertions.assert_population_bounded(self, max_deer=1000, max_wolves=200)
|
|
|
|
for deer in self.deer:
|
|
SimulationAssertions.assert_energy_valid(deer)
|
|
|
|
return result
|
|
|
|
# TESTING WITH ASSERTIONS
|
|
sim = AssertingSimulation(seed=42)
|
|
for tick in range(1000):
|
|
sim.step() # Fails immediately if any assertion violated
|
|
```
|
|
|
|
### Pattern 5: Property-Based Testing
|
|
|
|
```python
|
|
# Requires: pip install hypothesis
|
|
from hypothesis import given, strategies as st
|
|
import hypothesis
|
|
|
|
class PropertyTests:
|
|
"""Property-based tests for simulation invariants"""
|
|
|
|
@staticmethod
|
|
@given(seed=st.integers(min_value=0, max_value=1000000))
|
|
def test_determinism_property(seed):
|
|
"""Property: Same seed always produces same result"""
|
|
sim1 = DeterministicSimulation(seed=seed)
|
|
sim2 = DeterministicSimulation(seed=seed)
|
|
|
|
for _ in range(10):
|
|
checksum1 = sim1.step()
|
|
checksum2 = sim2.step()
|
|
assert checksum1 == checksum2, "Determinism violated"
|
|
|
|
@staticmethod
|
|
@given(seed=st.integers(min_value=0, max_value=1000000))
|
|
def test_energy_conservation_property(seed):
|
|
"""Property: Total energy never increases without input"""
|
|
sim = DeterministicSimulation(seed=seed)
|
|
|
|
for _ in range(100):
|
|
energy_before = sum(d.energy for d in sim.deer) + sim.grass
|
|
sim.step()
|
|
energy_after = sum(d.energy for d in sim.deer) + sim.grass
|
|
|
|
# Energy can only decrease (metabolism) or stay same (eating just moves it)
|
|
assert energy_after <= energy_before + 10, "Energy created from nothing!"
|
|
|
|
@staticmethod
|
|
@given(seed=st.integers(min_value=0, max_value=1000000))
|
|
def test_population_bounded_property(seed):
|
|
"""Property: Populations stay within reasonable bounds"""
|
|
sim = DeterministicSimulation(seed=seed)
|
|
|
|
for _ in range(100):
|
|
sim.step()
|
|
assert len(sim.deer) <= 1000, "Deer population explosion"
|
|
assert len(sim.deer) >= 0, "Negative deer population"
|
|
|
|
@staticmethod
|
|
def test_replay_reproducibility_property():
|
|
"""Property: Replay always produces identical result"""
|
|
sim = DeterministicSimulation(seed=123)
|
|
replay = ReplaySystem()
|
|
replay.start_recording(sim)
|
|
|
|
# Run simulation
|
|
for tick in range(50):
|
|
sim.step()
|
|
replay.record_event(tick, 'tick', {}, state_checksum=sim.compute_checksum())
|
|
|
|
# Replay should be identical
|
|
assert replay.validate_replay(DeterministicSimulation(seed=123))
|
|
|
|
# RUN PROPERTY TESTS
|
|
if __name__ == '__main__':
|
|
PropertyTests.test_determinism_property()
|
|
PropertyTests.test_energy_conservation_property()
|
|
PropertyTests.test_population_bounded_property()
|
|
PropertyTests.test_replay_reproducibility_property()
|
|
print("✅ All property tests passed")
|
|
```
|
|
|
|
|
|
## Common Pitfalls
|
|
|
|
### Pitfall 1: Unseeded Random Number Generation
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ Uses system entropy - different on each client
|
|
import random
|
|
if random.random() < 0.5:
|
|
spawn_deer()
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- Each client generates different random numbers
|
|
- Multiplayer: Instant desync
|
|
- Testing: Bug not reproducible (different each run)
|
|
|
|
**Real Example:**
|
|
Multiplayer ecosystem - Client A rolls 0.48 (spawns deer), Client B rolls 0.52 (doesn't spawn). Now Client A has 51 deer, Client B has 50. Error compounds over time.
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Seeded RNG - same seed produces same sequence
|
|
class SeededRandom:
|
|
def __init__(self, seed):
|
|
self.rng = random.Random(seed)
|
|
|
|
def random(self):
|
|
return self.rng.random()
|
|
|
|
# All clients use same seed
|
|
rng = SeededRandom(seed=12345)
|
|
if rng.random() < 0.5:
|
|
spawn_deer()
|
|
```
|
|
|
|
### Pitfall 2: Map/Dictionary Iteration Order
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ Iteration order is undefined (Python < 3.7, JavaScript, C++)
|
|
entities = {'deer_1': deer1, 'wolf_2': wolf2, 'deer_3': deer3}
|
|
for entity_id, entity in entities.items():
|
|
entity.update() # ORDER VARIES!
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- Iteration order affects which entity updates first
|
|
- If entity1 eats all food before entity2 can, order matters
|
|
- Multiplayer: Clients iterate in different orders → desync
|
|
|
|
**Real Example:**
|
|
Deer A and Deer B both targeting same grass patch. On Client 1, Deer A updates first (eats grass). On Client 2, Deer B updates first (eats grass). Now Deer A and Deer B have different energy on each client.
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Sort keys before iterating
|
|
for entity_id in sorted(entities.keys()):
|
|
entities[entity_id].update()
|
|
```
|
|
|
|
### Pitfall 3: Floating Point Accumulation
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ Accumulating floating point creates divergence
|
|
position = 0.0
|
|
for _ in range(10000):
|
|
position += 0.1 # Rounding errors compound!
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- `0.1` cannot be represented exactly in binary floating point
|
|
- Error: ~0.0000000000000001 per addition
|
|
- After 10,000 additions: ~0.001 total error
|
|
- Multiplayer: Different CPU architectures round differently
|
|
|
|
**Real Example:**
|
|
After 1 hour of gameplay (360,000 frames at 60 FPS):
|
|
- Client A (x86): deer position = 1234.567
|
|
- Client B (ARM): deer position = 1234.571
|
|
- 0.004 difference → deer on different tiles → different grass eaten → DESYNC
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Use fixed-point or integer math for critical values
|
|
position_int = 0 # Store as integer (units = 0.001)
|
|
for _ in range(10000):
|
|
position_int += 100 # 0.1 * 1000 = 100
|
|
position = position_int / 1000.0 # Convert to float for display
|
|
```
|
|
|
|
Or use integer positions:
|
|
```python
|
|
# Position in millimeters instead of meters
|
|
x_mm = 5000 # 5 meters
|
|
y_mm = 3000 # 3 meters
|
|
|
|
# No floating point errors
|
|
x_mm += velocity_mm_per_tick
|
|
```
|
|
|
|
### Pitfall 4: Using Wall-Clock Time
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ Uses real-world time - varies between clients
|
|
import time
|
|
current_time = time.time()
|
|
if current_time % 60 < 30:
|
|
spawn_enemy()
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- Clients start at different real-world times
|
|
- Network latency causes time skew
|
|
- Frame rate differences mean different number of checks
|
|
|
|
**Real Example:**
|
|
- Client A checks at 14:30:25 → time % 60 = 25 → spawns enemy
|
|
- Client B checks at 14:30:26 (1 second lag) → time % 60 = 26 → spawns enemy
|
|
- Now clients have enemies at different positions
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Use simulation ticks (deterministic)
|
|
self.tick += 1
|
|
if self.tick % 600 == 0: # Every 600 ticks
|
|
spawn_enemy()
|
|
```
|
|
|
|
### Pitfall 5: No Replay System (Can't Reproduce Bugs)
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ No recording - bug happens once, can't debug
|
|
def run_simulation():
|
|
for tick in range(10000):
|
|
simulate_tick()
|
|
# Bug happened at tick 7345, but you don't know that
|
|
# And you can't reproduce it
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- Can't reproduce bug to debug
|
|
- Can't verify fix actually works
|
|
- Can't create regression tests
|
|
- Can't analyze what happened
|
|
|
|
**Real Example:**
|
|
"Deer population goes to zero after 30 minutes" - without replay, you don't know:
|
|
- Which tick did first deer die?
|
|
- What was energy level before death?
|
|
- Was there grass available?
|
|
- What were other deer doing?
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Record inputs for replay
|
|
replay = ReplaySystem()
|
|
replay.start_recording(sim)
|
|
|
|
for tick in range(10000):
|
|
checksum = simulate_tick()
|
|
replay.record_event(tick, 'tick', {}, checksum=checksum)
|
|
|
|
# Save when bug occurs
|
|
if bug_detected():
|
|
replay.save('bug_7345.replay')
|
|
# Now you can replay and debug at exactly tick 7345
|
|
```
|
|
|
|
### Pitfall 6: No Debug Visualization (Debugging Blind)
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ No visualization - can't see what's happening
|
|
simulation.step()
|
|
# Is deer moving toward grass or away?
|
|
# Is wolf actually chasing deer?
|
|
# Why did population spike?
|
|
# NO IDEA - just looking at numbers
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- Can't see spatial relationships
|
|
- Can't see trends over time
|
|
- Can't spot anomalies visually
|
|
- Waste hours guessing
|
|
|
|
**Real Example:**
|
|
"Deer population oscillates weirdly" - without graph, you don't see:
|
|
- Oscillation period getting shorter (approaching chaos)
|
|
- Oscillation amplitude growing (instability)
|
|
- Sudden spike at tick 500 (bug trigger)
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Visualize everything
|
|
debug = DebugVisualization()
|
|
|
|
# Draw entities
|
|
for deer in simulation.deer:
|
|
debug.draw_circle(deer.x, deer.y, radius=2, color='green')
|
|
debug.draw_text(deer.x, deer.y + 3, f"E:{deer.energy:.0f}")
|
|
|
|
# Graph populations
|
|
debug.record_values('population', tick, deer=len(simulation.deer))
|
|
debug.plot_graph('population') # See trends immediately
|
|
```
|
|
|
|
### Pitfall 7: No Assertions (Silent Failures)
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ Bug happens, simulation keeps running with corrupted state
|
|
deer.energy = -50 # INVALID but no error
|
|
simulation.deer.append(existing_deer) # DUPLICATE but no error
|
|
# Corrupted state → weird behavior 1000 ticks later → impossible to debug
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- Bug symptoms appear far from root cause
|
|
- Corrupted state compounds over time
|
|
- Hard to trace back to original error
|
|
|
|
**Real Example:**
|
|
Tick 100: Deer energy goes negative (bug)
|
|
Tick 500: Negative energy deer reproduces (shouldn't happen)
|
|
Tick 1000: Population explosion from immortal deer (visible symptom)
|
|
Without assertion, you debug tick 1000, never find root cause at tick 100.
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Fail fast with assertions
|
|
deer.energy -= consumption
|
|
assert deer.energy >= 0, f"Deer {deer.id} energy negative: {deer.energy}"
|
|
# Fails immediately at tick 100, shows exact problem
|
|
```
|
|
|
|
### Pitfall 8: Testing Only Happy Path (Missing Edge Cases)
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ Only test normal conditions
|
|
def test_ecosystem():
|
|
sim = Simulation()
|
|
sim.step() # With 10 deer, 2 wolves
|
|
assert len(sim.deer) > 0 # Passes!
|
|
|
|
# But never test:
|
|
# - What if 0 deer?
|
|
# - What if 1000 deer?
|
|
# - What if 0 grass?
|
|
# - What if deer.energy = 0.0001?
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- Edge cases trigger bugs
|
|
- Integer overflow at extreme values
|
|
- Division by zero when populations hit zero
|
|
- Floating point precision at very small values
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Test edge cases
|
|
def test_edge_cases():
|
|
# Zero population
|
|
sim = Simulation()
|
|
sim.deer = []
|
|
sim.step() # Should not crash
|
|
|
|
# Huge population
|
|
sim.deer = [Deer() for _ in range(10000)]
|
|
sim.step() # Should not explode
|
|
|
|
# Extreme values
|
|
deer = Deer()
|
|
deer.energy = 0.0001 # Nearly dead
|
|
deer.update(dt=1.0) # Should handle gracefully
|
|
```
|
|
|
|
### Pitfall 9: Not Understanding Butterfly Effects
|
|
|
|
**The Mistake:**
|
|
```python
|
|
# ❌ Thinks "close enough" is fine
|
|
if abs(deer_count_a - deer_count_b) < 5:
|
|
print("Practically the same!") # WRONG!
|
|
|
|
# Small difference compounds exponentially
|
|
```
|
|
|
|
**Why This Fails:**
|
|
- Chaotic systems amplify small differences
|
|
- 0.1% error at tick 0 → 10% error at tick 1000
|
|
- Feedback loops compound errors
|
|
- "Close" is not deterministic
|
|
|
|
**Real Example:**
|
|
Tick 0: 100 deer vs 100.01 deer (0.01% difference)
|
|
Tick 100: 150 deer vs 151 deer (0.7% difference - grew 70x)
|
|
Tick 500: 300 deer vs 325 deer (8% difference - grew 800x)
|
|
Tick 1000: 450 deer vs 520 deer (15% difference - grew 1500x)
|
|
|
|
**The Fix:**
|
|
```python
|
|
# ✅ Require exact match
|
|
assert deer_count_a == deer_count_b, "Must be EXACTLY equal"
|
|
# Or checksums
|
|
assert checksum_a == checksum_b, "States must be bit-for-bit identical"
|
|
```
|
|
|
|
|
|
## Real-World Examples
|
|
|
|
### Example 1: Rocket League - Replay System
|
|
|
|
**Architecture:** Deterministic physics + input recording
|
|
|
|
**How It Works:**
|
|
- Game records: Player inputs, initial ball/car positions, RNG seed
|
|
- File size: ~100 KB for 5-minute match (tiny!)
|
|
- Replay: Re-runs physics simulation with recorded inputs
|
|
- Result: Bit-for-bit identical to original match
|
|
|
|
**Technical Details:**
|
|
```python
|
|
# Conceptual Rocket League replay
|
|
class RocketLeagueReplay:
|
|
def __init__(self):
|
|
self.initial_state = {
|
|
'ball_pos': (0, 0, 100),
|
|
'ball_vel': (0, 0, 0),
|
|
'cars': [
|
|
{'pos': (-2000, 0, 17), 'vel': (0, 0, 0)},
|
|
{'pos': (2000, 0, 17), 'vel': (0, 0, 0)}
|
|
],
|
|
'rng_seed': 42
|
|
}
|
|
self.inputs = []
|
|
|
|
def record_input(self, tick, player_id, throttle, steer, boost):
|
|
"""Record player input"""
|
|
self.inputs.append({
|
|
'tick': tick,
|
|
'player': player_id,
|
|
'throttle': throttle,
|
|
'steer': steer,
|
|
'boost': boost
|
|
})
|
|
|
|
def replay(self):
|
|
"""Replay match from inputs"""
|
|
physics = DeterministicPhysics(seed=self.initial_state['rng_seed'])
|
|
physics.set_initial_state(self.initial_state)
|
|
|
|
for tick in range(18000): # 5 min @ 60fps
|
|
# Apply player inputs
|
|
for input_event in self.get_inputs_for_tick(tick):
|
|
physics.apply_input(input_event)
|
|
|
|
# Step physics (deterministic)
|
|
physics.step(dt=1/60.0)
|
|
|
|
return physics.get_state()
|
|
```
|
|
|
|
**Key Lessons:**
|
|
- Deterministic physics engine required
|
|
- Replays compress 1000x (100 KB vs 100 MB video)
|
|
- Can analyze pro plays in extreme detail
|
|
- Can debug "phantom hits" by inspecting exact frame
|
|
|
|
**Determinism Requirements:**
|
|
- Fixed timestep (1/60 second)
|
|
- Seeded RNG for ball bounce variance
|
|
- Fixed-point math for position/velocity
|
|
- Sorted player update order
|
|
|
|
### Example 2: StarCraft II - Deterministic RTS
|
|
|
|
**Challenge:** 300+ units, complex AI, must stay in sync across clients.
|
|
|
|
**Architecture:** Lockstep networking + deterministic simulation
|
|
|
|
**How It Works:**
|
|
```
|
|
Tick 0:
|
|
Client A: Send "Move unit 5 to (100, 200)"
|
|
Client B: Send "Attack unit 12 with unit 8"
|
|
|
|
Tick 1:
|
|
Both clients receive both commands
|
|
Both clients execute in SAME ORDER (sorted by player ID)
|
|
Both clients simulate identically
|
|
|
|
Result: Clients stay in sync, only send tiny command messages
|
|
```
|
|
|
|
**Determinism Techniques:**
|
|
```python
|
|
# SC2-style deterministic update
|
|
class SC2Simulation:
|
|
def __init__(self, seed):
|
|
self.rng = SeededRandom(seed)
|
|
self.units = {}
|
|
self.tick = 0
|
|
|
|
def step(self, commands):
|
|
"""
|
|
Execute one tick
|
|
commands: List of player commands (already sorted)
|
|
"""
|
|
self.tick += 1
|
|
|
|
# Execute commands in deterministic order
|
|
for command in sorted(commands, key=lambda c: (c.player_id, c.unit_id)):
|
|
self.execute_command(command)
|
|
|
|
# Update all units in deterministic order (sorted by unit ID)
|
|
for unit_id in sorted(self.units.keys()):
|
|
self.units[unit_id].update(self.rng)
|
|
|
|
# Collision detection (deterministic order)
|
|
self.detect_collisions()
|
|
|
|
return self.compute_checksum()
|
|
|
|
def execute_command(self, command):
|
|
"""Execute player command deterministically"""
|
|
unit = self.units[command.unit_id]
|
|
|
|
if command.type == 'move':
|
|
unit.target = command.position
|
|
elif command.type == 'attack':
|
|
unit.target_unit = command.target_unit_id
|
|
|
|
def compute_checksum(self):
|
|
"""Hash game state to detect desyncs"""
|
|
state = {
|
|
'tick': self.tick,
|
|
'units': sorted([u.to_dict() for u in self.units.values()],
|
|
key=lambda u: u['id'])
|
|
}
|
|
import hashlib, json
|
|
return hashlib.sha256(json.dumps(state, sort_keys=True).encode()).hexdigest()
|
|
```
|
|
|
|
**Desync Detection:**
|
|
```python
|
|
# Clients send checksums periodically
|
|
if self.tick % 60 == 0: # Every 1 second
|
|
checksum = self.compute_checksum()
|
|
send_to_server({'type': 'checksum', 'tick': self.tick, 'checksum': checksum})
|
|
|
|
# Server compares
|
|
if client_a_checksum != client_b_checksum:
|
|
print(f"DESYNC DETECTED at tick {tick}!")
|
|
# Pause game, log replay, investigate
|
|
```
|
|
|
|
**Key Lessons:**
|
|
- Sort EVERYTHING (commands, unit updates, collisions)
|
|
- Seeded RNG for random events (critical hits)
|
|
- Checksum validation every second
|
|
- Pause game immediately on desync (don't let it compound)
|
|
|
|
### Example 3: Age of Empires II - Floating Point Determinism
|
|
|
|
**Problem:** Different CPUs (x86 vs ARM) computed math differently → desyncs.
|
|
|
|
**Solution:** Fixed-point math for all positions and calculations.
|
|
|
|
**Before (Floating Point):**
|
|
```cpp
|
|
// ❌ Non-deterministic across platforms
|
|
float unit_x = 100.0f;
|
|
unit_x += velocity * deltaTime; // Different rounding on x86 vs ARM!
|
|
```
|
|
|
|
**After (Fixed-Point):**
|
|
```cpp
|
|
// ✅ Deterministic - integers are always same
|
|
typedef int32_t fixed_t; // 16.16 fixed-point
|
|
#define FIXED_SHIFT 16
|
|
#define FIXED_ONE (1 << FIXED_SHIFT)
|
|
|
|
fixed_t unit_x = 100 * FIXED_ONE; // 100.0 in fixed-point
|
|
unit_x += velocity * deltaTime / FIXED_ONE; // Integer math, always identical
|
|
```
|
|
|
|
**Python Example:**
|
|
```python
|
|
class FixedPoint:
|
|
"""16.16 fixed-point number (16 bits integer, 16 bits fraction)"""
|
|
SHIFT = 16
|
|
ONE = 1 << SHIFT
|
|
|
|
def __init__(self, value=0):
|
|
if isinstance(value, float):
|
|
self.raw = int(value * self.ONE)
|
|
else:
|
|
self.raw = value
|
|
|
|
def to_float(self):
|
|
return self.raw / self.ONE
|
|
|
|
def __add__(self, other):
|
|
result = FixedPoint()
|
|
result.raw = self.raw + other.raw
|
|
return result
|
|
|
|
def __mul__(self, other):
|
|
result = FixedPoint()
|
|
# Multiply then shift to avoid overflow
|
|
result.raw = (self.raw * other.raw) >> self.SHIFT
|
|
return result
|
|
|
|
# Usage
|
|
x = FixedPoint(100.5) # 100.5 in fixed-point
|
|
y = FixedPoint(2.25)
|
|
z = x + y # 102.75
|
|
print(z.to_float()) # 102.75 - deterministic on all platforms!
|
|
```
|
|
|
|
**Key Lessons:**
|
|
- Fixed-point eliminates platform differences
|
|
- Slight precision loss acceptable for determinism
|
|
- Use for: positions, velocities, health values
|
|
- Keep floats for: rendering, UI, non-critical values
|
|
|
|
### Example 4: Factorio - Deterministic Factory Simulation
|
|
|
|
**Challenge:** 1000s of entities (assemblers, belts, inserters) must stay in sync.
|
|
|
|
**Architecture:** Entity component system + deterministic update order
|
|
|
|
**Determinism Strategy:**
|
|
```python
|
|
class FactorioSimulation:
|
|
def __init__(self, seed):
|
|
self.rng = SeededRandom(seed)
|
|
self.entities = {} # entity_id → entity
|
|
self.tick = 0
|
|
|
|
def step(self):
|
|
"""One tick - update all entities deterministically"""
|
|
self.tick += 1
|
|
|
|
# Update in deterministic order (sorted by entity ID)
|
|
for entity_id in sorted(self.entities.keys()):
|
|
entity = self.entities[entity_id]
|
|
entity.update(self)
|
|
|
|
# Process entity interactions (deterministic order)
|
|
self.process_item_transfers()
|
|
self.process_crafting()
|
|
self.process_power_network()
|
|
|
|
def process_item_transfers(self):
|
|
"""Move items between entities deterministically"""
|
|
# Get all transfer requests
|
|
transfers = []
|
|
for entity_id in sorted(self.entities.keys()):
|
|
entity = self.entities[entity_id]
|
|
if hasattr(entity, 'get_transfer_requests'):
|
|
transfers.extend(entity.get_transfer_requests())
|
|
|
|
# Sort transfers deterministically
|
|
transfers.sort(key=lambda t: (t.source_id, t.dest_id, t.item_type))
|
|
|
|
# Execute transfers in order
|
|
for transfer in transfers:
|
|
self.execute_transfer(transfer)
|
|
```
|
|
|
|
**Item on Belt Determinism:**
|
|
```python
|
|
class TransportBelt:
|
|
def __init__(self, id):
|
|
self.id = id
|
|
self.items = [] # List of items on belt
|
|
|
|
def update(self, simulation):
|
|
"""Move items along belt"""
|
|
# Sort items by position for deterministic processing
|
|
self.items.sort(key=lambda item: item.position)
|
|
|
|
# Move each item
|
|
for item in self.items:
|
|
item.position += 0.03125 # Fixed speed (1/32 per tick)
|
|
|
|
# If at end, try to transfer
|
|
if item.position >= 1.0:
|
|
self.try_transfer_item(item, simulation)
|
|
```
|
|
|
|
**Key Lessons:**
|
|
- Sort entities before update
|
|
- Sort items on belts before processing
|
|
- Fixed speed (no floating point accumulation)
|
|
- Deterministic collision resolution (sort by ID)
|
|
|
|
### Example 5: Overwatch - Replay Debugging
|
|
|
|
**Use Case:** Debug "favor the shooter" issues where client sees hit but server disagrees.
|
|
|
|
**Architecture:** Server records authoritative replay + client replays
|
|
|
|
**Debugging Process:**
|
|
```python
|
|
class OverwatchDebugger:
|
|
def debug_hit_registration(self, bug_report):
|
|
"""Debug why shot didn't register"""
|
|
|
|
# 1. Load server replay
|
|
server_replay = load_replay(bug_report.match_id)
|
|
server_replay.jump_to_tick(bug_report.tick)
|
|
|
|
# 2. Load client replay (if available)
|
|
client_replay = load_replay(bug_report.client_replay_id)
|
|
client_replay.jump_to_tick(bug_report.tick)
|
|
|
|
# 3. Compare states
|
|
server_state = server_replay.get_state()
|
|
client_state = client_replay.get_state()
|
|
|
|
# 4. Visualize differences
|
|
debug = DebugVisualization()
|
|
|
|
# Draw server's view of player positions
|
|
debug.draw_circle(server_state.shooter.x, server_state.shooter.y,
|
|
radius=1, color='blue', label='Server')
|
|
debug.draw_circle(server_state.target.x, server_state.target.y,
|
|
radius=1, color='red', label='Server Target')
|
|
|
|
# Draw client's view
|
|
debug.draw_circle(client_state.shooter.x, client_state.shooter.y,
|
|
radius=1, color='cyan', label='Client')
|
|
debug.draw_circle(client_state.target.x, client_state.target.y,
|
|
radius=1, color='orange', label='Client Target')
|
|
|
|
# Draw raycast
|
|
debug.draw_line(server_state.shooter.x, server_state.shooter.y,
|
|
server_state.raycast_hit.x, server_state.raycast_hit.y,
|
|
color='yellow')
|
|
|
|
# 5. Identify discrepancy
|
|
distance = math.sqrt((server_state.target.x - client_state.target.x)**2 +
|
|
(server_state.target.y - client_state.target.y)**2)
|
|
|
|
print(f"Target position difference: {distance:.3f} meters")
|
|
print(f"Network latency: {bug_report.ping}ms")
|
|
print(f"Conclusion: Client prediction error due to {bug_report.ping}ms lag")
|
|
```
|
|
|
|
**Key Lessons:**
|
|
- Both client and server record replays
|
|
- Can replay side-by-side to compare
|
|
- Visualize discrepancies (where did each think target was?)
|
|
- Identify root cause (lag, prediction error, actual bug)
|
|
|
|
|
|
## Cross-References
|
|
|
|
### Use This Skill WITH:
|
|
- **ecosystem-simulation**: Ecosystem desyncs, population divergence
|
|
- **physics-simulation-patterns**: Physics determinism, floating point errors
|
|
- **ai-and-agent-simulation**: Deterministic AI, sorted agent updates
|
|
- **crowd-simulation**: Deterministic crowd movement, collision resolution
|
|
- **economic-simulation-patterns**: Deterministic economy calculations
|
|
|
|
### Use This Skill BEFORE:
|
|
- **multiplayer-implementation**: Must ensure determinism first
|
|
- **replay-systems**: Determinism required for replays
|
|
- **competitive-esports**: Tournament integrity requires bug-free determinism
|
|
|
|
### Broader Context:
|
|
- **systematic-debugging** (superpowers): General debugging methodology
|
|
- **test-driven-development** (superpowers): Write tests for determinism
|
|
- **root-cause-tracing** (superpowers): Trace desync to root cause
|
|
|
|
|
|
## Testing Checklist
|
|
|
|
### Determinism Validation
|
|
- [ ] Run simulation twice with same seed → identical checksums
|
|
- [ ] Test on different platforms (x86, ARM, Windows, Linux, Mac)
|
|
- [ ] Verify RNG is seeded per simulation (not global)
|
|
- [ ] Check all dict/map iteration is sorted
|
|
- [ ] Validate no `Date.now()` or wall-clock time usage
|
|
- [ ] Test multithreading is deterministic (or disabled)
|
|
- [ ] Run 10,000 ticks without divergence
|
|
|
|
### Replay System
|
|
- [ ] Record inputs and initial state
|
|
- [ ] Replay produces identical result to original
|
|
- [ ] Checksum validation detects any divergence
|
|
- [ ] Can save and load replay files
|
|
- [ ] Can jump to any tick using snapshots
|
|
- [ ] Replay file size reasonable (< 1MB for 10 min gameplay)
|
|
|
|
### Debug Visualization
|
|
- [ ] Debug draw shows entity positions
|
|
- [ ] Population graphs show trends over time
|
|
- [ ] Can compare two simulation states side-by-side
|
|
- [ ] State inspector shows entity internal values
|
|
- [ ] Frame-by-frame stepping works
|
|
- [ ] Visual diff highlights discrepancies
|
|
|
|
### Chaos/Butterfly Effect
|
|
- [ ] Test sensitivity to initial conditions (0.01% perturbation)
|
|
- [ ] Identify feedback loops that amplify errors
|
|
- [ ] Verify "close enough" is not used (must be exact)
|
|
- [ ] Test stability under parameter changes
|
|
- [ ] Check oscillations dampen (not grow exponentially)
|
|
|
|
### Edge Cases
|
|
- [ ] Zero population (ecosystem empty)
|
|
- [ ] Extreme populations (10,000+ entities)
|
|
- [ ] Zero resources (grass depleted)
|
|
- [ ] Boundary conditions (map edges, value limits)
|
|
- [ ] Rapid events (1000 entities spawned at once)
|
|
- [ ] Long runtime (10+ hours without issues)
|
|
|
|
### Property-Based Testing
|
|
- [ ] Determinism property (same seed → same result)
|
|
- [ ] Energy conservation (total energy <= initial)
|
|
- [ ] Population bounds (never negative, never explode)
|
|
- [ ] Replay reproducibility (replay always matches)
|
|
- [ ] State invariants (no duplicate IDs, valid ranges)
|
|
|
|
### Multiplayer Sync
|
|
- [ ] Clients exchange checksums periodically
|
|
- [ ] Desync detected within 1 second
|
|
- [ ] Pause game on desync detection
|
|
- [ ] Log replay for debugging
|
|
- [ ] Can identify which subsystem diverged
|
|
|
|
|
|
## Summary
|
|
|
|
Debugging simulation chaos requires **determinism**, **replay systems**, **debug visualization**, and **systematic methodology**.
|
|
|
|
**Core Principles:**
|
|
1. **Determinism is non-negotiable** - Same inputs → Same outputs, always
|
|
2. **Replay everything** - Can't debug what you can't reproduce
|
|
3. **Visualize everything** - Make invisible bugs visible
|
|
4. **Test systematically** - Property-based tests, edge cases, chaos tests
|
|
5. **Fail fast** - Assertions catch bugs at root cause, not symptoms
|
|
6. **Understand chaos** - Butterfly effects mean 0.0001% error → 10% divergence
|
|
|
|
**Most Common Failures:**
|
|
- ❌ Unseeded RNG → desync
|
|
- ❌ Unordered map iteration → desync
|
|
- ❌ Floating point accumulation → desync
|
|
- ❌ Wall-clock time → desync
|
|
- ❌ No replay system → can't reproduce bugs
|
|
- ❌ No visualization → debugging blind
|
|
- ❌ "Close enough" tolerance → chaotic systems compound errors
|
|
|
|
**Success Pattern:**
|
|
```python
|
|
# 1. Deterministic simulation
|
|
sim = DeterministicSimulation(seed=42)
|
|
|
|
# 2. Record replay
|
|
replay = ReplaySystem()
|
|
replay.start_recording(sim)
|
|
|
|
# 3. Add assertions
|
|
assert sim.tick >= 0
|
|
assert len(sim.deer) <= MAX_DEER
|
|
|
|
# 4. Visualize
|
|
debug = DebugVisualization()
|
|
debug.plot_graph('population')
|
|
|
|
# 5. Test determinism
|
|
assert sim1.checksum() == sim2.checksum()
|
|
```
|
|
|
|
Master these patterns, avoid the pitfalls, and your simulations will be deterministic, debuggable, and rock-solid reliable.
|