DuckDB Simulation Engine

ERAD includes a high-performance simulation engine powered by DuckDB for million+ asset scale simulations.

Overview¶

The DuckDB engine replaces the Python loop-based approach with vectorized SQL computation:

Feature	Legacy Engine	DuckDB Engine
Asset scale	~10K	1M+
Parallelism	None (single-threaded)	All CPU cores
Storage	In-memory Pydantic objects	Columnar DuckDB tables
Export	SQLite (row-by-row ORM)	Parquet, SQLite, CSV, Arrow

The engine is the default for HazardSimulator.run(). Use engine='legacy' to fall back.

Quick Start¶

from erad.runner import HazardSimulator
from erad.systems.asset_system import AssetSystem
from erad.systems.hazard_system import HazardSystem

# Standard API (DuckDB engine is default)
# sim = HazardSimulator(asset_system)
# sim.run(hazard_system)

# Access engine for direct export (skip Pydantic hydration)
# sim.run(hazard_system, hydrate=False)
# sim.engine.export_to_parquet('results.parquet')

print('Engine modes: duckdb (default), legacy')

Direct Engine Usage (Scale Path)¶

For 100K+ assets, bypass Pydantic objects entirely and load data from arrays:

import numpy as np
from uuid import uuid4
from datetime import datetime
from shapely.geometry import Point
from infrasys.quantities import Distance

from erad.engine import SimulationEngine
from erad.systems.hazard_system import HazardSystem
from erad.models.hazard.earthquake import EarthQuakeModel
from erad.default_fragility_curves import DEFAULT_FRAGILTY_CURVES

# Generate synthetic assets
N = 1000
np.random.seed(42)

engine = SimulationEngine()
engine.load_assets_from_arrays(
    asset_ids=[str(uuid4()) for _ in range(N)],
    asset_names=[f'pole_{i}' for i in range(N)],
    asset_types=np.full(N, 6, dtype=np.int32),
    asset_type_names=['distribution_poles'] * N,
    latitudes=36.0 + np.random.random(N),
    longitudes=-121.0 + np.random.random(N),
    heights_m=np.full(N, 10.0),
)

# Create hazard system
eq = EarthQuakeModel(
    name='eq1', timestamp=datetime(2024, 1, 1),
    origin=Point(-120.5, 36.5), depth=Distance(15, 'km'), magnitude=5.5
)
hs = HazardSystem(name='demo')
hs.add_component(eq)

engine.load_hazards(hs)
engine.load_fragility_curves(DEFAULT_FRAGILTY_CURVES)
engine._loaded = True
engine.run()

df = engine.export_to_dataframe()
print(f'Computed {len(df):,} records')
print(f'Mean survival: {df["survival_probability"].mean():.4f}')
df.head()

Conversion Utilities¶

Convert between Pydantic models and DuckDB engine in either direction:

from erad.engine import SimulationEngine
from erad.systems.asset_system import AssetSystem

# Pydantic → DuckDB: Convert existing results for fast export
# engine = SimulationEngine.from_asset_system(asset_system)
# engine.export_to_parquet('results.parquet')

# DuckDB → Pydantic: Convert engine results back to objects
# asset_system = AssetSystem.from_engine(engine)
# asset_system.plot()

# Parquet → DuckDB: Load previous results for analysis
# engine = SimulationEngine.from_parquet('results.parquet')
# failed = engine.get_failed_assets(threshold=0.3)

print('Conversion paths:')
print('  SimulationEngine.from_asset_system(system)  → Pydantic to DuckDB')
print('  AssetSystem.from_engine(engine)             → DuckDB to Pydantic')
print('  SimulationEngine.from_parquet(path)         → Parquet to DuckDB')
print('  engine.to_asset_states()                    → DuckDB to AssetState list')

CLI Commands¶

The erad engine CLI provides command-line access to the DuckDB engine:

# Run simulation and export to Parquet
erad engine run <model_name> <hazard_name> -o results.parquet -f parquet

# Convert between formats
erad engine convert results.parquet results.db
erad engine convert results.db results.csv

# Query results with SQL
erad engine query results.parquet --sql "SELECT * FROM asset_states WHERE survival_probability < 0.5"

# Show summary statistics
erad engine info results.parquet

Export Formats¶

Format	Method	Best For
Parquet	`engine.export_to_parquet(path)`	Long-term storage, analytics (columnar, compressed)
SQLite	`engine.export_to_sqlite(path)`	Compatibility with legacy ERAD, simple queries
CSV	`engine.export_to_csv(path)`	Sharing with non-Python tools
Arrow	`engine.export_to_arrow()`	Zero-copy interop with pandas/polars
DataFrame	`engine.export_to_dataframe()`	Interactive analysis in notebooks

Custom SQL Queries¶

Access the underlying DuckDB connection for custom analytics:

# After running a simulation with the engine:
# engine.query("SELECT asset_type_name, AVG(survival_probability) FROM asset_states JOIN assets USING(asset_id) GROUP BY 1")
# engine.get_failed_assets(threshold=0.5)

print('Available query methods:')
print('  engine.query(sql)                    → Run arbitrary SQL')
print('  engine.get_failed_assets(threshold)  → Assets below survival threshold')
print('  engine.get_asset_count()             → Number of loaded assets')
print('  engine.get_timestamp_count()         → Number of time steps')