Created
February 21, 2026 04:47
-
-
Save davekennewell/0f1d09e6d6e1d0be850da691f5a7f761 to your computer and use it in GitHub Desktop.
ANUGA Refactor Plan - Sprint 1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>ANUGA Refactor Plan — Sprint 1</title> | |
| <style> | |
| :root { | |
| --bg: #0d1117; --surface: #161b22; --border: #30363d; | |
| --text: #c9d1d9; --text-muted: #8b949e; --text-bright: #f0f6fc; | |
| --accent: #58a6ff; --accent2: #3fb950; --warn: #d29922; --danger: #f85149; | |
| --code-bg: #1c2128; | |
| } | |
| * { margin: 0; padding: 0; box-sizing: border-box; } | |
| body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif; | |
| background: var(--bg); color: var(--text); line-height: 1.6; padding: 2rem; max-width: 1200px; margin: 0 auto; } | |
| h1 { color: var(--text-bright); font-size: 2rem; margin-bottom: 0.5rem; border-bottom: 1px solid var(--border); padding-bottom: 1rem; } | |
| h2 { color: var(--accent); font-size: 1.5rem; margin: 2.5rem 0 1rem; border-bottom: 1px solid var(--border); padding-bottom: 0.5rem; } | |
| h3 { color: var(--text-bright); font-size: 1.2rem; margin: 1.5rem 0 0.75rem; } | |
| h4 { color: var(--accent2); font-size: 1rem; margin: 1rem 0 0.5rem; } | |
| p { margin-bottom: 0.75rem; } | |
| a { color: var(--accent); text-decoration: none; } | |
| a:hover { text-decoration: underline; } | |
| code { background: var(--code-bg); padding: 0.15em 0.4em; border-radius: 3px; font-size: 0.9em; font-family: 'SFMono-Regular', Consolas, monospace; } | |
| pre { background: var(--code-bg); padding: 1rem; border-radius: 6px; overflow-x: auto; margin: 0.75rem 0; font-size: 0.85em; border: 1px solid var(--border); } | |
| pre code { background: none; padding: 0; } | |
| table { width: 100%; border-collapse: collapse; margin: 1rem 0; font-size: 0.9rem; } | |
| th, td { padding: 0.6rem 0.8rem; border: 1px solid var(--border); text-align: left; } | |
| th { background: var(--surface); color: var(--text-bright); font-weight: 600; } | |
| tr:nth-child(even) { background: rgba(22, 27, 34, 0.5); } | |
| .card { background: var(--surface); border: 1px solid var(--border); border-radius: 6px; padding: 1.25rem; margin: 1rem 0; } | |
| .card-warn { border-left: 4px solid var(--warn); } | |
| .card-danger { border-left: 4px solid var(--danger); } | |
| .card-good { border-left: 4px solid var(--accent2); } | |
| .card-info { border-left: 4px solid var(--accent); } | |
| .tag { display: inline-block; padding: 0.15em 0.5em; border-radius: 12px; font-size: 0.75rem; font-weight: 600; margin-right: 0.3rem; } | |
| .tag-red { background: rgba(248,81,73,0.15); color: var(--danger); } | |
| .tag-yellow { background: rgba(210,153,34,0.15); color: var(--warn); } | |
| .tag-green { background: rgba(63,185,80,0.15); color: var(--accent2); } | |
| .tag-blue { background: rgba(88,166,255,0.15); color: var(--accent); } | |
| .toc { background: var(--surface); border: 1px solid var(--border); border-radius: 6px; padding: 1.25rem; margin: 1.5rem 0; } | |
| .toc ol { padding-left: 1.5rem; } | |
| .toc li { margin: 0.3rem 0; } | |
| .subtitle { color: var(--text-muted); font-size: 1rem; margin-bottom: 1.5rem; } | |
| .metric { font-size: 2rem; font-weight: 700; color: var(--accent); } | |
| .metric-label { font-size: 0.8rem; color: var(--text-muted); text-transform: uppercase; } | |
| .grid-3 { display: grid; grid-template-columns: repeat(3, 1fr); gap: 1rem; margin: 1rem 0; } | |
| .grid-2 { display: grid; grid-template-columns: repeat(2, 1fr); gap: 1rem; margin: 1rem 0; } | |
| @media (max-width: 768px) { .grid-3, .grid-2 { grid-template-columns: 1fr; } } | |
| details { margin: 0.5rem 0; } | |
| details summary { cursor: pointer; color: var(--accent); font-weight: 600; padding: 0.5rem 0; } | |
| details summary:hover { text-decoration: underline; } | |
| .flow-diagram { background: var(--code-bg); border: 1px solid var(--border); border-radius: 6px; padding: 1.5rem; margin: 1rem 0; font-family: monospace; white-space: pre; font-size: 0.8rem; line-height: 1.4; overflow-x: auto; } | |
| ul, ol { padding-left: 1.5rem; margin-bottom: 0.75rem; } | |
| li { margin-bottom: 0.3rem; } | |
| hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; } | |
| </style> | |
| </head> | |
| <body> | |
| <h1>ANUGA Refactor Plan</h1> | |
| <p class="subtitle">Sprint 1: Make run_anuga explorable and trustworthy for engineering consultancies<br> | |
| Research completed 2026-02-21 — 10 parallel agents across 10 repos</p> | |
| <div class="toc"> | |
| <strong>Contents</strong> | |
| <ol> | |
| <li><a href="#exec">Executive Summary</a></li> | |
| <li><a href="#arch">Current Architecture</a></li> | |
| <li><a href="#repos">Repository Map</a></li> | |
| <li><a href="#flow">End-to-End Simulation Flow</a></li> | |
| <li><a href="#package">The run_anuga Package Format</a></li> | |
| <li><a href="#business">Business Context</a></li> | |
| <li><a href="#gaps">Gap Analysis</a></li> | |
| <li><a href="#sprint1">Sprint 1 Plan</a></li> | |
| <li><a href="#questions">Open Questions</a></li> | |
| </ol> | |
| </div> | |
| <!-- ============================================ --> | |
| <h2 id="exec">1. Executive Summary</h2> | |
| <div class="grid-3"> | |
| <div class="card"> | |
| <div class="metric-label">Repos Involved</div> | |
| <div class="metric">7</div> | |
| <p style="font-size:0.8rem;color:var(--text-muted)">run_anuga, compute_anuga, gn_anuga, hydrata, hydrology, anuga_core, docs_hydrata</p> | |
| </div> | |
| <div class="card"> | |
| <div class="metric-label">run_anuga Docs</div> | |
| <div class="metric">1 line</div> | |
| <p style="font-size:0.8rem;color:var(--text-muted)">"Run an ANUGA simulation using hydrata.com format."</p> | |
| </div> | |
| <div class="card"> | |
| <div class="metric-label">docs.hydrata.com</div> | |
| <div class="metric">13/22 stub</div> | |
| <p style="font-size:0.8rem;color:var(--text-muted)">Last updated May 2023, all screenshots from GN4</p> | |
| </div> | |
| </div> | |
| <div class="card card-info"> | |
| <strong>The situation:</strong> ANUGA is a technically strong 2D hydraulic solver (20+ years, Geoscience Australia) wrapped by Hydrata into a web-based SaaS. The physics works. Everything else — documentation, packaging, developer experience, training, consultancy trust — is missing or incomplete. The <code>run_anuga</code> repo has zero docs, zero working tests, isn't pip-installable, and has half its dependencies unlisted. The public docs site is 60% lorem ipsum stubs from 2023. | |
| </div> | |
| <div class="card card-good"> | |
| <strong>The opportunity:</strong> Business planning research shows no VC-backed startup is innovating on engineering modeling tools (all money went to insurance/finance risk). Engineers spend 68% of time on model setup and data prep, not compute. Cloud compute addresses only 8% of engineer time. The real value is in making ANUGA <em>learnable, usable, and trustworthy</em> for consulting engineers who currently default to free HEC-RAS or expensive TUFLOW. | |
| </div> | |
| <!-- ============================================ --> | |
| <h2 id="arch">2. Current Architecture</h2> | |
| <div class="flow-diagram"> | |
| Browser (MapStore React) | |
| | | |
| | AnugaPlugin: draw boundaries, upload elevation, configure scenarios | |
| v | |
| Django REST API (gn_anuga app, /anuga/api/) | |
| | | |
| |-- 18 models: Project, Scenario, Boundary, Elevation, Inflow, Friction, | |
| | Structure, MeshRegion, FullMesh, Network, Catchment, Nodes, Links, | |
| | Run, Comparison, Publication, ComputeInstanceTemplate, UserAgreement | |
| | | |
| |-- Celery "anuga" queue (10 workers) | |
| | |-- make_package_async: exports GeoJSON + clips TIF + creates mesh + ZIPs to S3 | |
| | |-- create_elevation_gn_layer: reproject, hillshade, DEM style | |
| | |-- make_gn_layer_results: upload result TIFs as GeoServer WMS layers | |
| | | |
| |-- scenario.run() -> POST package URL to compute server | |
| v | |
| Compute Server (compute_anuga app, ephemeral EC2) | |
| | | |
| |-- Downloads ZIP from S3, extracts | |
| |-- mpirun -np N python run_anuga/run.py --package_dir /path | |
| | | | |
| | |-- anuga.Domain: mesh, elevation, friction, boundary conditions | |
| | |-- anuga.distribute: MPI parallelism | |
| | |-- domain.evolve: shallow water equations on triangular mesh | |
| | |-- Memory watchdog: kill at 90%, restart from checkpoint | |
| | |-- post_process_sww: SWW -> GeoTIFF (depth, velocity, stage, momentum) | |
| | v | |
| |-- Uploads result ZIP to S3 | |
| |-- POSTs result_package_key back to control server | |
| v | |
| Result Processing (back on web server) | |
| |-- Downloads 3 max TIFs: depth_max, velocity_max, depthIntegratedVelocity_max | |
| |-- Publishes as GeoServer raster layers with SLD styles | |
| |-- Terminates compute EC2 instance | |
| v | |
| User views styled flood maps in MapStore | |
| </div> | |
| <h3>Key Infrastructure Details</h3> | |
| <table> | |
| <tr><th>Component</th><th>Technology</th><th>Notes</th></tr> | |
| <tr><td>Web servers</td><td>4x EC2 (Ubuntu 24.04)</td><td>hydrata.com, theswamm.com, sararaportal.com, nicaraguahydroportal.com</td></tr> | |
| <tr><td>Compute</td><td>Ephemeral EC2 (c7a.12xlarge, 48 vCPU, 96 GB)</td><td>Launched on-demand per simulation, terminated after</td></tr> | |
| <tr><td>Database</td><td>RDS PostgreSQL 16</td><td>Dual DB: {site}_app (metadata) + {site}_data (PostGIS)</td></tr> | |
| <tr><td>Object storage</td><td>S3 (4 buckets)</td><td>anuga-run-storage, anuga-data-storage, anuga-result-storage, anuga-stac-storage</td></tr> | |
| <tr><td>Message broker</td><td>Redis</td><td>Two Celery pools: web (13 queues) + anuga (1 queue)</td></tr> | |
| <tr><td>ANUGA version</td><td>3.2.0 (Hydrata fork)</td><td>Minimal fork: +16 lines (EPSG property, log fix, filename padding)</td></tr> | |
| <tr><td>Solver</td><td>Shallow water eqs, finite volume, triangular mesh</td><td>MPI parallel, checkpoint/restart, adaptive yieldstep</td></tr> | |
| <tr><td>Deploy</td><td>Ansible</td><td><code>cd ansible && make <sitename></code></td></tr> | |
| </table> | |
| <!-- ============================================ --> | |
| <h2 id="repos">3. Repository Map</h2> | |
| <table> | |
| <tr><th>Repo</th><th>Branch</th><th>Role</th><th>Health</th></tr> | |
| <tr> | |
| <td><code>Hydrata/run_anuga</code></td> | |
| <td><code>main</code></td> | |
| <td>Simulation execution engine. Reads "package format" (scenario.json + GeoJSON + GeoTIFF), runs ANUGA, outputs GeoTIFF rasters.</td> | |
| <td><span class="tag tag-red">Critical</span> No docs, no working tests, not pip-installable, half deps missing</td> | |
| </tr> | |
| <tr> | |
| <td><code>Hydrata/compute_anuga</code></td> | |
| <td><code>4.3.x_ubuntu24.04</code></td> | |
| <td>Celery task wrapper on compute nodes. Downloads package from S3, runs MPI, uploads results, memory watchdog.</td> | |
| <td><span class="tag tag-yellow">No 5.x branch</span> Still on 4.3.x</td> | |
| </tr> | |
| <tr> | |
| <td><code>Hydrata/gn_anuga</code></td> | |
| <td>In monolith (<code>5.x</code>)</td> | |
| <td>Django app: 18 models, REST API, Celery tasks, GeoServer layer management, permissions.</td> | |
| <td><span class="tag tag-green">Active</span> 563 commits, comprehensive</td> | |
| </tr> | |
| <tr> | |
| <td><code>Hydrata/hydrata</code></td> | |
| <td><code>5.x</code></td> | |
| <td>Monolith: 8 Django apps. Routes, settings, Celery config, database routers.</td> | |
| <td><span class="tag tag-green">Active</span></td> | |
| </tr> | |
| <tr> | |
| <td><code>Hydrata/hydrology</code></td> | |
| <td>In monolith (<code>5.x</code>)</td> | |
| <td>IDF tables, temporal patterns, time series generation for rainfall inputs.</td> | |
| <td><span class="tag tag-green">Good</span> Has tests</td> | |
| </tr> | |
| <tr> | |
| <td><code>Hydrata/anuga_core</code></td> | |
| <td><code>main</code> (v3.2.0)</td> | |
| <td>Fork of community ANUGA. 10-file diff from upstream (EPSG code, log fix, etc).</td> | |
| <td><span class="tag tag-green">Minimal fork</span></td> | |
| </tr> | |
| <tr> | |
| <td><code>Hydrata/docs_hydrata</code></td> | |
| <td><code>4.1.x</code> (deployed)</td> | |
| <td>Sphinx docs at docs.hydrata.com. 9 real pages, 13 lorem ipsum stubs.</td> | |
| <td><span class="tag tag-red">Abandoned</span> Last commit Jul 2023</td> | |
| </tr> | |
| <tr> | |
| <td><code>Hydrata/projects</code></td> | |
| <td>In monolith (<code>5.x</code>)</td> | |
| <td>GIS data management: organisations, datasets, custom SLD styles, MapStore integration.</td> | |
| <td><span class="tag tag-blue">Separate concern</span> Not ANUGA-specific</td> | |
| </tr> | |
| </table> | |
| <!-- ============================================ --> | |
| <h2 id="flow">4. End-to-End Simulation Flow</h2> | |
| <h3>4.1 What happens when a user runs a simulation</h3> | |
| <ol> | |
| <li><strong>Create Project</strong> — POST <code>/anuga/api/project/</code> auto-creates a MapStore map from <code>anuga_map_config.json</code> + a Publication (GeoStory)</li> | |
| <li><strong>Upload Elevation</strong> — TIF uploaded to S3, Celery chain: reproject to UTM, create hillshade, apply dynamic DEM SLD, auto-create default Boundary/Inflow/Friction/Structure/MeshRegion layers</li> | |
| <li><strong>Draw Inputs</strong> — User draws on empty PostGIS-backed GeoServer vector layers in MapStore (boundary lines, inflow lines, friction polygons, mesh regions, structures)</li> | |
| <li><strong>Create Scenario</strong> — Select which inputs to use, set resolution (m²) and duration (seconds). Triggers <code>make_package_async</code></li> | |
| <li><strong>Build Package</strong> (Celery, ~minutes): | |
| <ul> | |
| <li>Clone <code>run_anuga</code> from GitHub into package dir</li> | |
| <li>Export all inputs as GeoJSON via WFS</li> | |
| <li>Download elevation TIF from S3, clip to boundary +250m buffer</li> | |
| <li>Resolve hydrology TimeSeries references in inflow data</li> | |
| <li>Generate mesh (ANUGA native or mesher library)</li> | |
| <li>Write <code>scenario.json</code></li> | |
| <li>ZIP everything, upload to S3</li> | |
| </ul> | |
| </li> | |
| <li><strong>Run</strong> — POST to <code>scenario/{id}/run/</code>: | |
| <ul> | |
| <li>Auto-provision EC2 from launch template (c7a.12xlarge)</li> | |
| <li>Associate Elastic IP, wait for HTTP readiness</li> | |
| <li>POST S3 download URL to compute server</li> | |
| </ul> | |
| </li> | |
| <li><strong>Compute</strong> (on ephemeral EC2): | |
| <ul> | |
| <li>Download + extract ZIP to <code>/opt/anuga_data/{run_id}/</code></li> | |
| <li><code>mpirun -np N python run.py --package_dir ...</code></li> | |
| <li>Create ANUGA Domain from mesh, set elevation/friction/boundaries</li> | |
| <li>Apply rainfall operators (Polygonal_rate_operator) and surface inflows (Inlet_operator)</li> | |
| <li><code>domain.evolve(yieldstep, finaltime)</code> with checkpointing</li> | |
| <li>Memory watchdog: kill at 90%, wait for <40%, restart from checkpoint</li> | |
| <li>Post-process SWW to GeoTIFF (depth, velocity, stage, momentum — per-timestep and max)</li> | |
| <li>ZIP results, upload to S3, POST result key back</li> | |
| </ul> | |
| </li> | |
| <li><strong>Result Processing</strong>: download 3 max TIFs from S3, publish as GeoServer raster layers with SLD styles (depth: blue-purple 0-6m, velocity: 0-6m/s, flow: 0-20m³/s), terminate EC2</li> | |
| <li><strong>View</strong>: Result layers auto-added to MapStore map in "Results" group. Compare scenarios via raster difference.</li> | |
| </ol> | |
| <h3>4.2 Celery Task Inventory</h3> | |
| <table> | |
| <tr><th>Task</th><th>Queue</th><th>Limit</th><th>Purpose</th></tr> | |
| <tr><td><code>make_package_async</code></td><td>anuga</td><td>1h</td><td>Build simulation package, ZIP to S3</td></tr> | |
| <tr><td><code>start_next_compute_instance_async</code></td><td>anuga</td><td>1h</td><td>Provision EC2 compute instance</td></tr> | |
| <tr><td><code>run_on_local_celery</code></td><td>anuga</td><td>1h</td><td>Run simulation locally (dev mode)</td></tr> | |
| <tr><td><code>create_elevation_gn_layer</code></td><td>anuga</td><td>30m</td><td>Reproject TIF, create GeoServer layer + hillshade</td></tr> | |
| <tr><td><code>create_supporting_models</code></td><td>anuga</td><td>5m</td><td>Auto-create default input layers after elevation upload</td></tr> | |
| <tr><td><code>make_gn_layer_results</code></td><td>anuga</td><td>30m</td><td>Upload result TIFs as GeoServer layers with SLD</td></tr> | |
| <tr><td><code>compare_runs_async</code></td><td>anuga</td><td>30m</td><td>Compute raster differences between two runs</td></tr> | |
| <tr><td><code>run_anuga_async</code></td><td>broadcast</td><td>none</td><td>Execute MPI simulation on compute node</td></tr> | |
| </table> | |
| <!-- ============================================ --> | |
| <h2 id="package">5. The run_anuga Package Format</h2> | |
| <p>This is the core artifact that Sprint 1 aims to clean up. A "package" is a directory (zipped for transport) with this structure:</p> | |
| <pre><code>package_dir/ | |
| scenario.json # Master config | |
| inputs/ | |
| ele_*.tif # Elevation raster (GeoTIFF, UTM) | |
| bdy_*_boundary_01.json # Boundary conditions (GeoJSON) | |
| inf_*_inflow_01.json # Rainfall/surface inflows (GeoJSON) | |
| fric_*.json # Friction zones (GeoJSON, optional) | |
| str_*.json # Structures (GeoJSON, optional) | |
| mr_*.json # Mesh regions (GeoJSON, optional) | |
| net_*.json # Drainage network (GeoJSON, optional) | |
| cat_*.json # Catchments (GeoJSON, optional) | |
| nodes_*.json # Network nodes (GeoJSON, optional) | |
| links_*.json # Network links (GeoJSON, optional) | |
| buffer.shp/shx/dbf/prj # Buffer shapefile | |
| run_anuga/ # Clone of run_anuga repo (!) | |
| run_anuga/ | |
| run.py | |
| run_utils.py | |
| outputs_{project}_{scenario}_{run}/ # Created during simulation | |
| run_*_depth_max.tif | |
| run_*_velocity_max.tif | |
| run_*_depthIntegratedVelocity_max.tif | |
| run_*_stage_max.tif | |
| run_*_depth_NNNNNN.tif # Per-timestep rasters | |
| run_*.sww # ANUGA native output (NetCDF) | |
| run_*.msh # Mesh file | |
| checkpoints/*.pickle # MPI checkpoint files | |
| run_anuga_*.log # Simulation log</code></pre> | |
| <h3>5.1 scenario.json Schema (undocumented, reverse-engineered from code)</h3> | |
| <table> | |
| <tr><th>Field</th><th>Type</th><th>Required</th><th>Description</th></tr> | |
| <tr><td><code>id</code></td><td>int</td><td>Yes</td><td>Scenario ID (any integer for standalone use)</td></tr> | |
| <tr><td><code>run_id</code></td><td>int</td><td>Yes</td><td>Run ID</td></tr> | |
| <tr><td><code>project</code></td><td>int</td><td>Yes</td><td>Project ID</td></tr> | |
| <tr><td><code>epsg</code></td><td>string</td><td>Yes</td><td>CRS (e.g. "EPSG:32756")</td></tr> | |
| <tr><td><code>name</code></td><td>string</td><td>Yes</td><td>Human-readable name</td></tr> | |
| <tr><td><code>elevation</code></td><td>string</td><td>Yes</td><td>Elevation TIF filename in inputs/</td></tr> | |
| <tr><td><code>boundary</code></td><td>string</td><td>Yes</td><td>Boundary GeoJSON filename</td></tr> | |
| <tr><td><code>inflow</code></td><td>string</td><td>Yes</td><td>Inflow GeoJSON filename</td></tr> | |
| <tr><td><code>friction</code></td><td>string|null</td><td>No</td><td>Friction zones filename</td></tr> | |
| <tr><td><code>structure</code></td><td>string|null</td><td>No</td><td>Structures filename</td></tr> | |
| <tr><td><code>mesh_region</code></td><td>string|null</td><td>No</td><td>Mesh regions filename</td></tr> | |
| <tr><td><code>resolution</code></td><td>float</td><td>Yes</td><td>Base mesh triangle resolution (meters)</td></tr> | |
| <tr><td><code>duration</code></td><td>int</td><td>Yes</td><td>Simulation duration (seconds)</td></tr> | |
| <tr><td><code>simplify_mesh</code></td><td>bool</td><td>No</td><td>Use mesher tool vs ANUGA native</td></tr> | |
| <tr><td><code>control_server</code></td><td>string</td><td>No</td><td>Hydrata server URL for callbacks (omit for standalone)</td></tr> | |
| </table> | |
| <h3>5.2 Hardcoded Values in run_anuga</h3> | |
| <div class="card card-warn"> | |
| <table> | |
| <tr><th>Value</th><th>Location</th><th>Meaning</th></tr> | |
| <tr><td>5m</td><td>run_utils.py:176,415,889</td><td>Building burn-in height added to elevation</td></tr> | |
| <tr><td>0.04</td><td>run_utils.py:505</td><td>Default Manning's n (natural channels)</td></tr> | |
| <tr><td>10</td><td>run_utils.py:499</td><td>Manning's n for building structures</td></tr> | |
| <tr><td>1.0e-6</td><td>run_utils.py:186</td><td>Rainfall conversion: mm/hr → m/s</td></tr> | |
| <tr><td>0.005m</td><td>run.py:140</td><td>Minimum storable water height</td></tr> | |
| <tr><td>60s–30min</td><td>run.py:223-229</td><td>Yieldstep bounds</td></tr> | |
| <tr><td>100</td><td>run.py:223</td><td>Max yield steps</td></tr> | |
| <tr><td>0.99</td><td>run.py:121</td><td>Alpha parameter for elevation interpolation</td></tr> | |
| </table> | |
| <p>These should be configurable in scenario.json with sensible defaults, not hardcoded.</p> | |
| </div> | |
| <h3>5.3 Current Problems with run_anuga</h3> | |
| <div class="grid-2"> | |
| <div class="card card-danger"> | |
| <h4>Documentation</h4> | |
| <ul> | |
| <li>README is 1 sentence</li> | |
| <li>Zero docstrings in any function</li> | |
| <li>No schema docs for scenario.json or GeoJSON formats</li> | |
| <li>No explanation of hardcoded engineering values</li> | |
| <li>No architecture diagram</li> | |
| <li>No examples directory (just 48MB test zips)</li> | |
| </ul> | |
| </div> | |
| <div class="card card-danger"> | |
| <h4>Packaging & Tests</h4> | |
| <ul> | |
| <li>No setup.py, pyproject.toml, or setup.cfg</li> | |
| <li>Not pip-installable</li> | |
| <li>Half dependencies missing from requirements.txt</li> | |
| <li>The one parametrized test references a file that doesn't exist</li> | |
| <li>No CI/CD</li> | |
| <li>Self-cloned into every package (!) via git clone</li> | |
| </ul> | |
| </div> | |
| <div class="card card-danger"> | |
| <h4>Code Quality</h4> | |
| <ul> | |
| <li>No type hints</li> | |
| <li>No input validation or helpful error messages</li> | |
| <li>Several features disabled via commented-out code (STAC, video)</li> | |
| <li>Unusual self-import path: <code>from run_anuga.run_anuga.run_utils</code></li> | |
| <li>No linting or formatting</li> | |
| <li>Single contributor, single branch, 238 commits</li> | |
| </ul> | |
| </div> | |
| <div class="card card-danger"> | |
| <h4>Coupling</h4> | |
| <ul> | |
| <li>Tries to import Django settings (graceful fallback)</li> | |
| <li>Tries to import Celery logger (graceful fallback)</li> | |
| <li>Calls back to control_server with HTTP for status (skippable)</li> | |
| <li>S3 uploads in STAC code (disabled)</li> | |
| <li>But: <em>can</em> run standalone, which is the foundation to build on</li> | |
| </ul> | |
| </div> | |
| </div> | |
| <!-- ============================================ --> | |
| <h2 id="business">6. Business Context</h2> | |
| <div class="card card-info"> | |
| <h3>Strategic Thesis (from business-planning repo)</h3> | |
| <p>ANUGA has failed to achieve adoption in 20 years despite strong physics because <strong>everything around the solver is missing</strong>: GUI, training, certification, support, regulatory acceptance. No VC-backed startup targets engineering tools (all money in insurance/finance risk). The opportunity is the "missing middle" — engineers who can't afford $60K TUFLOW but find free HEC-RAS inadequate.</p> | |
| <h4>Where Engineers Spend Their Time</h4> | |
| <pre><code>Model Setup/Mesh Generation ████████████████████ 40% | |
| Data Gathering/Processing ██████████████ 28% | |
| Calibration/Validation ████████ 16% | |
| Running Simulations ████ 8% | |
| Post-Processing/Reporting ████ 8%</code></pre> | |
| <p><strong>Cloud compute solves only 8% of the problem.</strong> The real opportunity is the 68% on setup and data processing.</p> | |
| </div> | |
| <h3>6.1 What "Support a Consultancy Learn, Use, Trust ANUGA" Means</h3> | |
| <table> | |
| <tr><th>Phase</th><th>What Must Exist</th><th>Current State</th></tr> | |
| <tr> | |
| <td><strong>LEARN</strong></td> | |
| <td>Tutorials, guided workflows, example models, training materials, zero Python requirement</td> | |
| <td><span class="tag tag-red">Missing</span> 1 sentence README, 13 stub doc pages, no examples, last training workshop was 2008</td> | |
| </tr> | |
| <tr> | |
| <td><strong>USE</strong></td> | |
| <td>Web GUI for full workflow, cloud compute, collaboration, version control, report generation, templates</td> | |
| <td><span class="tag tag-yellow">Partial</span> Web UI works but needs polish; no reporting, no templates, no offline mode</td> | |
| </tr> | |
| <tr> | |
| <td><strong>TRUST</strong></td> | |
| <td>Validation docs, benchmarking, regulatory templates, support SLAs, certification, case studies</td> | |
| <td><span class="tag tag-red">Missing</span> Zero validation docs, zero case studies, zero regulatory templates</td> | |
| </tr> | |
| </table> | |
| <h3>6.2 Competitive Landscape</h3> | |
| <table> | |
| <tr><th>Competitor</th><th>Price</th><th>Threat</th><th>Gap vs Hydrata</th></tr> | |
| <tr><td>Flood Platform (Jacobs)</td><td>$0.23/hr compute</td><td><span class="tag tag-red">HIGH</span></td><td>Bundles TUFLOW + HEC-RAS + Flood Modeller</td></tr> | |
| <tr><td>MIKE on Azure (DHI)</td><td><$1/hr</td><td><span class="tag tag-yellow">MED</span></td><td>Enterprise, expensive base license</td></tr> | |
| <tr><td>HEC-RAS 2025 (USACE)</td><td>Free</td><td><span class="tag tag-yellow">MED</span></td><td>Going cloud-native with Docker + API</td></tr> | |
| <tr><td>TUFLOW</td><td>$40-60K AUD/yr</td><td><span class="tag tag-blue">LOW</span></td><td>35-year entrenchment in Australia</td></tr> | |
| </table> | |
| <div class="card card-good"> | |
| <h4>Remaining Differentiators</h4> | |
| <ul> | |
| <li>ANUGA triangular mesh (better for complex terrain than HEC-RAS structured grids)</li> | |
| <li>Coastal and tsunami modeling (genuine technical edge)</li> | |
| <li>Open-source transparency and auditability</li> | |
| <li>Multi-user collaboration with spatial version control (no competitor has this)</li> | |
| <li>AI features in-product (no competitor has shipped this yet — window closing)</li> | |
| <li><strong>The run_anuga package format itself</strong> — a standardized, inspectable, reproducible simulation package</li> | |
| </ul> | |
| </div> | |
| <!-- ============================================ --> | |
| <h2 id="gaps">7. Gap Analysis for Consultancy Adoption</h2> | |
| <h3>7.1 run_anuga Package Gaps</h3> | |
| <table> | |
| <tr><th>#</th><th>Gap</th><th>Impact</th><th>Sprint 1?</th></tr> | |
| <tr><td>1</td><td>Not pip-installable (no pyproject.toml)</td><td>Can't install with <code>pip install run-anuga</code></td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>2</td><td>Missing half of actual dependencies in requirements</td><td>Install fails silently, errors at runtime</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>3</td><td>Self-cloned into every package via git</td><td>Brittle, confusing, wastes space</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>4</td><td>No schema documentation for scenario.json</td><td>Must read source code to use</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>5</td><td>No schema docs for GeoJSON input formats</td><td>Must read source to know required properties</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>6</td><td>No input validation or error messages</td><td>Cryptic ANUGA/GDAL errors on malformed input</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>7</td><td>Hardcoded engineering constants</td><td>Can't configure without editing source</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>8</td><td>Zero working tests</td><td>No regression safety</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>9</td><td>No example packages</td><td>No starting point for new users</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>10</td><td>Unusual import path (<code>run_anuga.run_anuga.run_utils</code>)</td><td>Confusing for developers</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>11</td><td>No CLI with --help / --validate / --dry-run</td><td>No discoverability, no pre-flight checks</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>12</td><td>No JSON Schema for validation</td><td>Can't validate packages without running them</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| </table> | |
| <h3>7.2 Documentation Gaps</h3> | |
| <table> | |
| <tr><th>#</th><th>Gap</th><th>Impact</th><th>Sprint 1?</th></tr> | |
| <tr><td>1</td><td>13 of 22 doc pages are lorem ipsum</td><td>Looks abandoned to visitors</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>2</td><td>All screenshots from GN4 (2023)</td><td>Confusing — UI has changed significantly</td><td><span class="tag tag-yellow">Later</span></td></tr> | |
| <tr><td>3</td><td>No ANUGA conceptual introduction</td><td>Engineers don't know what they're using</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>4</td><td>No input data preparation guide</td><td>Can't prepare real project data</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>5</td><td>No Manning's n reference table</td><td>Must Google separately</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>6</td><td>No boundary condition theory</td><td>Wrong BC selection causes silent errors</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>7</td><td>No mesh resolution guidance</td><td>Over/under-meshing wastes compute or gives bad results</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>8</td><td>No results interpretation guide</td><td>Can't make engineering decisions from outputs</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>9</td><td>No API documentation</td><td>Can't script or integrate</td><td><span class="tag tag-yellow">Later</span></td></tr> | |
| <tr><td>10</td><td>No troubleshooting guide</td><td>No help when things go wrong</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| <tr><td>11</td><td>No SWAMM documentation</td><td>Major product feature undocumented</td><td><span class="tag tag-yellow">Later</span></td></tr> | |
| <tr><td>12</td><td>Branch divergence (main vs 4.1.x)</td><td>git clone gives worse version than live site</td><td><span class="tag tag-green">Yes</span></td></tr> | |
| </table> | |
| <!-- ============================================ --> | |
| <h2 id="sprint1">8. Sprint 1 Plan</h2> | |
| <div class="card card-info"> | |
| <strong>Sprint Goal:</strong> A consulting engineer can <code>pip install run-anuga</code>, run <code>run-anuga --help</code>, open an example package, understand every file, modify it for their site, validate it, and run a simulation — all without reading source code. | |
| </div> | |
| <h3>Track A: run_anuga Package Refactor</h3> | |
| <h4>A1. Make it pip-installable</h4> | |
| <div class="card"> | |
| <ul> | |
| <li>Add <code>pyproject.toml</code> with proper metadata, version, entry points</li> | |
| <li>Flatten package structure: <code>run_anuga/run_anuga/</code> → <code>src/run_anuga/</code> (standard src layout)</li> | |
| <li>Fix import paths: <code>from run_anuga.run_utils import ...</code> (no double nesting)</li> | |
| <li>Declare ALL dependencies including optional groups: | |
| <pre><code>[project] | |
| name = "run-anuga" | |
| version = "1.0.0" | |
| dependencies = [ | |
| "numpy>=2.0", "pandas>=2.0", "rasterio>=1.4", | |
| "requests>=2.30", "dill>=0.3", "psutil>=5.0", | |
| "shapely>=2.0", | |
| ] | |
| [project.optional-dependencies] | |
| mpi = ["mpi4py>=3.1"] | |
| video = ["opencv-python>=4.0", "matplotlib>=3.8"] | |
| stac = ["pystac>=1.10", "boto3>=1.34"] | |
| all = ["run-anuga[mpi,video,stac]"] | |
| [project.scripts] | |
| run-anuga = "run_anuga.cli:main"</code></pre> | |
| </li> | |
| <li>Stop self-cloning: gn_anuga.make_package() should <code>import run_anuga</code> from the installed package, not clone a repo</li> | |
| </ul> | |
| </div> | |
| <h4>A2. Proper CLI with argparse/click</h4> | |
| <div class="card"> | |
| <pre><code>run-anuga run <package_dir> # Run simulation | |
| run-anuga validate <package_dir> # Validate package without running | |
| run-anuga info <package_dir> # Show scenario summary | |
| run-anuga init <dir> --template urban # Create example package from template | |
| run-anuga post-process <sww_file> # Convert SWW to GeoTIFF | |
| run-anuga version # Show version info</code></pre> | |
| <p><code>--help</code> on every subcommand. <code>validate</code> checks: files exist, GeoJSON is valid, CRS matches, boundary forms closed polygon, required properties present. Returns structured JSON report.</p> | |
| </div> | |
| <h4>A3. JSON Schema for scenario.json</h4> | |
| <div class="card"> | |
| <ul> | |
| <li>Write a formal JSON Schema in <code>src/run_anuga/schemas/scenario.schema.json</code></li> | |
| <li>Write schemas for each GeoJSON input type (boundary, inflow, friction, structure, mesh_region)</li> | |
| <li>Use jsonschema library in <code>run-anuga validate</code></li> | |
| <li>Ship schemas so IDEs give autocomplete + validation</li> | |
| </ul> | |
| </div> | |
| <h4>A4. Make hardcoded values configurable</h4> | |
| <div class="card"> | |
| <p>Add optional fields to scenario.json with current values as defaults:</p> | |
| <pre><code>{ | |
| "defaults": { | |
| "manning_n": 0.04, | |
| "structure_manning_n": 10, | |
| "building_burn_height": 5.0, | |
| "minimum_storable_height": 0.005, | |
| "yieldstep_min": 60, | |
| "yieldstep_max": 1800, | |
| "max_yield_steps": 100, | |
| "alpha": 0.99, | |
| "rainfall_factor": 1.0e-6 | |
| } | |
| }</code></pre> | |
| <p>Code reads from scenario.json with fallback to current hardcoded values. Zero breaking changes for existing packages.</p> | |
| </div> | |
| <h4>A5. Input validation with helpful errors</h4> | |
| <div class="card"> | |
| <ul> | |
| <li>Validate all inputs before starting ANUGA (fail fast, not 20 minutes into a simulation)</li> | |
| <li>Check: elevation TIF is valid GeoTIFF, CRS matches EPSG, boundary forms closed polygon, inflow features have required properties, friction values are reasonable (0.001-10), duration > 0</li> | |
| <li>Error messages should say what's wrong AND how to fix it: | |
| <pre><code>ERROR: Boundary does not form a closed polygon. | |
| Gap of 12.3m between segments 'bdy_100_boundary_01' endpoints. | |
| Fix: Ensure all External boundary segments connect end-to-end.</code></pre> | |
| </li> | |
| </ul> | |
| </div> | |
| <h4>A6. Example packages</h4> | |
| <div class="card"> | |
| <ul> | |
| <li><strong>minimal/</strong> — Simplest possible: small DEM, rectangular boundary, constant rainfall. Runs in <1 min on 1 CPU.</li> | |
| <li><strong>urban-drainage/</strong> — Buildings (structures), variable mesh (mesh regions), friction zones. Realistic small-scale.</li> | |
| <li><strong>coastal/</strong> — Demonstrates ANUGA's coastal/tsunami strength (the key differentiator vs HEC-RAS).</li> | |
| <li>Each example includes a README.md explaining every file and the engineering context.</li> | |
| <li>Ship as <code>run-anuga init --template minimal</code></li> | |
| </ul> | |
| </div> | |
| <h4>A7. Working tests + CI</h4> | |
| <div class="card"> | |
| <ul> | |
| <li>Unit tests for: <code>setup_input_data()</code>, <code>create_boundary_polygon_from_boundaries()</code>, <code>validate_package()</code>, scenario.json parsing</li> | |
| <li>Integration test: run the minimal example package end-to-end (needs ANUGA installed)</li> | |
| <li>GitHub Actions CI: lint (ruff), type check (mypy), unit tests (no ANUGA needed), integration tests (ANUGA installed, nightly)</li> | |
| <li>Fix or remove the broken parametrized test</li> | |
| </ul> | |
| </div> | |
| <h3>Track B: docs.hydrata.com Overhaul</h3> | |
| <h4>B1. Fix repo structure</h4> | |
| <div class="card"> | |
| <ul> | |
| <li>Merge <code>4.1.x</code> content back to <code>main</code> (or switch default branch)</li> | |
| <li>Update to modern Sphinx + sphinx-book-theme</li> | |
| <li>Add GitHub Actions for build + deploy to S3</li> | |
| <li>Convert large GIFs to optimized MP4/WebP (91 MB → ~10 MB)</li> | |
| </ul> | |
| </div> | |
| <h4>B2. ANUGA Concepts section (new)</h4> | |
| <div class="card"> | |
| <p>Write from scratch:</p> | |
| <ul> | |
| <li><strong>What is ANUGA?</strong> — History, shallow water equations, finite volume method, triangular mesh, when to use 2D modeling</li> | |
| <li><strong>Core Concepts</strong> — Domain, mesh, quantities (stage, elevation, depth, momentum), boundary conditions, operators</li> | |
| <li><strong>Boundary Conditions</strong> — Dirichlet (fixed level), Reflective (wall), Transmissive (open outflow) — when to use each</li> | |
| <li><strong>Manning's Friction Reference</strong> — Table of typical values by land use (concrete: 0.013, mown grass: 0.035, dense bush: 0.15, etc.)</li> | |
| <li><strong>Mesh Resolution Guide</strong> — Resolution vs accuracy vs compute time, rules of thumb, convergence testing</li> | |
| <li><strong>Results Interpretation</strong> — Flood hazard categories (depth × velocity), freeboard, design flood levels</li> | |
| </ul> | |
| </div> | |
| <h4>B3. Package Format Reference (new)</h4> | |
| <div class="card"> | |
| <ul> | |
| <li>Complete reference for <code>scenario.json</code> with every field documented</li> | |
| <li>GeoJSON format specs for each input type (boundary, inflow, friction, structure, mesh_region)</li> | |
| <li>Output file descriptions (SWW, GeoTIFF, what each contains)</li> | |
| <li>Worked example: step by step from raw data to a complete package</li> | |
| </ul> | |
| </div> | |
| <h4>B4. Replace lorem ipsum stubs</h4> | |
| <div class="card"> | |
| <ul> | |
| <li>Replace all 13 stub pages with real content (inputs: elevation, boundaries, inflows, friction, structures, mesh regions, networks; running: configuring, selecting inputs, running on Hydrata, running locally)</li> | |
| <li>Note: screenshots will initially reference GN5 UI — this is acceptable for Sprint 1</li> | |
| </ul> | |
| </div> | |
| <h4>B5. Troubleshooting guide (new)</h4> | |
| <div class="card"> | |
| <ul> | |
| <li>Common errors and what they mean</li> | |
| <li>Model instability diagnosis (CFL condition, wet/dry oscillation)</li> | |
| <li>Mesh quality issues</li> | |
| <li>"My results look wrong" decision tree</li> | |
| <li>How to read the ANUGA log</li> | |
| </ul> | |
| </div> | |
| <h3>Sprint 1 Deliverables Summary</h3> | |
| <table> | |
| <tr><th>Deliverable</th><th>Track</th><th>Estimated Effort</th></tr> | |
| <tr><td><code>pyproject.toml</code> + src layout + flattened imports</td><td>A1</td><td>Medium</td></tr> | |
| <tr><td>CLI with run/validate/info/init subcommands</td><td>A2</td><td>Medium</td></tr> | |
| <tr><td>JSON Schemas for scenario + inputs</td><td>A3</td><td>Medium</td></tr> | |
| <tr><td>Configurable defaults in scenario.json</td><td>A4</td><td>Small</td></tr> | |
| <tr><td>Input validation with helpful errors</td><td>A5</td><td>Medium</td></tr> | |
| <tr><td>3 example packages with READMEs</td><td>A6</td><td>Medium</td></tr> | |
| <tr><td>Unit tests + CI</td><td>A7</td><td>Medium</td></tr> | |
| <tr><td>Fix docs repo structure + CI deploy</td><td>B1</td><td>Small</td></tr> | |
| <tr><td>ANUGA Concepts section (6 pages)</td><td>B2</td><td>Large</td></tr> | |
| <tr><td>Package Format Reference</td><td>B3</td><td>Medium</td></tr> | |
| <tr><td>Replace 13 stub pages</td><td>B4</td><td>Large</td></tr> | |
| <tr><td>Troubleshooting guide</td><td>B5</td><td>Medium</td></tr> | |
| </table> | |
| <h3>Dependency / Sequencing</h3> | |
| <div class="flow-diagram"> | |
| A1 (pip-installable) | |
| | | |
| +---> A2 (CLI) ---> A5 (validation) ---> A6 (examples) | |
| | | | |
| | v | |
| | A3 (JSON schemas) | |
| | | |
| +---> A4 (configurable defaults) | |
| | | |
| +---> A7 (tests + CI) -- depends on A1, A2, A5 | |
| B1 (fix repo) ---> B2 (concepts) ---> B3 (package format ref) | |
| | | |
| +---> B4 (replace stubs) | |
| | | |
| +---> B5 (troubleshooting)</div> | |
| <h3>Changes Required in Other Repos</h3> | |
| <div class="card card-warn"> | |
| <h4>gn_anuga (in Hydrata/hydrata monolith)</h4> | |
| <p>After run_anuga becomes pip-installable, <code>Scenario.make_package()</code> must stop cloning the repo and instead <code>import run_anuga</code> from the virtualenv. This is a follow-up change after Sprint 1 is merged.</p> | |
| <ul> | |
| <li>Remove the <code>git.Repo.clone_from()</code> call in <code>make_package()</code></li> | |
| <li>Update import paths from <code>run_anuga.run_anuga.run_utils</code> to <code>run_anuga.run_utils</code></li> | |
| <li>Update Ansible <code>extra_apps</code> to install via pip instead of git clone</li> | |
| </ul> | |
| </div> | |
| <!-- ============================================ --> | |
| <h2 id="questions">9. Open Questions for You</h2> | |
| <div class="card"> | |
| <ol> | |
| <li><strong>Scope of import path change:</strong> Flattening <code>run_anuga.run_anuga.X</code> to <code>run_anuga.X</code> will require updating imports in <code>gn_anuga/models.py</code> and <code>compute_anuga/tasks.py</code>. Should we do this in Sprint 1 (cleaner) or keep backward compat with a re-export shim?</li> | |
| <li><strong>Example package data licensing:</strong> The current test data includes Grand Canyon and Bullaburra (Australia) DEMs. Can we redistribute these? Should we create synthetic test DEMs instead?</li> | |
| <li><strong>docs.hydrata.com tech stack:</strong> Currently Sphinx + sphinx-book-theme deployed to S3. Want to stick with Sphinx, or migrate to something like MkDocs Material (faster builds, better search, easier for contributors)?</li> | |
| <li><strong>Standalone vs platform docs:</strong> Should the run_anuga package docs live in the run_anuga repo (close to code) or on docs.hydrata.com (centralized)? Recommendation: both — package README + reference in run_anuga repo, conceptual/tutorial content on docs.hydrata.com.</li> | |
| <li><strong>compute_anuga 5.x branch:</strong> This repo is still on 4.3.x. Should Sprint 1 include creating a 5.x branch, or defer this?</li> | |
| <li><strong>Customer discovery parallel track:</strong> The business-planning docs strongly recommend 10-15 customer conversations before building features. Should Sprint 1 include creating a "discovery interview guide" document alongside the technical work?</li> | |
| <li><strong>Package format versioning:</strong> Should scenario.json include a <code>"format_version": "1.0"</code> field to enable future evolution without breaking old packages?</li> | |
| </ol> | |
| </div> | |
| <hr> | |
| <p style="color: var(--text-muted); font-size: 0.85rem; text-align: center; margin-top: 2rem;"> | |
| Generated 2026-02-21 by Claude Code — 10 parallel research agents across run_anuga, compute_anuga, gn_anuga, hydrata, hydrology, anuga_core, docs_hydrata, business-planning, projects, and deploy repos | |
| </p> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment