Created
August 17, 2026 07:58
-
-
Save lassebenni/77f1ffbceaa21479f046cf2e32b61b36 to your computer and use it in GitHub Desktop.
Visual: Week 15 ingest Mode 1 vs Mode 2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <title>Visual: Ingest Mode 1 vs Mode 2</title> | |
| <style> | |
| body { | |
| background-color: #1a1a1a; | |
| color: #f0f0f0; | |
| font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; | |
| margin: 0; | |
| padding: 20px; | |
| display: flex; | |
| justify-content: center; | |
| align-items: center; | |
| min-height: 100vh; | |
| } | |
| svg { | |
| background-color: #252526; | |
| border-radius: 8px; | |
| box-shadow: 0 4px 12px rgba(0,0,0,0.3); | |
| width: 100%; | |
| max-width: 640px; | |
| } | |
| .panel-title { fill: #f0f0f0; font-size: 14px; font-weight: 700; } | |
| .box { stroke-width: 2; rx: 6; } | |
| .airflow { fill: rgba(210,168,255,0.12); stroke: #d2a8ff; } | |
| .aca { fill: rgba(255,166,87,0.12); stroke: #ffa657; } | |
| .dbx { fill: rgba(88,166,255,0.12); stroke: #58a6ff; } | |
| .pg { fill: rgba(63,185,80,0.12); stroke: #3fb950; } | |
| .label { fill: #e0e0e0; font-family: monospace; font-size: 12px; text-anchor: middle; dominant-baseline: middle; } | |
| .task { fill: #c9d1d9; font-size: 11px; font-weight: 600; } | |
| .arrow { stroke: #888; stroke-width: 2; fill: none; } | |
| .arrow-head { fill: #888; } | |
| .mode { fill: #8b949e; font-size: 11px; font-weight: 700; letter-spacing: 0.05em; } | |
| .note { fill: #a0a0a0; font-size: 11px; } | |
| </style> | |
| </head> | |
| <body> | |
| <svg viewBox="0 0 640 400" xmlns="http://www.w3.org/2000/svg"> | |
| <defs> | |
| <marker id="ah" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto"> | |
| <polygon points="0 0, 10 3.5, 0 7" class="arrow-head" /> | |
| </marker> | |
| </defs> | |
| <text class="panel-title" x="160" y="28" text-anchor="middle">Mode 1 · INGEST_MODE=local</text> | |
| <text class="panel-title" x="480" y="28" text-anchor="middle">Mode 2 · INGEST_MODE=aca</text> | |
| <!-- Mode 1 --> | |
| <rect class="box airflow" x="24" y="44" width="272" height="140" /> | |
| <text class="mode" x="36" y="62">ASTRO WORKER</text> | |
| <text class="task" x="160" y="88" text-anchor="middle">ingest (local Python)</text> | |
| <text class="task" x="160" y="112" text-anchor="middle">dbt_build (subprocess)</text> | |
| <text class="task" x="160" y="136" text-anchor="middle">publish_to_backend</text> | |
| <text class="note" x="160" y="168" text-anchor="middle">all three tasks in one container</text> | |
| <!-- Mode 2 ingest split --> | |
| <rect class="box airflow" x="368" y="44" width="120" height="88" /> | |
| <text class="mode" x="380" y="62">ASTRO</text> | |
| <text class="task" x="428" y="88" text-anchor="middle">ingest</text> | |
| <text class="task" x="428" y="112" text-anchor="middle">(triggers)</text> | |
| <rect class="box aca" x="508" y="44" width="108" height="88" /> | |
| <text class="mode" x="520" y="62">ACA JOB</text> | |
| <text class="task" x="562" y="96" text-anchor="middle">pipeline.py</text> | |
| <path class="arrow" d="M 488 88 L 504 88" marker-end="url(#ah)" /> | |
| <rect class="box airflow" x="368" y="148" width="248" height="72" /> | |
| <text class="task" x="492" y="172" text-anchor="middle">dbt_build + publish_to_backend</text> | |
| <text class="note" x="492" y="200" text-anchor="middle">same worker as Mode 1</text> | |
| <!-- Shared downstream --> | |
| <text x="320" y="228" fill="#8b949e" font-size="12" text-anchor="middle" font-weight="600">Both modes — tasks 2 and 3</text> | |
| <rect class="box dbx" x="80" y="248" width="160" height="48" /> | |
| <text class="label" x="160" y="274">Databricks SQL warehouse</text> | |
| <rect class="box pg" x="400" y="248" width="160" height="48" /> | |
| <text class="label" x="480" y="274">Postgres analytics_dev</text> | |
| <path class="arrow" d="M 160 184 L 160 244" marker-end="url(#ah)" /> | |
| <path class="arrow" d="M 492 220 L 200 248" marker-end="url(#ah)" /> | |
| <path class="arrow" d="M 240 274 L 396 274" marker-end="url(#ah)" /> | |
| <text class="note" x="320" y="330" text-anchor="middle">Mode 1 first (fast loop). Mode 2 after image is built and pushed to ACR.</text> | |
| <text class="note" x="320" y="352" text-anchor="middle">Prod schedule always uses Mode 2 (ACA).</text> | |
| </svg> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment