notes.txt

/notes.txt
# DevOps Notes – if you find this, I've gone feral

---

🟢 **Day 1: Vision**

Clean setup. One instance. Node backend.  
"Firehose" service captures ride events and pushes them to Postgres.  
Event types: 'pickup', 'dropoff', 'cancelled', 'requested', maybe more.  
Didn't really document the schema… I figured I'd remember it.

Goal: Active rides = last known state is 'pickup'.  
Easy, right?

TODO:  
- Deploy Firehose  
- Write everything to Postgres  
- Build /active endpoint  
- Add timestamps (tomorrow-me problem)  
- Document schema (lol)  

---

🟡 **Day 4: Uneasy Metrics**

We're at 6k+ RPS and climbing.  
Postgres is now a write-only database. Firehose is hammering it.  
CPU load is through the roof. Logs are unsearchable.  
Slow query logs look like poems. Sad poems.

Firehose is Node.  
Should've written it in Go.  
Should've taken that frontend job.

---

🟠 **Day 6: Dashboard is Lying to Us**

Built `/active` endpoint.  
It returns: nothing.  
Like, literally zero. Not even "0". Just... silence.

Thought it was a logic bug.  
Turns out I was querying the wrong thing entirely.  
I *think* I need to group by user_id and driver_id?  
Maybe sort them somehow?

Couldn't figure it out, but it's too late now.  
CEO says the ops team is "losing visibility."  
I said, "so am I."

---

🔴 **Day 8: The Firehose is Alive**

CPU stuck at 97%. Memory's boiling.  
I set `Restart=always` in systemd — now it's aggressively failing.  
It dies, respawns, logs 10,000 things, dies again.  
Might be a memory leak.  
Might be the Node garbage collector just giving up.

Dashboard still shows no rides.  
Ops team says: "We have 12,000."  
I say: "We have none. The dashboard says so."

---

🪦 **Day ?? – Unmoored from Time**

I started sketching out a rewrite in Go.  
Didn't finish.  
But it felt... cleaner. Saner.

Thought about switching to Kinesis.  
Wrote it down. Drew a vortex around it.  
Then underlined it three times. Not sure why.

---

🧠 **If you're still in the fight...**

- Don't trust the dashboard  
- Don't trust the logs  
- Don't trust the Firehose  
- Trust nothing

Except:  
`htop`

That's the only thing that ever told me the truth.

Oh, and the DB login:  
User: `postgres`  
Password: `postgres`

Good luck.  
Tell Cece I tried.

~ D