hilarious failure.md

System behaved weird...

root@tracing-integration-tester-0:/var/lib/juju# curl -v --data '{"a":42}' localhost:41293
* Host localhost:41293 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:41293...
* Connected to localhost (::1) port 41293
> POST / HTTP/1.1
> Host: localhost:41293
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Length: 8
> Content-Type: application/x-www-form-urlencoded
>
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

What it really was

root@tracing-integration-tester-0:/var/lib/juju# ls -lt /proc/6055/fd
total 0
lrwx------ 1 root root 64 Jun  3 07:07 0 -> /dev/null
lrwx------ 1 root root 64 Jun  3 07:07 1 -> '/dev/pts/1 (deleted)'
lrwx------ 1 root root 64 Jun  3 07:07 2 -> '/dev/pts/1 (deleted)'
lrwx------ 1 root root 64 Jun  3 07:07 3 -> /run/server.db
lrwx------ 1 root root 64 Jun  3 07:07 5 -> 'socket:[3837945]'
root@tracing-integration-tester-0:/var/lib/juju# echo "aaa" > /proc/6055/fd/1
bash: /proc/6055/fd/1: Input/output error

The process was started from a tmux session and carried the session's stdout/err.
The session was over and the process was stuck unable to do anything useful, as it tripped on print(...).
The file descriptor inheritance was intentional, as the idea was to run this in k8s where stdout/err would get logged by the cluster infrastructure.