Very good articles written by @tef_ebooks on resilient code and systems

tef_ebooks-articles.md

@tef_ebooks articles

• Write code that is easy to delete, not easy to extend
• Write code that’s easy to delete, and easy to debug too
• Repeat yourself, do more than one thing, and rewrite everything
• How do you cut a monolith in half?
• Scaling in the presence of errors—don’t ignore them

twitter: @tef_ebooks

How do you cut a monolith in half?

Back-pressure is about pushing work to the edges, letting the ends of the conversation find stability, rather than trying to optimise all of the links in-between in isolation

Pushing work to the edges is how your system scales.

Pushing recovery to the outer layers is how your system handles failure.

Error recovery in the lower layers of a system is an optimisation, and you can’t push work to the centre of a network and scale. This is the end-to-end principle, and it is one of the most important ideas in system design.

The end-to-end principle is realising that no matter how many exceptions you handle deep down inside your program, some will leak out, and something at the outer layer has to take responsibility.

The complexity of a system lies in its protocol not its topology, and a protocol is what you create when you cut your monolith into pieces. If modularity is about building software, protocol is about how we break it apart

Protocol is the reason why ‘it depends’, and the reason why you shouldn’t depend on a message broker: You can use a message broker to glue systems together, but never use one to cut systems apart.

---

Quando a coordenação central se torna o fator limitante, mover responsabilidades para as bordas pode produzir ganhos desproporcionais de escala e resiliência — desde que você aceite a complexidade adicional que inevitavelmente virá junto.

O verdadeiro desafio de sistemas distribuídos não é mover código para processos diferentes; é definir corretamente os protocolos que governam as interações entre esses processos.

O protocolo é:

• Quem pode falhar?
• Quem faz retry?
• Quem detecta duplicação?
• Quem possui a verdade?
• Quem compensa?
• Quem faz timeout?
• Quem inicia a conversa?
• Quem encerra a conversa?
• o que significa um evento?
• quem o interpreta?
• quem reage?
• quem corrige?
• quem compensa?
• quem possui a verdade?

O broker apenas entrega o envelope. O protocolo é o acordo social que existe entre os sistemas que estão trocando aquelas mensagens. É esse acordo que determina se sua arquitetura será simples ou se ela se transformará num conjunto de serviços aparentemente desacoplados, mas semanticamente inseparáveis.

Scaling in the presence of errors—don’t ignore them

The secret to error handling at scale isn’t giving up, ignoring the problem, or even it trying again—it is structuring a program for recovery, making errors stand out, allowing other parts of the program to make decisions.

Techniques like fail-fast, crash-only-software, process supervision, but also things like clever use of version numbers, and occasionally the odd bit of statelessness or idempotence. What these all have in common is that they’re all methods of recovery.

Recovery is the secret to handling errors. Especially at scale.

Giving up early so other things have a chance, continuing on so other things can catch up, restarting from a clean state to try again, saving progress so that things do not have to be repeated.

That, or put it off for a while. Buy a lot of disks, hire a few SREs, and add another graph to the dashboard.

The problem with scale is that you can’t approach it with optimism. As the system grows, it needs redundancy, or to be able to function in the presence of partial errors or intermittent faults. Humans can only fill in so many gaps.

Writing robust software means building systems that can exist in a state of partial failure (like incomplete output), and writing resilient software means building systems that are always in a state of recovery (like restarting)—neither come from engineering the happy path of your software.

The problem with avoiding error handling in code is that you’re only avoiding automating it.

In other words, the trick to scaling in the presence of errors is building software around the notion of recovery. Automated recovery.

---

Um sistema robusto é capaz de continuar operando enquanto partes dele estão quebradas. Um sistema resiliente é capaz de se recuperar continuamente dessas falhas.

Nenhuma dessas propriedades surge do fluxo ideal de execução (happy path); elas surgem da forma como o sistema lida com estados incompletos, inconsistentes e falhos.