2024-11-24
How a mediocre programmer did some useful things.
- Based in Nashville, TN, USA, moved here in late 90s to briefly pursue music
- Local community: church, transit advocacy
- Love all things outdoors: walking, running, cycling, hiking, gardening (credit to my wife)
- Anything physical I can do without a screen: fixing bikes, carpentry
- Catching up on reading that I didn’t love as a kid: fiction, biographies, etc.
- Two teenage boys who are into all the usual sports
- TRS-80
- Commodore VIC 20
- Pascal from two years in CS
- No exposure to lisp
- A seed started to grow that I have little patience for languages (and technology in general) that waste my time (joy of programming), cue up Abraham Lincoln axe-sharpening (ironically, at the time, I thought that was Pascal but I just didn’t know anything; it’s why I love quoting Odin’s creator because Odin was heavily inspired by Nikolas Wirth’s work)
- Computer Engineering
- College was all about Unix and networking, got introduced to Linux via Slackware diskettes
- Only had heard of lisp, friend who had an artificial intelligence class and would complain about having to do lisp!?)
- Mostly did shell, C, perl, got introduced to the most OOP of Java (caused me to rage against the JVM which I would later change my mind about)
- Got exposed to Linux through Slackware. Worked with a few gods of Unix at the central computing center of Virginia Tech
- perl finally clicked with programming solving real-world problems
- Worked at Vanderbilt University in a human genetics lab managing a sizable amount of Sun Solaris workstations and an E450, and a mail system that the PI wanted
- Obsessed with qmail mail server, its radical simplicity, close relationship with unix instead of weird m4 macros
- Main communication mechanism in those days was email
- Followed qmail mailing list, mostly a unixy vi person, used mutt for mail, since mail geek I always had the User-Agent visible when I read messages, noticed a lot of the smart people used mail clients from Emacs (Gnus, VM, etc.)
- Got me curious about doing more in my editor to save time on tools
- Learned Emacs simply so I could do email more efficiently
- That sent me down the lisp rabbit hole, though ironically I didn’t actually write much elisp, just enough to configure emacs
- Something clicked with lisp. Two reasons:
- It’s physically rewarding
- Homoiconicity reduces cognitive load
- More than the community of interesting people or the technology underneath: The physical act of writing it.
- I believe we are embodied souls. There is a deeper reality than our physical world that gives interacting in this physical world meaning.
- There is something spiritual when our brains and hands collaborate on a task.
- It’s why we love mechanical keyboards
- It’s similar to playing an instrument or a sport. I also loved golf growing up. The tactile feel of the club in my hands and the clean strike of a ball is very satisfying. I’ve also played guitar most of my life. Many people have made this connection between art and the folks you find in the software industry, but I think there is actually something more primal than the cognitive and creative aspect.
- We delete way more code than we write. Think about all the things you try before you end up with a working chunk of functionality. With any programming language, the syntax has structure to it. You have nouns, verbs, statements, functions. It’s a lot. IDEs can only do so well. That goes away when you just write data structures: the computer and the human can both navigate with ease. Being able to confidently remove an entire form at will (thanks to paredit, but that’s why paredit is even possible) makes for an extremely pleasant and efficient session with the machine.
- Passively looking for a way to incorporate lisp into my professional life
- Experimented with Common Lisp, Scheme48, etc. but didn’t click
2005: Higher Order Perl
- Seminal book by Mark-Jason Dominus
- Another crucial piece of my journey: functional programming
- Programming is fundamentally a data management exercise, and the introduction of concepts like higher-order functions and infinite streams changed my thinking
- I had watched some SICP videos which also influenced me, but again, I didn’t realize at the time how practical Scheme really was
- Tried out being a web developer thanks to Rails
- No Clojure content here other than Ruby at scale is hard to debug, the lisp-inspired aspects of Ruby aren’t worth it, and the functional aspects are as laughable as they are in python
- Developer at a small healthcare company, team of 3
- Stakeholders kept asking for reports from the database
- Wrote some Python to stream it out, quickly got frustrated with it (there wasn’t near the ecosystem of libraries then)
- Ran across Clojure from a LtU post, binged Rich’s NYC videos over Christmas break
- Finally the trifecta (quadfecta?):
- Platform: JVM (which Rich changed my mind about)
- Syntax: Lisp
- Paradigm: Functional
- Performance: Fast
- It also had a BDFL which in general makes for better software
- Ported my Python to Clojure in a few days, made it parallelized and lazy, and it… worked.
- Continued to experiment, evolve the codebase, immersed myself in the Clojure community (IRC and mailing list in those days)
- “Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.”
- I had been emailing those data reports to various people in the company
- Factored out that code into a library called postal
- Simplest thing that could work
- Found a company who was hiring for an all-Clojure team
- Amazon-backed, e-discovery space, early AWS
- Dataset of email attachments in the petabyte range. We were in the top 5 users of AWS. We pushed every boundary.
- The Clojure team was reimplementing the whole search backend, from Ruby/Ferret to Solr, then Elasticsearch
- We were the first large user of Elasticsearch, funded its development, and moved that PB of data into it over the next year
- No one moves a PB of data to early Elasticsearch on magnetic EBS on early EC2, so we ran into lots of fun bugs both in Elasticsearch and AWS
- Got to know its creator Shay Banon pretty well from late-night debugging sessions. We were also heavy contributors.
- In 2012 Shay founded the company around Elasticsearch (which later became Elastic) and I jumped at the opportunity to join.
- Besides supporting Elasticsearch’s customers and wearing every other hat, I also built stuff for engineering
- Clojure-powered infrastructure
- Naturally that was Clojure-powered, some of which is still running 12 years
later.
- Initially used Pallet to deploy everything, which was like delicious alien technology (virtualbox in a repl!)
- Wrote a KPI engine for our software downloads
- Wrote an execution wrapper for CI builds to get around Jenkins’s limitations
- Wrote an Elasticsearch plugin (it’s a massive Java project) where you could connect to a running instance and get a Clojure repl. Huge help in the early days when ES didn’t have facilities for preventing shard rebalancing when a node restarted.
- We had several disjoint efforts at “infrastructure” – random tools, Puppet/Chef code, CI/CD – we wanted to consolidate that group and I took it on
- Fostered what we lovingly called the Infra Team, which was similar to my career: a kitchen-sink, no-limits approach to solving problems that weren’t in the mainline product remits. We also had the most experience at actually running Elasticsearch so we were a point where several concerns merged.
- Went on to hire more people, some of whom were already Clojure experts, some of whom began writing Clojure at Elastic and thrived. The team wrote things like a really nice alerting tool based on Riemann, tile server proxy for geo visualizations, etc.
- Close the chapter on Elastic after 10 years
- Joined a real estate company, leading engineering for two of the “digital” product lines
- Hired a bunch of Clojure titans (Ryan Senior of StrangeLoop, Chris Houser (chouser) of The Joy of Clojure, Joel Martin of mal fame, et al), several ex-Lonocloud (ViaSat) folks
- We’re now reinventing the IAM space by radically simplifying the service ecosystem at Equinix: standards-based and composable. We built an initial version, but production-quality, in a matter of weeks.
- I owe my career to Clojure
- Don’t settle for existing technology. Lean into the anger you feel when something is bad. Larry Wall’s Three Virtues
- Hire people who can communicate. A computer language is more like a human language than we want to admit.