Posted 2020-06-16
Some time ago, at a tech conference, I found myself in a talk about cargo cults, delivered by a respected senior software developer.
If you haven't heard of cargo cults, I recommend the detour - they're fascinating, and the details are not the subject of this blog post.
The talk was somewhat meandering, but in summary, the speaker draws parallels between the seemingly irrational behavior of Melanesian cargo cultists, and a certain category of software developers. In more detail:
ps -ef
and ps -eaf
? Did you learn it from the man page, or did you memorise it when you saw it somewhere? (It's a trick question, by the way - a
is a subset of e
, so ps -ea
and ps -e
do the same thing - but perhaps more importantly, it's arguably wrong to expect ps -ea
to reliably do either one).
Ultimately, the folly of ps -eaf
is a decent example of tribal anti-knowledge, but hardly cargo cultish.The speaker's underlying angle ultimately comes to a (broad) point:
Surely there is some truth to a number of these statements, but the black and white mentality must also raise a number of red flags. For me, there are too many red flags to count, so I've chosen just a few:
The projects that surrounded this senior engineer's career were impressive, high-performance, and reliable. Many of those achievements demanded a deep understanding of fundamental computing that some would expect all software engineers to know. In this particular case, this was an individual (and a company) that valued OS knowledge, bit-fiddling, resource management, and network protocols.
A part of me is proud to have worked in that environment, and wielded those tools, but another part of me considers other fundamentals equally or perhaps more critical: functional programming, asynchronous design, composition, distributed systems, and sandboxing.
A third, more quiet, but irreplaceable part of me also values agile practices, operational excellence, microservices, stateless design, and learning/development.
In this environment, it was ultimately not atypical to encounter software that was 10x or 100x as efficient or powerful (for either the p50 or p99 case) versus a more naive approach - but, this software was an operational nightmare, and the story for scaling higher than "high" was non-existent.
These engineers had placed their software, and their careers into a relative maximum: go fast on a constrained environment, but otherwise fall flat.
If you want to hire the 100 best university graduates, and money is no object, you would be a fool to select purely by the ones that understand bit fiddling better than they understand popular framework that appear on hackernews. Students learn frameworks because it's what they're exposed to and it creates them value - we should celebrate that. In contrast, students that learned bit fiddling were attracted to a puzzle.
This concept is really my only justification for why $BigCompany hire python and golang developers: even if the technology is garbage, the median skill level is high. 100 skilled graduate engineers will beat your one C programming god on any metric (apart from monetary cost).
There's another side to the hiring concern though: selecting individuals based on perceived technical knowledge and attraction to puzzles seems to optimise for a sort of cliche INTJ individual with poor communication skills and a strong maths background. If that sounds like the ideal coworker, then this might be the wrong blog post for you. I will prefer a developer who can explain their problems and solutions, collaborate on issues, and design (artistic) good software solutions, over one who can implement paxos from memory ten out of ten times.
The speaker of this talk never worked on a project that was cancelled or considered a failure. That's not to say that the speaker always made their project a success, and not to say that the speaker was too good to be put on projects that were bound to fail. Ultimately, I believe that this means there was a management failure in only scheduling projects that were too reliable to fail - but this is a topic for another post.
Read the key part again: every project they worked on was completed and shipped.
If you think about what $BigCompany work on and promote, you'll notice the stark contrast. $BigCompany experiment. They try unusual and outlandish software, with unusual and outlandish deployment/business models. They experiment with developer practices, operational practices, team sizes, programming languages, and support models.
$BigCompany is more akin to a venture capital fund, throwing resources at decent ideas and looking for what sticks.
$BigCompany is the real-life capitalist result of the Infinite monkey theorem.
This is not all to say that companies which focus on single or few products are doing things wrong - but rather, optimality lies in risk.
We need bit-fiddling OS experts, and we need engineers that can scale bottleneck network infrastructure to hardware limits.
But, software engineering companies don't just need more and more of those engineers: they also need competent project and product managers, risk takers, exploration, new graduates with different views, and strong, branching paths for senior developer career paths.
We also need to acknowledge and celebrate our python and SQL friends, even if they make us cringe from time to time.
And if you find yourself using ps -eaf
, do you really care if you can't explain it exactly?