Your org is at greater risk of being Luddite than a Melanesian cargo cult

Posted 2020-06-16

Some time ago, at a tech conference, I found myself in a talk about cargo cults, delivered by a respected senior software developer.

If you haven't heard of cargo cults, I recommend the detour - they're fascinating, and the details are not the subject of this blog post.

The talk was somewhat meandering, but in summary, the speaker draws parallels between the seemingly irrational behavior of Melanesian cargo cultists, and a certain category of software developers. In more detail:

  • The content that appears on hackernews/reddit-programming seems to follow a cycle - the fashionable subjects we see on the front page today were popular some years ago, and by extension, they will fall out of favor again.
  • Newer developers tend to be heavily influenced by what appears on hackernews, concentrated over a short time span - naturally, they get a smaller aperture compared to the range of content (and cycles of content) that a more tenured developer would be familiar with.
  • Software developers tend to be critically-minded, but that does not make them immune to sensationalised marketing and products. In some sense, perhaps we are even blinded by our own expertise, and by our criticality.
  • Stack Overflow and the proliferation of APIs and frameworks lead us to copy-pasting things we don't understand. At this point in the talk, some find themselves disagreeing ("I don't copy code from stack overflow") - to which the speaker has prepared a counterpoint: do you know the difference between ps -ef and ps -eaf? Did you learn it from the man page, or did you memorise it when you saw it somewhere? (It's a trick question, by the way - a is a subset of e, so ps -ea and ps -e do the same thing - but perhaps more importantly, it's arguably wrong to expect ps -ea to reliably do either one). Ultimately, the folly of ps -eaf is a decent example of tribal anti-knowledge, but hardly cargo cultish.

The speaker's underlying angle ultimately comes to a (broad) point:

  • We should lean towards writing C, writing things from scratch, and be skeptical of reusing anything at all, really.
  • C++ is strictly worse than C - harder to debug, and more fragmentation. (Tangentially, I recommend what color is your function?).
  • python is hipster glorified webshit, java is an operational joke with poor type safety and insanely poor long-tail latency, golang is garbage with a dangerous package manager, javascript is for children, and rust is flavor-of-the-month. We should be dismissive of anyone who disagrees or utilises these technologies.
  • APIs enable graduates to become successful developers without building practical or theoretical knowledge of software engineering. Instead, one builds operational experience and strong pattern-matching/copy-paste skills, which lead to the illusion of growth. We should be wary to ensure that we invest time in the right kind of growth.

Surely there is some truth to a number of these statements, but the black and white mentality must also raise a number of red flags. For me, there are too many red flags to count, so I've chosen just a few:

A relative optima

The projects that surrounded this senior engineer's career were impressive, high-performance, and reliable. Many of those achievements demanded a deep understanding of fundamental computing that some would expect all software engineers to know. In this particular case, this was an individual (and a company) that valued OS knowledge, bit-fiddling, resource management, and network protocols.

A part of me is proud to have worked in that environment, and wielded those tools, but another part of me considers other fundamentals equally or perhaps more critical: functional programming, asynchronous design, composition, distributed systems, and sandboxing.

A third, more quiet, but irreplaceable part of me also values agile practices, operational excellence, microservices, stateless design, and learning/development.

In this environment, it was ultimately not atypical to encounter software that was 10x or 100x as efficient or powerful (for either the p50 or p99 case) versus a more naive approach - but, this software was an operational nightmare, and the story for scaling higher than "high" was non-existent.

These engineers had placed their software, and their careers into a relative maximum: go fast on a constrained environment, but otherwise fall flat.

Hiring and thinking styles

If you want to hire the 100 best university graduates, and money is no object, you would be a fool to select purely by the ones that understand bit fiddling better than they understand popular framework that appear on hackernews. Students learn frameworks because it's what they're exposed to and it creates them value - we should celebrate that. In contrast, students that learned bit fiddling were attracted to a puzzle.

This concept is really my only justification for why $BigCompany hire python and golang developers: even if the technology is garbage, the median skill level is high. 100 skilled graduate engineers will beat your one C programming god on any metric (apart from monetary cost).

There's another side to the hiring concern though: selecting individuals based on perceived technical knowledge and attraction to puzzles seems to optimise for a sort of cliche INTJ individual with poor communication skills and a strong maths background. If that sounds like the ideal coworker, then this might be the wrong blog post for you. I will prefer a developer who can explain their problems and solutions, collaborate on issues, and design (artistic) good software solutions, over one who can implement paxos from memory ten out of ten times.

Chaos borns value

The speaker of this talk never worked on a project that was cancelled or considered a failure. That's not to say that the speaker always made their project a success, and not to say that the speaker was too good to be put on projects that were bound to fail. Ultimately, I believe that this means there was a management failure in only scheduling projects that were too reliable to fail - but this is a topic for another post.

Read the key part again: every project they worked on was completed and shipped.

If you think about what $BigCompany work on and promote, you'll notice the stark contrast. $BigCompany experiment. They try unusual and outlandish software, with unusual and outlandish deployment/business models. They experiment with developer practices, operational practices, team sizes, programming languages, and support models.

$BigCompany is more akin to a venture capital fund, throwing resources at decent ideas and looking for what sticks.

$BigCompany is the real-life capitalist result of the Infinite monkey theorem.

This is not all to say that companies which focus on single or few products are doing things wrong - but rather, optimality lies in risk.

Wrap up

We need bit-fiddling OS experts, and we need engineers that can scale bottleneck network infrastructure to hardware limits.

But, software engineering companies don't just need more and more of those engineers: they also need competent project and product managers, risk takers, exploration, new graduates with different views, and strong, branching paths for senior developer career paths.

We also need to acknowledge and celebrate our python and SQL friends, even if they make us cringe from time to time.

And if you find yourself using ps -eaf, do you really care if you can't explain it exactly?