Erik McClure

Software Optimizes to Single Points of Failure

Whenever people talk about removing single points of failure, most of the suggestions involve “distributed systems” that are resilient to hardware failures. For software, we’ve invented code signing and smart contracts via blockchain to ensure the code we’re running is what we expected to run.

But none of these technologies can prevent a bug from taking down the entire system.

A lot of people point to Google being a single point of failure. They are only partially correct, because Google’s hardware is distributed and extremely redundant. No single piece of hardware in a Google Data center failing can take down the entire data center. You could probably nuke the entire data center and most Google services could fall back to another data center. In fact, Google has developed software canaries to catch bugs from propagating too far into production in an attempt to address the problem of their software being a single point of failure.

But something did take down the entirety of Google Compute once. It was a software bug in the canary itself. Of course, all the canaries were running the same software, so all of them had the same bug, and none of them could catch the configuration bug that was being propagated to all of their routers.

By creating a software canary, Google had simply shifted the single point of failure to its canary software. It was much harder for it to fail, but it was still a single point of failure, so when it did fail, it took down the entire system.

We’ve put a lot of work into trying to reduce the amount of bugs in mission critical systems, going so far as to try to create provably correct software. The problem is that no system can prove that it is free of design-flaws, which occur when the software operates correctly, but does something nobody actually wanted it to do. All of our code-signing and trusted computing initiatives do is make it very difficult for someone to sneak bad code into a widely used library. None of them, however, remove the single point of failure. Should the NSA ever succeed in sneaking in a backdoor to a widely used open source library, it will propagate to everything.

A very well guarded single point of failure is still a single point of failure, no matter how remote the chances of it actually failing. Tom Scott has an excellent video about how a trusted engineer at Google that is allowed to bypass all their security checks could go rogue and remove all the password checks on everything and it would be incredibly hard to stop them.

Physical infrastructure is much more resilient to these kinds of problems, because even if every piece of infrastructure has the same problem, you still have to physically get to it in order to exploit the problem. This makes it very hard for anyone to simultaneously sabotage any country’s offline infrastructure without an incredible amount of work. Software, however, lets us access everything from everywhere. The internet removes physical access as a last resort.

Of course, this is not an insurmountable problem, but it is deceptively difficult to overcome. For example, let’s say we have a bunch of drones we’re controlling. To avoid one bug from taking all of them out at once, half of them run one flying program and the other run a completely different flying program, developed independently. Unfortunately, both of these programs rely on the same library that reads the gyroscope data. If that library has a bug, the entire swarm will crash into a mountain. Having the swarm calculate the data for each other and compare results doesn’t help, because everyone gets the wrong result. The software logic itself is wrong.

The reason this is so insidious is that it runs counter to sane software development practices. To minimize bugs, we minimize complexity, which means writing the least amount of code possible, which inadvertently optimizes to a single point of failure. We re-use libraries and share code. We deliberately try to solve problems exactly once and re-use this code everywhere in our program. This a good thing because it means any bugs we fix propagate everywhere else, but this comes at the cost of propagating any bugs we introduce.

Soon, our world will be consumed by automation, one way or another. Cory Doctorow suggests that hardware should only run software the user trusts, but what if I end up trusting buggy software? If all our self-driving cars run the same software, what happens when it has a bug? Even worse, what if all the different self-driving car companies have their own software, custom built by highly paid engineers… that all use OpenSSL to securely download updates?

What if OpenSSL has a bug?

It’s not clear what can be done about this. Obviously we shouldn’t go around introducing unnecessary complexity that creates even more bugs, but at the same time we shouldn’t delude ourselves into thinking our distributed systems have no single point of failure. They may be robust to hardware failures, but the software they run on will continue to be a single point of failure for the foreseeable future.

Migrating To A Static Blog

I’ve finished constructing a new personal website for myself using hugo, and I’m moving my blog over there so I have more control over what gets loaded, and more importantly, so the page doesn’t attempt to load Blogger’s 5 MB worth of bloated javascript nonsense just to read some text. It also fixes math and code highlighting while reading on mobile. If you reached this post using Blogger, you’ll be redirected or will soon be redirected to the corresponding post on my new website.

All comments have been preserved from the original posts, but making new comments is currently disabled - I haven’t decided if I want to use Disqus or attempt something else. An RSS feed is available on the bottom of the page for tracking new posts that should mimic the Blogger RSS feed, if you were using that. If something doesn’t work, poke me on twitter and I’ll try to fix it.

I implemented share buttons with simple links, without embedding any crazy javascript bullshit. In fact, the only external resource loaded is a Google tracking ID for pageviews. Cloudflare is used to enforce an HTTPS connection over the custom domain even though the website is hosted on Github Pages.

Hopefully, the new font and layout is easier to read than Blogger’s tiny text and bullshit theme nonsense.

How To Avoid Memorizing Times Tables

I was recently told that my niece was trying to memorize her times tables. As an applied mathematician whose coding involves plenty of multiplication, I was not happy to hear this. Nobody who does math actually memorizes times tables, and furthermore, forcing a child to memorize anything is probably the worst possible thing you can do in modern society. No one should memorize their times tables, they should learn how to calculate them. Forcing children to memorize useless equations for no reason is a great way to either ensure they hate math, teach them they should blindly memorize and believe anything adults tell them, or both. So for any parents who wish to teach their children how to be critical thinkers and give them an advantage on their next math test, I am going to describe how to derive the entire times tables with only 12 rules.

  1. Anything multiplied by 1 is itself. Note that I said anything, that includes fractions, pies, cars, the moon, or anything else you can think of. Multiplying it by 1 just gives you back the same result.

  2. Any number multiplied by 10 has a zero added on the end. 1 becomes 10, 2 becomes 20, 72 becomes 720, 9999 becomes 99990, etc.

  3. Any single digit multiplied by 11 simply adds itself on the end instead of 0. 1 becomes 11, 2 becomes 22, 5 becomes 55, etc. This is because you never need to multiply something by eleven. Instead, multiply it by 10 (add a zero to it) then add itself.

    \[ \begin{aligned} 11*11 = 11*(10 + 1) = 11*10 + 11 = 110 + 11 = 121\\ 12*11 = 12*(10 + 1) = 12*10 + 12 = 120 + 12 = 132 \end{aligned} \]

  4. You can always reverse the numbers being multiplied and the same result comes out. $$ 12*2 = 2*12 $$, $$ 8*7 = 7*8 $$, etc. This is a simple rule, but it’s very easy to forget, so keep it in mind.

  5. Anything multiplied by 2 is doubled, or added to itself, but you only need to do this up to 9. For example, $$ 4*2 = 4 + 4 = 8 $$. Alternatively, you can count up by 2 that many times:

    \[ 4*2 = 2 + 2 + 2 + 2 = 4 + 2 + 2 = 6 + 2 = 8 \]
    To multiply any large number by two, double each individual digit and carry the result. Because you multiply each digit by 2 separately, the highest result you can get from this is 18, so you will only ever carry a 1, just like in addition. This method is why multiplying anything by 2 is one of the easiest operations in math, and as a result the rest of our times table rules are going to rely heavily on it. Don’t worry about memorizing these results - you’ll memorize them whether you want to or not simply because of how often you use them.

  6. Any number multiplied by 3 is multiplied by 2 and then added to itself. For example:

    \[ 6*3 = 6*(2 + 1) = 6*2 + 6 = 12 + 6 = 18 \]
    Alternatively, you can add the number to itself 3 times: $$ 3*3 = 3 + 3 + 3 = 6 + 3 = 9 $$

  7. Any number multiplied by 4 is simply multiplied by 2 twice. For example: $$ 7*4 = 7*2*2 = 14*2 = 28 $$

  8. Any number multiplied by 5 is the same number multiplied by 4 and then added to itself.

    \[ 6*5 = 6*(4 + 1) = 6*4 + 6 = 6*2*2 + 6 = 12*2 + 6 = 24 + 6 = 30 \]
    Note that I used our rule for 4 here to break it up and calculate it using only 2. Once kids learn division, they will notice that it is often easier to calculate 5 by multiplying by 10 and halving the result, but we assume no knowledge of division.

  9. Any number multiplied by 8 is multiplied by 4 and then by 2, which means it’s actually just multiplied by 2 three times. For example: $$ 7*8 = 7*4*2 = 7*2*2*2 = 14*2*2 = 28*2 = 56 $$

  10. Never multiply anything by 12. Instead, multiply it by 10, then add itself multiplied by 2. For example: $$ 12*12 = 12*(10 + 2) = 12*10 + 12*2 = 120 + 24 = 144 $$

  11. Multiplying any single digit number by 9 results in a number whose digits always add up to nine, and whose digits decrease in the right column while increasing in the left column.

    \[ \begin{aligned} 9 * 1 = 09\\ 9 * 2 = 18\\ 9 * 3 = 27\\ 9 * 4 = 36\\ 9 * 5 = 45\\ 9 * 6 = 54\\ 9 * 7 = 63\\ 9 * 8 = 72\\ 9 * 9 = 81 \end{aligned} \]
    10, 11, and 12 can be calculated using rules for those numbers.

  12. For both 6 and 7, we already have rules for all the other numbers, so you just need to memorize 3 results:

    \[ \begin{aligned} 6*6 = 36\\ 6*7 = 42\\ 7*7 = 49 \end{aligned} \]
    Note that $$ 7*6 = 6*7 = 42 $$. This is where people often forget about being able to reverse the numbers. Every single other multiplication involving 7 or 6 can be calculated using a rule for another number.

And there you have it. Instead of trying to memorize a bunch of numbers, kids can learn rules that build on top of each other, each taking advantage of the rules established before it. It’s much more engaging then trying to memorize a giant table of meaningless numbers, a task that’s so mind-numbingly boring I can’t imagine forcing an adult to do it, let alone a small child. More importantly, this task teaches you what math is really about. It’s not about numbers, or adding things together, or memorizing a bunch of formulas. It’s establishing simple rules, and then combining those rules together into more complex rules you can use to solve more complex problems.

This also establishes a fundamental connection to computer science that is often glossed over. Both math and programming are repeated abstraction and generalization. It’s about combining simple rules into a more generalized rule, which can then be abstracted into a simpler form and combined to create even more complex rules. Programs start with machine instructions, while math starts with propositions. Programs have functions, and math has theorems. Both build on top of previous results to create more powerful and expressive tools. Both require a spark of creativity to recognize similarities between seemingly unrelated concepts and unite them in a more generalized framework.

We can demonstrate all of this simply by refusing to memorize our times tables.

Ignoring Outliers Creates Racist Algorithms

Have you built an algorithm that mostly works? Does it account for almost everyone’s needs, save for a few weird outliers that you ignore because they make up 0.0001% of the population? Congratulations, your algorithm is racist! To illustrate how this happens, let’s take a recent example from Facebook. My friend’s message was removed for “violating community standards”. Now, my friend has had all sorts of ridiculous problems with Facebook, so to test my theory, I posted the exact same message on my page, and then had him report it.

Golly gee, look at that, Facebook confirmed the message I sent does not violate community guidelines, but he’s still banned for 12 hours for posting the exact same thing. What I suspect happened is this: Facebook has gotten mad at my friend for having a weird name multiple times, but he can’t prove what his name is because he doesn’t have access to his birth certificate because of family problems, and he thinks someone’s been falsely reporting a bunch of his messages. The algorithm for determining whether or not something is “bad” probably took these misleading inputs, combined it with a short list of so-called “dangerous” topics like “terrorism”, and then decided that if anyone reported one of his messages, it was probably bad. On the other hand, I have a very western name and nobody reports anything I post, so either the report actually made it to a human being, or the algorithm simply decided it was probably fine.

Of course, the algorithm was wrong about my friend’s message. But Facebook doesn’t care. I’m sure a bunch of self-important programmers are just itching to tell me we can’t deal with all the edge-cases in a commercial algorithm because it’s infeasible to account for all of them. What I want to know is, have any of these engineers ever thought about who the edge-cases are? Have they ever thought about the kind of people who can’t produce birth certificates, or don’t have a driver’s license, or have strange names that don’t map to unicode properly because they aren’t western enough?

Poor people. Minorities. Immigrants. Disabled people. All these people they claim to care about, all this talk of diversity and equal opportunity and inclusive policies, and they’re building algorithms that by their very nature will exclude those less fortunate than them. Facebook’s algorithm probably doesn’t even know that my friend is asian, yet it’s still discriminating against him. Do you know who can follow all those rules and assumptions they make about normal people? Rich people. White people. Privileged people. These algorithms benefit those who don’t need help, and disproportionately punish those who don’t need any more problems.

What’s truly terrifying is that Silicon Valley wants to run the world, and it wants to automate everything using a bunch of inherently flawed algorithms. Algorithms that might be impossible to perfect, given the almost unlimited number of edge-cases that reality can come up with. In fact, as I am writing this article, Chrome doesn’t recognize “outlier” as a word, even though Google itself does.

Of course, despite this, Facebook already built an algorithm that tries to detect “toxicity” and silences “unacceptable” opinions. Even if they could build a perfect algorithm for detecting “bad speech”, do these companies really think forcibly restricting free speech will accomplish anything other than improving their own self-image? A deeply cynical part of me thinks the only thing these companies actually care about is looking good. A slightly more optimistic part of me thinks a bunch of well-meaning engineers are simply being stupid.

You can’t change someone’s mind by punching them in the face. Punching people in the face may shut them up, but it does not change their opinion. It doesn’t fix anything. Talking to them does. I’m tired of this industry hiding problems behind shiny exteriors instead of fixing them. That’s what used car salesmen do, not engineers. Programming has devolved into an art of deceit, where coders hide behind pretty animations and huge frameworks that sweep all their problems under the rug, while simultaneously screwing over the people who were supposed to benefit from an “egalitarian” industry that seems less and less egalitarian by the day.

Either silicon valley needs to start dealing with people that don’t fit in neat little boxes, or it will no longer be able to push humanity forward. If we’re going to move forward as a species, we have to do it together. Launching a bunch of rich people into space doesn’t accomplish anything. Curing cancer for rich people doesn’t accomplish anything. Inventing immortality for rich people doesn’t accomplish anything. If we’re going to push humanity forward, we have to push everyone forward, and that means dealing with all 7 billion outliers.

I hope silicon valley doesn’t drag us back to the feudal age, but I’m beginning to think it already has.

Sexist Programmers Are Awful Engineers

Men and women are fundamentally different. So are white people and black people and autistic people and gay people and transgender people and conservatives and liberals and every other human being along every imaginable axis of discrimination. Some of these differences are cultural. Others are genetic. Others depend on environmental factors. These differences mean that some of us are inherently better at certain tasks than others. On average, men are better at spatial temporal reasoning, women are better at reading comprehension and writing ability, and psychopaths can sometimes be excellent CEOs.

Whenever I meet a programmer who insists on doing everything a certain way, the chances I’ll hire them drop off a cliff. Just as object-oriented programming didn’t fix everything, neither will functional programming, or data-oriented programming or array-based programming or any other language. They are different tools that allow you to attack a problem from different directions, much like we have different classes of algorithms to attack certain classes of problems. Greedy algorithms, lazy evaluation, dynamic programming, recursive-descent, maximum flow, all of these are different ways to approach a problem. They represent looking at a problem from different perspectives. A problem that is difficult from one angle might be trivial when examined from a different angle.

When I stumbled upon this anti-diversity memo written by a Google employee, I wonder just how dysfunctional of an engineer that person is. Problems are never solved by being closed-minded. They are solved by opening ourselves to new possibilities and exploring the problem space as an infinitely-dimensional fabric of possible configurations. You do not find possible edge-cases by being closed-minded. You find them by probing the outer edges of your solution, trying to find singularities and inflection points that hint at unusual behavior.

You cannot build a great company by hiring people who are good at the same things you are. Attempting to maximize diversity only comes at a perceived cost of aptitude if you are measuring the wrong things. If your concept of what makes a “good programmer” is an extremely narrow set of skills, then you will inevitably select towards a specific ethnicity, culture, or sex, because the tiny statistical differences will be grossly magnified by the extremely narrow job requirements. Demand that all your programmers invert a binary tree on a whiteboard and you’ll filter out the guy who wrote the software 90% of your company uses.

If you think the field of computer science is really this narrow, you’re a terrible programmer. Turing completeness is a fundamental property of the universe, and we are only just beginning to explore the full implications of information theory, the foundations of type theory, NP-completeness, and the nature of computation itself. Disregarding other people because they can’t do something without ever considering what they can do will only hurt your team, and your company. Diversity inclusion programs shouldn’t try to hire more women and ethnic groups because they’re the same, they should be trying to hire them because they are different.

When hiring someone to complete a job, you should hire whoever is the best fit for the job. In a vacuum where there is a single task that needs to be completed, gender and ethnicity should be ignored in favor of a purely meritocratic assessment. However, if you have a company that must respond to a changing world, diversity can reveal solutions you never even knew existed. An established company like Google must actively seek to increase diversity so that it can explore new perspectives that may give it an edge over its rivals. They cannot select on a purely meritocratic basis, because all measures of merit would be based on what the company is already good at, not what it could be good at. You cannot explore new opportunities by hiring the same people.

Intelligent people value feedback from people who think differently than them. This is why many executives will deliberately hire people they disagree with so they can have someone challenge their views. This helps avoid creating an echo-chamber, which is the ultimate irony of a memo that’s called “Google’s Ideological Echo Chamber”, because scrapping the diversity inclusion programs as the memo suggests would itself create a new echo-chamber. You can’t remove an echo-chamber by removing diversity - the author’s premise is self-defeating. If they had stuck with only claiming that conservative ideologies should not be discriminated against, they would have been correct. Unfortunately, telling everyone they shouldn’t discriminate against your perspective, which itself involves discriminating against other perspectives, is by definition a contradiction.

We aren’t going to write better programs by doing the same thing we’ve been doing for the past 10 years. To improve is to change, and those who seek to become better software engineers must themselves embrace change, or they will be left behind to rot in the sewers of forgotten programs, maintaining rancid enterprise code for the rest of their lives. If we are unwilling to change who is writing the programs, we’ll be stuck making the same thing over and over again. A business that thinks nothing needs to change is one ripe for disruption. If you really think only hiring white males who correctly answer all your questions about graph theory and B-trees will help your business in the long-term, you’re an idiot.



  1. 2018
  2. 2017
  3. 2016
  4. 2015
  5. 2014
  6. 2013
  7. 2012
  8. 2011
  9. 2010
  10. 2009