RSS Feeds

These are archived from the now defunct su3su2u1 tumblr. Since there was some controversy over su3su2u1's identity, I'll note that I am not su3su2u1 and that hosting this material is neither an endorsement nor a sign of agreement.

Harry Potter and the Methods of Rationality full review

I opened up a bottle of delicious older-than-me scotch when Terry Pratchett died, and I’ve been enjoying it for much of this afternoon, so this will probably be a mess and cleaned up later.

Out of 5 stars, I’d give HPMOR a 1.5. Now, to the review (this is almost certainly going to be long)

The good

HPMOR contains some legitimately clever reworkings of the canon books to fit with Yudkowsky’s modified world:

A few examples- In HPMOR, the “interdict of Merlin” prevents wizards from writing down powerful spells, so Slytherin put the Basilisk in the chamber of secrets to pass on his magical lore. The prophecy “the dark lord will mark him as his own” was met when Voldemort gave Hariezer the same grade he himself had received.

Yudkowsky is also well read, and the story is peppered with reference to legitimately interesting science. If you google and research every reference, you’ll learn a lot. The problem is that most of the in-story references are incorrect, so if you don’t google around you are likely to pick up dozens of incorrect ideas.

The writing style during action scenes is pretty good. It keeps the pace moving and brisk and can be genuinely fun to read.

The bad

Stilted, repetitive writing

A lot of this story involves conversations that read like ham-fisted attempts at manipulation, filled with overly stilted language. Phrases like “Noble and Most Ancient House,” “General of Sunshine,” “ General of Chaos,”etc are peppered in over and over again. It’s just turgid. It smooths out when events are happening, but things are rarely happening.

Bad ideas

HPMOR is full of ideas I find incredibly suspect- the only character trait worth anything in the story (both implicitly and explicitly) is intelligence, and the primary use of intelligence within the story is manipulation. This leads to cloying levels of a sort of nerd elitism. Ron and Hagrid are basically dismissed out of hand in this story (Ron explicitly as being useless, Hagrid implicitly so) because they aren’t intelligent enough, and Hariezer explicitly draws implicit NPC vs real-people distinctions.

The world itself is constructed to back up these assertions- nothing in the wizarding world makes much sense, and characters often behave in silly ways (”like NPCs”) to be a foil for Hariezer.

The most ridiculous example of this is that the wizarding world justice is based on two cornerstones- poltiicans decide guilt or innocence for all wizard crimes, and the system of blood debts. All of the former death eaters who were pardoned (for claiming to be imperius cursed) apparently owe a blood debt to Hariezer, and so as far as wizarding justice is concerned he is above the law. He uses this to his advantage at a trial for Hermione.

Bad pedagogy

Hariezer routinely flubs the scientific concepts the reader is supposed to be learning. Almost all of the explicit in story science references are incorrect, as well as being overly-jargon filled.

Some of this might be on purpose- Hariezer is supposed to be only 11. However, this is terrible pedagogy. The reader’s guide to rationality is completely unreliable. Even weirder, the main antagonist, Voldemort, is also used as author mouthpiece several times. So the pedagogy is wrong at worst, and completely unreliable at best.

And implicitly, the method Hariezer relies on for the majority of his problem solving is Aristotelian science. He looks at things, thinks real hard, and knows the answer. This is horrifyingly bad implicit pedagogy.

Bad plotting

Over the course of the story, Hariezer moves from pro-active to no-active. At the start of the story he has a legitimate positive agenda- he wants to use science to uncover the secrets of magic. As the story develops, however, he completely loses sight of that goal, and he instead becomes just a passenger in the plot- he competes in Quirrell’s games and goes through school like any other student. When Voldemort starts including Hariezer in his plot, Hariezer floats along in a completely reactive way,etc.

Not until Hermione dies, near the end of the story, does Hariezer pick up a positive goal again (ending death) and he does absolutely nothing to achieve it. He floats along reacting to everything, and Voldemort defeats death and revives Hermione with no real input from Hariezer at all.

For a character who is supposed to be full of agency, he spends very little time exercising it in a proactive way.

Nothing has consequences (boring!)

And this brings me to another problem with the plotting- nothing in this story has any consequences. Nothing that goes wrong has any lasting implications for the story at all, which makes all the evens on hand ultimately boring. Several examples- early in the story Hariezer uses his time turner to solve even the simplest problems. Snape is asking you questions about potions you don’t know? Time travel. Bullies are stealing a meaningless trinket? Time travel,etc. As a result of these rule violations, his time turner is locked down by Professor Mcgonagall. Despite this Hariezer continues to use his time turner to solve all of his problems- the plot introduces another student willing to send a time turner message for a small amount of money via. “slytherin mail” it’s even totally anonymous.

Another egregious example of this is Quirrell’s battle game- the prize for the battle game is handed out by Quirrell in chapter 35 or so, and there are several more battle games after the prize! The reader knows that it doesn’t at all matter who wins these games- the prize is already awarded! What’s the point? The reader knows the prize has been given out, why are they invested in the proceedings at all?

When Hariezer becomes indebted to Luscious Malfoy, it never constrains him in any way. He becomes in debt, Dumbledore tells him it’s bad, he does literally nothing to deal with the problem. Two weeks later, Hermione dies and the debt gets cancelled.

When Hermione DIES Hariezer does nothing, and a few weeks later Voldemort brings her back. Nothing that happens ever matters.

The closest thing to long term repercussions is Hariezer helping Bellatrix Black escape- but we literally never see Bellatrix after that.

Hariezer never acts positively to fix his problems, he just bounces along whining about how humans need to defeat death until his problems get solved for him.

Mystery, dramatic irony and genre savvy

If you’ve read the canon books, you know at all times what is happening in the story. Voldemort has possessed Quirrell, Hariezer is a horcrux, Quirrell wants the philsopher’s stone, etc. There are bits and pieces that are modified, but the shape of the story is exactly canon. So all the mystery is just dramatic irony.

This is fine, as far as it goes, but there is a huge amount of tension because Hariezer is written as “genre savvy” and occasionally says things like “the hero of story such-and-such would do this” or “I understand mysterious prophecies from books.” The story is poking at cliches that the story wholeheartedly embraces. Supposedly Hariezer has read enough books just like this that dramatic irony liked this shouldn’t happen, as the story points out many times,- he should be just as informed as the reader. AND YET…

The author is practically screaming “wouldn’t it be lazy that Harry’s darkside is because he is a horcrux?” And yet, Harry’s darkside is because he is a horcrux.

Even worse, the narration of the book takes lots of swipes at the canon plots while “borrowing” the plot of the books.

Huge tension between the themes/lessons and the setting

The major themes of this book are in major conflict with the setting throughout the story.

One major theme is the need for secretive science to hide dangerous secrets- it’s echoed in the way Hariezer builds his “bayesian conspiracy,” reinforced by Hariezer and Quirrell’s attitudes toward nuclear weapons (and their explicit idea that people smart enough to build atomic weapons wouldn’t use them), and it’s reinforced at the end of the novel when Hariezer’s desire to dissolve some of the secrecy around magic is thwarted by a vow he took to not-end-the-world.

Unfortunately, that same secrecy is portrayed as having stagnated the progress of the wizarding world, and preventing magic from spreading. That same secrecy might well be why the wizarding world hasn’t already ended death and made thousands of philosopher’s stones.

Another major theme is fighting death/no-afterlife. But this is a fantasy story with magic. There are ghosts, a gate to the afterlife, a stone to talk to your dead loved ones,etc. The story tries to lamp shade it a bit, but that fundamental tension doesn’t go away. Some readers even assumed that Hariezer was simply wrong about an afterlife in the story- because they felt the tension and used my point above (unreliable pedagogy) to put the blame on Hariezer. In the story, the character who actually ended death WAS ALSO THE ANTAGONIST. Hariezer’s attempts are portrayed AS SO DANGEROUS THEY COULD END THE WORLD.

And finally- the major theme of this story is the supremacy of Bayesian reasoning. Unfortunately, as nostalgebraist pointed out explicitly, a world with magic is a world where your non-magic based Bayesian prior is worthless. Reasoning time and time again from that prior leads to snap conclusions unlikely to be right- and yet in the story this works time and time again. Once again, the world is fighting the theme of the story in obvious ways.

Let’s talk about Hermoine

The most explicitly feminist arc in this story is the arc where Hermione starts SPHEW, a group dedicated to making more wizarding heroines. The group starts out successful, gets in over their head, and Hariezer has to be called in to save the day (with the help of Quirrell).

At the end of the arc, Hariezer and Dumbledore have a long conversation about whether or not they should have let Hermione and friends play their little bully fighting game- which feels a bit like retroactively removing the characters agency. Sure, the women got to play at their fantasy, but only at the whim of the real heroes.

By the end of the story, Hermione is an indestructible part-unicorn/part-troll immortal. And what is she going to do with this power? Become Hariezer’s lab assistant, more or less. Be sent on quests by him. It just feels like Hermione isn’t really allowed to grow into her own agency in a meaningful way.

This isn’t to say that it’s intentional (pretty much the only character with real, proactive agency in this story is Quirrell) - but it does feel like women get the short end of the stick here.

Sanderson’s law of magic

So I’ve never read Sanderson, but someone point me to his first law of magic

Sanderson’s First Law of Magics: An author’s ability to solve conflict with magic is DIRECTLY PROPORTIONAL to how well the reader understands said magic. The idea here is that if your magic is laid out with clear rules, the author should feel free to solve problems with it- if your magic is mysterious and vague like Gandolf you shouldn’t solve all the conflict with magic, but if you lay out careful rules you can have the characters magic up the occasional solution. I’m not sure I buy into the rule fully, but it does make a good point- if the reader doesn’t understand your magic the solution might feel like it comes out of nowhere.

Yudkowsky never clearly lays out most of the rules of magic, and yet still solves all his problems via magic (and magic mixed with science). We don’t know how brooms work, but apparently if you strap one to a rocket you can actually steer the rocket, you won’t fall off the thing, and you can go way faster than other broomsticks.

This became especially problematic when he posted his final exam- lots of solutions were floated around each of which relied on some previously ill-defined aspect of the magic. Yudkowsky’s own solution relied on previously ill-defined transfiguration.

And when he isn’t solving problems like that, he is relying on the time turner over and over again. Swatting flies with flame throwers over and over again.

Coupled with the world being written as “insane” and it just feels like it’s lazy conflict resolution.

Conclusion

A largely forgettable, overly long nerd power fantasy, with a bit of science (most of it wrong) and a lot of bad ideas. 1.5 stars.

Individual chapter reviews below.

HPMOR 1

While at lunch, I dug into the first chapter of HPMOR. A few notes:

This isn’t nearly as bad as I remember, the writing isn’t amazing but its serviceable.. Either some editing has taken place in the last few years, or I’m less discerning than I once was.

There is this strange bit, where Harry tries to diffuse an argument his parents are having with:

“"Mum,” Harry said. “If you want to win this argument with Dad, look in chapter two of the first book of the Feynman Lectures on Physics. There’s a quote there about how philosophers say a great deal about what science absolutely requires, and it is all wrong, because the only rule in science is that the final arbiter is observation - that you just have to look at the world and report what you see. ”

This seems especially out of place, because no one is arguing about what science is.

Otherwise, this is basically an ok little chapter. Harry and Father are skeptical magic could exist, so send a reply letter to Hogwarts asking for a professor to come and show them some magics.

HPMOR 2: in which I remember why I hated this

This chapter had me rolling my eyes so hard that I now I have quite the headache. In this chapter, Professor McGonagall shows up and does some magic, first levitating Harry’s father, and then turning into a cat. Upon seeing the first, Harry drops some Bayes, saying how anticlimatic it was ‘to update on an event of infinitesimal probability,’ upon seeing the second, Hariezer Yudotter greets us with this jargon dump:

“You turned into a cat! A SMALL cat! You violated Conservation of Energy! That’s not just an arbitrary rule, it’s implied by the form of the quantum Hamiltonian! Rejecting it destroys unitarity and then you get FTL signalling!”

First, this is obviously atrocious writing. Most readers will get nothing out of this horrific sentence. He even abbreviated faster-than-light as FTL, to keep the density of understandable words to a minimum.

Second, this is horrible physics for the following reasons:

• the levitation already violated conservation of energy,which you found anticlimactic fuck you Hariezer
• the deep area of physics concerned with conservation of energy is not quantum mechanics, its thermodynamics. Hariezer should have had a jargon dump about perpetual motion machines. To see how levitation violates conservation of energy, imagine taking a generator like the Hoover dam and casting a spell to levitate all the water from the bottom of the dam back up to the top to close the loop. As long as you have wizard to move the water, you can generate power forever. Exercise for the reader- devise a perpetual motion machine powered by shape changers (hint:imagine an elevator system of two carts hanging over a pully. On one side, an elephant, on the other a man. Elephant goes down, man goes up. At the bottom, the elephant turns into a man and at the top the man turns into an elephant. What happens to the pulley over time?) -the deeper area related to conservation of energy is not unitarity, as is implied in the quote. There is a really deep theorem in physics, due to Emmy Noether, that tells us that conservation of energy really means that physics is time translationaly invariant. This means there aren’t special places in time, the laws tomorrow are basically the same as yesterday and today. (tangential aside- this is why we shouldn’t worry about a lack of energy conservation at the big bang, if the beginning of time was a special point, no one would expect energy to be conserved there). Unitarity in quantum mechanics is basically a fancy way of saying probability is conserved. You CAN have unitarity without conservation of energy. Technical aside- its easy to show that if the unitary operator is time-translation invariant, there is an operator that commutes with the unitary operator, usually called the hamiltonian. Without that assumption, we lose the hamiltonian but maintain unitarity. -None of this has much to do at all with faster than light signalling, which would be the least of our concern if we had just discovered a source of infinite energy.

I used to teach undergraduates, and I would often have some enterprising college freshman (who coincidentally was not doing well in basic mechanics) approach me to talk about why string theory was wrong. It always felt like talking to a physics madlibs book. This chapter let me relive those awkward moments.

Sorry to belabor this point so much, but I think it sums up an issue that crops up from time to time in Yudkowsky’s writing, when dabbling in a subject he doesn’t have much grounding in, he ends up giving actual subject matter experts a headache.

Summary of the chapter- McGonagall visits and does some magic, Harry is convinced magic is real, and they are off to go shop for Harry’s books.

Never Read Comments

I read the comments on an HPMOR chapter, which I recommend strongly against. I wish I could talk to several of the commentators, and gently talk them out of a poor financial decision.

Poor, misguided random internet person- your donation to MIRI/LessWrong will not help save the world. Even if you grant all their (rather silly) assumptions MIRI is a horribly unproductive research institute- in more than a decade, it has published fewer peer reviewed papers than the average physics graduate student does while in grad school. The majority of money you donate to MIRI will go into the generation of blog posts and fan fiction. If you are fine with that, then go ahead and spend your money, but don’t buy into the idea that this money will save the world.

HPMOR 3: uneventful, inoffensive

This chapter is worse than the previous chapters. As Hariezer (I realize this portmanteau isn’t nearly as clever as I seem to think it is, but I will continue to use it) enters diagon alley, he remarks

It was like walking through the magical items section of an Advanced Dungeons and Dragons rulebook (he didn’t play the game, but he did enjoy reading the rulebooks).

For reasons not entirely clear to me, the line filled me with rage.

As they walk McGonagall tells Hariezer about Voldemort, noting that other countries failed to come to Britain’s aid. This prompts Hariezer to immediately misuse the idea of the Bystander Effect (an exercise left to the reader- do social psychological phenomena that apply to individuals also apply to collective entities, like countries? Are the social-psychological phenomena around failure to act in people likely to also explain failure to act as organizations?)

Thats basically it for this chapter. Uneventful chapter- slightly misused scientific stuff, a short walk through diagon alley, standard Voldemort stuff. The chapter ends with some very heavy handed foreshadowing:

(And somewhere in the back of his mind was a small, small note of confusion, a sense of something wrong about that story; and it should have been a part of Harry’s art to notice that tiny note, but he was distracted. For it is a sad rule that whenever you are most in need of your art as a rationalist, that is when you are most likely to forget it.)

If Harry had only attended more CFAR workshops…

HPMOR 4: in which, for the first time, I wanted the author to take things further

So first, I actually like this chapter more than the previous few, because I think its beginning to try to deliver on what I want in the story. And now, my bitching will commence:

A recurring theme of the LessWrong sequences that I find somewhat frustrating is that (apart from the Bayesian Rationalist) the world is insane. This same theme pops up in this MOR chapter, where the world is created insane by Yudkowsky, so that Hariezer can tell you why.

Upon noticing the wizarding world uses coins of silver and gold, Hariezer asks about exchange rates, and asks the bank goblin how much it would cost to get a big chunk of silver turned into coins, the goblin says he’ll check with his superiors, Hariezer asks him to estimate, and the estimate is that the fee is about 5% of the silver.

This prompts Hariezer to realize that he could do the following:

1. Take gold coins and buy silver with them in the muggle world
2. bring the silver to Gringots and have it turned into coins
3. convert the silver coins to gold coins, ending up with more gold than you started with, start the loop over until the muggle prices make it not profitable

(of course, the in-story explanation is overly-jargon filled as usual)

This is somewhat interesting, and its the first look at what I want in a story like this- the little details of the wizarding world that would never be covered in a children’s story. Stross wrote a whole book exploring money/economics in a far future society (Neptune’s Brood, its only ok), there is a lot of fertile ground for Yudkowsky here.

In a world where wizards can magic wood into gold, how do you keep counterfeiting at bay? Maybe the coins are made of special gold only goblins know how to find (maybe the goblin hordes hoard (wordplay!) this special gold like De beers hoards diamonds).

Maybe the goblins carefully magic money into and out of existence in order to maintain a currency peg. Maybe its the perfect inflation- instead of relying on banks to disperse the coins every and now and then the coins in people’s pockets just multiply at random.

Instead, we get a silly, insane system (don’t blame Rowling either- Yudkowsky is more than willing to go off book, AND the details of this simply aren’t discussed, for good reason, in the genre Rowling wrote the books in), and rationalist Hariezer gets an easy ‘win’. Its not a BAD section, but it feels lazy.

And a brief note on the writing style- its still oddly stilted, and I wonder how good it would be at explaining ideas to someone unfamiliar. For instance, Hariezer gets lost in thought McGonagall says something, Hariezer replies:

“"Hm?” Harry said, his mind elsewhere. “Hold on, I’m doing a Fermi calculation." "A what? ” said Professor McGonagall, sounding somewhat alarmed. “It’s a mathematical thing. Named after Enrico Fermi. A way of getting rough numbers quickly in your head…”“

Maybe it would feel less awkward for Hariezer to say "Hold on, I’m trying to estimate how much gold is in the vault.” And then instead of saying “its a math thing,” we could follow Hariezer’s thoughts as he carefully constructs his estimate (as it is, the estimate is crammed into a hard-to-read paragraph).

Its a nitpick, sure, but the story thus far is loaded with such nits.

Chapter summary- Harry goes to gringots, takes out money.

HPMOR 5: in which the author assures us repeatedly this chapter is funny

This chapter is, again, mostly inoffensive, although there is a weird tonal shift. The bulk of this chapter is played broadly for laughs. There is actually a decent description of the fundamental attribution error, although its introduced with this twerpy bit of dialogue

Harry looked up at the witch-lady’s strict expression beneath her pointed hat, and sighed. “I suppose there’s no chance that if I said fundamental attribution error you’d have any idea what that meant.”

This sort of thing seems like awkward pedagogy. If the reader doesn’t know it, Hariezer is now exasperated with the reader as well as with whoever Yudkowsky is currently using as a foil.

Now, the bulk of this chapter involves Hariezer being left alone to buy robes, where he meets and talks to Draco Malfoy. Hariezer, annoyed at having people say to him “OMG, YOU ARE HARRY POTTER!” upon meeting and learning Malfoy’s name, exclaims “OMG, YOU ARE DRACO MALFOY!”. Malfoy accepts this as a perfectly normal reaction to his imagined fame, and a mildly amusing conversation occurs. Its a fairly clever idea.

Unfortunately, its marred by the literary equivalent of a sitcom laugh track. Worried that the reader isn’t sure if they should be laughing, Yudkowsky interjects phrases like these throughout:

Draco’s attendant emitted a sound like she was strangling but kept on with her work One of the assistants, the one who’d seemed to recognise Harry, made a muffled choking sound. One of Malkin’s assistants had to turn away and face the wall. Madam Malkin looked back silently for four seconds, and then cracked up. She fell against the wall, wheezing out laughter, and that set off both of her assistants, one of whom fell to her hands and knees on the floor, giggling hysterically.

The reader is constantly told that the workers in the shop find it so funny they can barely contain their laughter. It feels like the author constantly yelling GET IT YOU GUYS? THIS IS FUNNY!

As far as the writing goes, the tonal shift to broad comedy feels a bit strange and happens with minimal warning (there is a brief conversation earlier in the chapter thats also played for a laugh), and everything is as stilted as its always been. For example, when McGonagall walks into the robe shop in time to hear Malfoy utter some absurdities, Harry tells her

“He was in a situational context where those actions made internal sense -”

Luckily, Hariezer gets cut off before he starts explaining what a joke is.

Chapter summary- Hariezer buys robes, talks to Malfoy.

HPMOR 6: Yud lets it all hang out

The introduction suggested that the story really gets moving after chapter 5. If this is an example of what “really moving” looks like, I fear I’ll soon stop reading. Apart from my rant about chapter 2, things had been largely light, and inoffensive up until this chapter. Here, I found myself largely recoiling. We shift from the broad comedy of the last chapter to a chapter filled with weirdly dark little rants.

As should be obvious by now, I find the line between Eliezer and Harry to be pretty blurry (hence my annoying use of Hariezer). In this chapter, that line disappears completely as we get passages like this

Harry had always been frightened of ending up as one of those child prodigies that never amounted to anything and spent the rest of their lives boasting about how far ahead they’d been at age ten. But then most adult geniuses never amounted to anything either. There were probably a thousand people as intelligent as Einstein for every actual Einstein in history. Because those other geniuses hadn’t gotten their hands on the one thing you absolutely needed to achieve greatness. They’d never found an important problem.

There are dozens of such passages that could be ripped directly from some of Hariezer’s friendly AI writing and pasted right into MOR. Its a bit disconcerting, in part because its forcing me to face just how much of Eliezer’s other writing of wasted time with.

The chapter begins strongly enough, Hariezer starts doing some experiments with his magic pouch. If he asks for 115 gold coins, it comes, but not if he asks for 90+25 gold coins. He tries using other words for gold in other languages, etc. Unfortunately, it leads him to say this:

“I just falsified every single hypothesis I had! How can it know that ‘bag of 115 Galleons’ is okay but not ‘bag of 90 plus 25 Galleons’? It can count but it can’t add? It can understand nouns, but not some noun phrases that mean the same thing?…The rules seem sorta consistent but they don’t mean anything! I’m not even going to ask how a pouch ends up with voice recognition and natural language understanding when the best Artificial Intelligence programmers can’t get the fastest supercomputers to do it after thirty-five years of hard work,”

So here is the thing- it would be very easy to write a parser that behaves exactly like what Hariezer describes with his bag. You would just have a look-up table with lots of single words for gold in various languages. Nothing fancy at all. Its behaving oddly ENTIRELY BECAUSE ITS NOT DOING NATURAL LANGUAGE. I hope we revisit the pouch in a later chapter to sort this out. I reiterate, its stuff like this that (to me at least) were the whole premise of this story- flesh out the rules of this wacky universe.

Immediately after this, the story takes a truly bizarre turn. Hariezer spots a magic first aid kit, and wants to buy it. In order to be a foil for super-rationalist Harry, McGonagall then immediately becomes immensely stupid, and tries to dissuade him from purchasing it. Note, she doesn’t persuade him by saying “Oh, there are magical first aid kits all over the school,” or “there are wizards watching over the boy who lived who can heal you with spells if something happens” or anything sensible like that, she just starts saying he’d never need it.

This leads Harry to a long description of the planning fallacy, and he says to counter it he always tries to assume the worst possible outcomes. (Note to Harry and the reader: the planning fallacy is a specific thing that occurs when people or organizations plan to accomplish a task. What Harry is trying to overcome is more correctly optimism bias.).

This leads McGonagall to start lightly suggesting (apropos of nothing) that maybe Harry is an abused child. Hariezer responds with this tale:

"There’d been some muggings in our neighborhood, and my mother asked me to return a pan she’d borrowed to a neighbor two streets away, and I said I didn’t want to because I might get mugged, and she said, ‘Harry, don’t say things like that!’ Like thinking about it would make it happen, so if I didn’t talk about it, I would be safe. I tried to explain why I wasn’t reassured, and she made me carry over the pan anyway. I was too young to know how statistically unlikely it was for a mugger to target me, but I was old enough to know that not-thinking about something doesn’t stop it from happening, so I was really scared.” … I know it doesn’t sound like much,” Harry defended. “But it was just one of those critical life moments, you see? … That’s when I realised that everyone who was supposed to protect me was actually crazy, and that they wouldn’t listen to me no matter how much I begged them So we are back to the world is insane, as filtered through this odd little story.

Then McGonagall asks if Harry wants to buy an owl, and Harry says no he’d be too worried he’d forget to feed it or something. Which prompts McGonagall AGAIN to suggest Harry had been abused, which leads Harry into an odd rant about how false accusations of child abuse ruin families (which is true, but seriously, is this the genre for this rant? What the fuck is happening with this chapter?) This ends up with McGonagall implying Harry must have been abused because he is so weird, and maybe some cast a spell to wipe his memory of it (the spell comes up after Harry suggests repressed memories are BS pseudoscience, which again, is true, BUT WHY IS THIS HAPPENING IN THIS STORY?)

Harry uses his ‘rationalist art’ (literally “Harry’s rationalist skills begin to boot up again”) to suggest an alternative explanation

"I’m too smart, Professor. I’ve got nothing to say to normal children. Adults don’t respect me enough to really talk to me. And frankly, even if they did, they wouldn’t sound as smart as Richard Feynman, so I might as well read something Richard Feynman wrote instead. I’m isolated, Professor McGonagall. I’ve been isolated my whole life. Maybe that has some of the same effects as being locked in a cellar. And I’m too intelligent to look up to my parents the way that children are designed to do. My parents love me, but they don’t feel obliged to respond to reason, and sometimes I feel like they’re the children - children who won’t listen and have absolute authority over my whole existence. I try not to be too bitter about it, but I also try to be honest with myself, so, yes, I’m bitter.

After that weird back and forth the chapter moves on, Harry goes and buys a wand, and then from conversation begins to suspect that the Voldemort might still be alive. When McGonagall doesn’t want to tell him more, “a terrible dark clarity descended over his mind, mapping out possible tactics and assessing their consequences with iron realism.”

This leads Hariezer to blackmail McGonagall- he won’t tell people Voldemort is still alive if she tells him about the prophecy. Its another weird bit in a chapter absolutely brimming with weird bits.

Finally they go to buy a trunk, but they are low on gold (note to the reader: here would have been an excellent example of the planning fallacy). But luckily Hariezer had taken extra from the vault. Rather than simply saying “oh, I brought some extra”, he says

So - suppose I had a way to get more Galleons from my vaultwithout us going back to Gringotts, but it involved me violating the role of an obedient child. Would I be able to trust you with that, even though you’d have to step outside your own role as Professor McGonagall to take advantage of it?

So he logic-chops her into submission, or whatever, and they buy the trunk.

This chapter for me was incredibly uncomfortable. McGonagall behaves very strangely so she can act as a foil for all of Hariezer’s rants, and when the line between Hariezer and Eliezer fell away completely, it felt a bit oddly personal.

Oh, right, there was also a conversation about the rule against underage magic

"Ah,” Harry said. “That sounds like a very sensible rule. I’m glad to see the wizarding world takes that sort of thing seriously.”

I can’t help but draw parallels to the precautions Yud wants with AI.

Summary: Harry finished buying school supplies(I hope).

HPMOR 7: Uncomfortable elitism, and rape threats

A brief warning: Like always I’m typing this thing on my phone, so strange spell-check driven typos almost certainly abound. However, I’m also pretty deep in my cups (one of the great privileges of leaving academia is that I can afford to drink Lagavulin more than half my age like its water. The downside is no longer get to teach and so must pour my wisdom out in the form of a critique of a terrible fan fiction, that all of one person is probably reading)

This chapter took the weird tonal shift from the last chapter and just ran with it.

We are finally heading toward Hogwarts,so the chapter opens with the classic platform 9 3/4 bit from the book. And then it takes an uncomfortable elitist tone: Harry asks Ron Weasley to call him “Mr. Spoo” so that he can remain incognito, and Ron, a bit confused says “Sure Harry.” That one slip up allows Hariezer to immediately peg Ron as an idiot. In the short conversation that follows he mentally thinks of Ron as stupid several times and then he tries to explain to Ron why Quidditch is a stupid game.

It is my understanding from a (rather loose reading of the) books, that like cricket, quidditch games last weeks, EVEN MONTHS. In a game lasting literally weeks, one team could conceivably be up by 15 goals. In one of the books, I believe an important match went against the team that caught the snitch in one of the books. This is not to entirely defend quidditch, but it doesn’t HAVE to be an easy target. I think part of the ridicule that quidditch gets is that non-British/non-Indian audiences are perhaps not capable of appreciating that there are sports (cricket) that are played out over weeks that are very high scoring.

Either way, the WAY that Hariezer attacks quidditch is at expense of Ron, and it feels like a nerd sneering at a jock for liking sports. But thats just the lead up to the cloying nerd-elitism. Draco comes over, Hariezer is quick to rekindle that budding friendship, and we get the following conversation about Ron:

If you didn’t like him,” Draco said curiously, “why didn’t you just walk away?” "Um… his mother helped me figure out how to get to this platform from the King’s Cross Station, so it was kind of hard to tell him to get lost. And it’s not that I hate this Ron guy,” Harry said, “I just, just…” Harry searched for words. "Don’t see any reason for him to exist?" offered Draco. "Pretty much.

Just cloying, uncomfortable levels of nerd-elitism.

Now that Hariezer and Draco are paired back up, they can have lot of uncomfortable conversations. First, Draco shares something only slightly personal, which leads to this

"Why are you telling me that? It seems sort of… private…” Draco gave Harry a serious look. “One of my tutors once said that people form close friendships by knowing private things about each other, and the reason most people don’t make close friends is because they’re too embarrassed to share anything really important about themselves.” Draco turned his palms out invitingly. “Your turn?”

Hariezer consider this a masterful use of the social psychology idea of reciprocity (which just says if you do something for someone, they’re likely to do it for you). Anyway, this exchange is just a lead up to this, which feels like shock value for no reason:

"Hey, Draco, you know what I bet is even better for becoming friends than exchanging secrets? Committing murder." "I have a tutor who says that," Draco allowed. He reached inside his robes and scratched himself with an easy, natural motion. "Who’ve you got in mind?" Harry slammed The Quibbler down hard on the picnic table. “The guy who came up with this headline.” Draco groaned. “Not a guy. A girl. A ten-year-old girl, can you believe it? She went nuts after her mother died and her father, who owns this newspaper, is convinced that she’s a seer, so when he doesn’t know he asks Luna Lovegood and believes anything she says.” … Draco snarled. “She has some sort of perverse obsession about the Malfoys, too, and her father is politically opposed to us so he prints every word. As soon as I’m old enough I’m going to rape her.”

So, Hariezer is joking about the murder (its made clear later), but WHAT THE FUCK IS HAPPENING? These escalating friendship-tests feel contrived, reciprocity is effective when you don’t make demands immediately, which is why when you get a free sample at the grocery store the person at the counter doesn’t say “did you like that? Buy this juice or we won’t be friends anymore.” This whole conversation feels ham fisted, Hariezer is consistently telling us about all the manipulative tricks they are both using. Its less a conversation and more people who just sat through a shitty marketing seminar trying to try out what they learned. WITH RAPE.

After that, Draco has a whole spiel about how the legal system of the wizard world is in the pocket of the wealthy, like the Malfoys, which prompts Hariezer to tell us that only countries descended from the enlightenment have law-and-order (and I take it from comments that originally there was some racism somewhere in here that has since been edited out). Note: the wizarding world HAS LITERAL MAGIC TRUTH POTIONS, but we are to believe our enlightenment legal system works better? This seems like an odd, unnecessary narrative choice.

Next, Hariezer tries to recruit Draco to the side of science with this:

Science doesn’t work by waving wands and chanting spells, it works by knowing how the universe works on such a deep level that you know exactly what to do in order to make the universe do what you want. If magic is like casting Imperio on someone to make them do what you want, then science is like knowing them so well that you can convince them it was their own idea all along. It’s a lot more difficult than waving a wand, but it works when wands fail, just like if the Imperiusfailed you could still try persuading a person.

I’m not sure why you’d use persuasion/marketing as a shiny metaphor for science, other than its the theme of this chapter. ”If you know science you can manipulate people as if you were literally in control of them” seems like a broad, and mostly untrue claim. AND IT FOLLOWS IMMEDIATELY AFTER HARRY EXPLAINED THE MOON LANDING TO DRACO. Science can take you to the fucking moon, maybe thats enough.

This chapter also introduces comed-tea, a somewhat clever pun drink. If you open a can, at some point you’ll do a spit-take before finishing it. I’m not sure the point of this new magical introduction, hopefully Hariezer gets around to exploring it (seriously, hopefully Hariezer begins to explore ANYTHING to do with the rules of magic. I’m 7 full chapters in and this fanfic has paid lip service to science without using it to explore magic at all).

Chapter summary: Hariezer makes it to platform 9 3/4, ditches Ron as somehow sub-human. Has a conversation with Draco that is mostly framed as conversation-as-explicit manipulation between Hariezer and Draco, and its very ham-fisted, but luckily Hariezer assures us its actual masterful manipulation, saying things like this, repeatedly:

And Harry couldn’t help but notice how clumsy, awkward, graceless his attempt at resisting manipulation / saving face / showing off had appeared compared to Draco.

Homework for the interested reader: next time you are meeting someone new, share something embarrassingly personal and then ask them immediately to reciprocate, explicitly saying ‘it’ll make us good friends.’ See how that works out for you.

WHAT DO PEOPLE SEE IN THIS? It wouldn’t be so bad, but we are clearly supposed to identify with Hariezer, who looks at Draco as someone he clearly wants on his side, and who instantly dismisses someone (with no “Bayesian updates” whatsoever as basically less than human). I’m honestly surprised that anyone read past this chapter. But I’m willing to trudge on, for posterity. Two more glasses of scotch, and then I start chapter 8. I’m likely to need alcohol to soldier on from here on out.

Side note: I’ve consciously not mentioned all the “take over the world” Hariezer references, but there are probably 3 or 4 per chapter. They seem at first like bad jokes, but they keep getting revisited so much that I think Hariezer’s explicit goal is perhaps not curiosity driven (figure out the rules of magic), but instead power driven (find out the rules of magic in order to take over the world). He assures Draco he really is Ravenclaw, but if he were written with consistency maybe he wouldn’t need to be? Hariezer doesn’t ask questions (like I would imagine a Ravenclaw would), he gives answers. Thus far, he has consistently decided the wizarding world has nothing to teach him. Arithmancy books he finds only go up to trigonometry, etc. He certainly has shown only limited curiosity this far. Its unclear to me why a curiosity driven, scientist character would feel a strong desire to impress, manipulate Draco Malfoy, as written here. This is looking less like a love-song to science, and more a love-song to some weird variant of How to Win Friends and Influence People.

A few Observations Regarding Hariezer Yudotter

After drunkenly reading chapters 8,9 and 10 last night (I’ll get to the posts soon, hopefully), I was flipping channels and somehow settled on an episode of that old TV show with Steve Urkel (bear with me, this will get relevant in a second).

In the episode, the cool kid Eddie gets hustled at billiards, and Urkel comes in and saves the day because his knowledge of trigonometry and geometry makes him a master at the table.

I think perhaps this is a common dream of the science fetishist- if only I knew ALL OF THE SCIENCE I would be unstoppable at everything. Hariezer Yudotter is a sort of wish fulfillment character of that dream. Hariezer isn’t motivated by curiosity at all really, he wants to grow his super-powers by learning more science. Its why we can go 10 fucking chapters without Yudotter really exploring much in the way of the science of magic (so far I count one lazy paragraph exploring what his pouch can do, in 10 chapters). Its why he constantly describes his project as “taking over the world.” And its frustrating, because this obviously isn’t a flaw to be overcome its part of Yudotter’s “awesomeness.”

I have a phd in a science, and it has granted me these real world super-powers:

1. I fix my own plumbing, do my own home repairs,etc.
2. I made a robot of legos and a raspberry pi that plays connect 4 incredibly well (my robot sidekick, I guess)
3. Via techniques I learned in the sorts of books that in fictional world Hariezer uses to be a master manipulator, I can optimize ads on webpages such that up to 3% of people will click on them (that is, seriously, the power of influence in reality. Not Hannibal Lector but instead advertisers trying to squeeze an extra tenth of a percent on conversions), for which companies sometime pay me
4. If you give me a lot of data, I can make a computer find patterns in it, for which companies sometimes pay me.

Thats basically it. Back when I worked in science, I spent nearly a decade of my life calculating various background processes related to finding a Higgs boson, and I helped design some software theorists now use to calculate new processes quickly. These are the sorts of projects scientists work on, and most days its hard work and total drudgery, and there is no obvious ‘instrumental utility’- BUT I REALLY WANTED TO KNOW IF THERE WAS A HIGGS FIELD.

And thats why I think the Yudotter character doesn’t feel like a scientist- he wants to be stronger, more powerful, take over the world, but he doesn’t seem to care what the answers are. Its all well and good to be driven, but most importantly, you have to be curious.

HPMOR 8: Back to the inoffensive chapters of yesteryear

And a dramatic tonal shift and we are back to a largely inoffensive chapter.

There is another lesson in this chapter, this time the lesson is confirmation bias (though Yudkowsky/Hariezer refer to it as ‘positive bias’), but once again, the pedagogical choices are strange. As Hariezer winds into his lesson to Hermione, she thinks the following:

She was starting to resent the boy’s oh-so-superior tone…but that was secondary to finding out what she’d done wrong.

So Yudkowsky knows his Hariezer has a condescending tone, but he runs with it. So as a reader, if I already know the material I get to be on the side of truth, and righteousness and I can condescend to the simps with Hariezer, OR, I don’t know the material, and then Hermione is my stand in, and I have to swallow being condescended to in order to learn.

Generally, its not a good idea when you want to teach someone something to immediately put them on the defensive- I’ve never stood in front of a class, or tutored someone by saying

"The sad thing is you probably did everything the book told you to do… unless you read the very, very best sort of books, they won’t quite teach you how to do science properly…

And Yudkowsky knows enough that his tone is off-putting to point to it. So I wonder- is this story ACTUALLY teaching people things? Or is it just a way for people who already know some of the material to feel superior to Hariezer’s many foils? Do people go and read the sequences so that they can move from Hariezer-foil, to Hariezer’s point of view? (these are not rhetorical questions, if anyone has ideas on this).

As for the rest of the chapter- its good to see Hermione merits as human, unlike Ron. There is a strange bit in the chapter where Neville asks a Gryffindor prefect to find his frog, and the prefect says no (why? what narrative purpose does this serve?).

Chapter summary: Hariezer meets Neville and Hermione on the train to Hogwarts. Still no actual exploration of magic rules. None of the fun candy of the original story.

HPMOR 9 and 10

EDIT: I made a drunken mistake in this one, see this response. I do think my original point still goes through because the hat responds to the attempted blackmail with:

I know you won’t follow through on a threat to expose my nature, condemning this event to eternal repetition. It goes against the moral part of you too strongly, whatever the short-term needs of the part of you that wants to win the argument.

So the hat doesn’t say “I don’t care about this,” the hat says “you won’t do it.” My point is, however, substantially weakened.

END EDIT

Alright, the lagavulin is flowing, and I’m once more equipped to pontificate.

These chapters are really one chapter split in two. I’m going to use them to argue against Yudkowsky’s friendly AI concept a bit. There is this idea, called ‘orthgonality’ that says that an AIs goals can be completely independent of its intelligence. So you can say ‘increase happiness’ and this uber-optimizer can tile the entire universe with tiny molecular happy faces, because its brilliant at optimizing but incapable of evaluating its goals. Just setting the stage for the next chapter.

In this chapter, Harry gets sorted. When the sorting hat hits his head, Harry wonders if its self-aware, which because of some not-really-explained magical hat property, instantly makes the hat self-aware. The hat finds being self aware uncomfortable, and Hariezer worries that he’ll destroy an intelligent being when the hat is removed. The hat assures us that the hat cares only for sorting children. As Hariezer notes

It [the hat] was still imbued with only its own strange goals…

Even still, Hariezer manages to blackmail the hat- he threatens to tell all the other kids to wonder if the hat is self-aware. The hat concedes to the demand.

So how does becoming self-aware over and over effect the hat’s goals of sorting people? It doesn’t. The blackmail should fail. Yudkowsky imagines that the minute it became self-aware, the hat couldn’t help but pick up some new goals. Even Yudkowsky imagines that becoming self-aware will have some effects on your goals.

This chapter also has some more weirdly personal seeming moments when the line between Yudkowsky’s other writing and HPMOR breaks down completely.

Summary: Harry gets sorted into ravenclaw.

I am immensely frustrated that I’m 10 chapters into this thing, and we still don’t have any experiments regarding the rules of magic.

HPMOR 11

Chapter 11 is “omake.” This is a personal pet-peeve of mine, because I’m a crotchety old man at heart. The anime culture takes Japanese words, for which we have perfectly good english words, and snags them (kawaii/kawaisa is a big one). Omake is another one. I have nothing against Japanese (I’ve been known to speak it), just don’t like unnecessary loaner words in general. I know this is my failing, BUT I WANT SO BAD TO HOLD IT AGAINST THIS FANFIC.

Either way, I’m skipping the extra content, because I can only take so much.

HPMOR 12

Nothing much in this chapter. Dumbledore gives his post-dinner speech.

Harry cracks open a can of comed-tea and does the requisite spit-take when Dumbledore starts his dinner speech with random nonsense. He considers casting a spell to make his sense of humor very specific, and then he can use comed-tea to take over the world.

Chapter summary: dinner is served and eaten

HPMOR 13: Bill and Teds Excellent Adventure

There is a scene in Bill and Ted’s Excellent Adventure, toward the end, where they realize their time machine gives them super-human powers. They need to escape a jail, so they agree to get the keys later and travel back in time and hide them, and suddenly there the keys are. After yelling to remember a trash can, they have a trash can to incapacitate a guard with,etc. They can do anything they want.

Anyway, this chapter is that idea, but much longer. The exception is that we don’t know there has been a time machine (actually, I don’t KNOW for sure thats what it is, but the Bill and Ted fan in me says thats what happened this chapter, I won’t find out until next chapter. If I were a Bayesian rationalist, I would say that the odds ratio is pi*10^^^^3 in my favor. ).

Hariezer wakes up and finds a note saying he is part of a game. Everywhere he looks, as if by magic he finds more notes, deducting various game “points,” and some portraits have messages for him. The notes lead him to a pack of bullies beating up some hufflepuffs, and pies myseriously appears for Hariezer to attack with. The final note tells him to go to Mcgonagall’s office, and the chapter ends.

I assume next chapter, Hariezer will recieve his time machine and future Hariezer will use it to set up the “game” as a prank on past Hariezer. Its a clever enough chapter.

This chapter was actually decent, but what the world really needs is Harry Potter/Bill and Ted’s Excellent Adventure cross over fiction.

HPMOR 14: Lets talk about computability

This chapter has created something in my brain like Mr Burn’s Three Stooges Syndrome. So many things I want to talk about, I don’t know where to start!

First, I was slightly wrong about last chapter. It wasn’t a time machine Hariezer used to accomplish the prank in the last chapter, it was a time machine AND an invisibility cloak. BIll and Ted did not lead me astray.

On to the chapter- Hariezer gets a time machine (Hariezer lives 26 hour days, so he needs he is given a time turner to correct his sleep schedule) this prompts this:

Say, Professor McGonagall, did you know that time-reversed ordinary matter looks just like antimatter? Why yes it does! Did you know that one kilogram of antimatter encountering one kilogram of matter will annihilate in an explosion equivalent to 43 million tons of TNT? Do you realise that I myself weigh 41 kilograms and that the resulting blast would leave A GIANT SMOKING CRATER WHERE THERE USED TO BE SCOTLAND?

Credit where credit is due- this is correct physics. In fact, its completely possible (though a bit old-fashioned and unwieldy), to treat quantum field theory such that all anti-matter is simply normal matter moving backward in time. Here is an example, look at this diagram:

If we imagine time moving from the bottom of the diagram toward the top, we see two electrons traveling forward in time, and exchanging a photon and changing directions.

But now imagine time moves left to right in the diagram instead- what we see is one electron and one positron coming together and destroying each other, and then a new pair forming from the photon. BUT, we COULD say that what we are seeing is really an electron moving forward in time, and an electron moving backward in time. The point where they “disappear” is really the point where the forward moving electron changed directions and started moving backward in time.

This is probably very confusing, if anyone wants a longer post about this, I could probably try for it sober. I need to belabor this though- the takeaway point I need you to know- the best theory we have of physics so far can be interpreted as having particles that change direction in time, AND HARIEZER KNOWS THIS AND CORRECTLY NOTES IT.

Why is this important? Because a paragraph later he says this:

You know right up until this moment I had this awful suppressed thought somewhere in the back of my mind that the only remaining answer was that my whole universe was a computer simulation like in the book Simulacron 3 but now even that is ruled out because this little toy ISN’T TURING COMPUTABLE! A Turing machine could simulate going back into a defined moment of the past and computing a different future from there, an oracle machine could rely on the halting behavior of lower-order machines, but what you’re saying is that reality somehow self-consistently computes in one sweep using information that hasn’t… happened… yet..

This is COMPLETE NONSENSE (this is also terrible pedagogy again, either you know what Turing computable means are you drown in jargon). For this discussion, Turing computable means ‘capable of being calculated using a computer’ The best theory of physics we have (a theory Hariezer already knows about) allows the sort of thing that Hariezer is complaining about. Both quantum mechanics and quantum field theory are Turing computable. Thats not to say Hariezer’s time machine won’t require you to change physics a bit- you definitely will have to, but its almost certainly computable.

Now computable does not mean QUICKLY computable (or even feasibly computable). The new universe might not be computable in polynomial time (quantum field theory may not be, at least one problem with in it, the fermion sign problem, is not).

I don’t think the time machine makes P = NP either. Having a time machine will allow you to speed up computations (you could wait until a computation was done, and then send the answer back in time). However, Hariezer’s time machine is limited, it can only be used to move back 6 hours total, and can only be used 3 time in a day, so I don’t think it could generally solve an NP-complete problem in polynomial time (after your 6 hour head start is up, things proceed at the original scaling). If you don’t know anything about computational complexity, I guess if I get enough asks I can explain it in another, non-potter post.

But my point here is- the author here is supposedly an AI theorist. How is he flubbing computability stuff? This should be bread and butter stuff.

I have so much more to say about this chapter. Another post will happen soon.

Edit: I was’t getting the P = NP thing, but I get the argument now (thanks Nostalgebraist), the idea is that you say “I’m going to compute some NP problem and come back with the solution” and then ZIP, out pops another you from the time machine, and hands you a slip of paper with the answer on it. Now you have 6 hours to verify the calculation, and then zip back to give it your former self.

But any problem in NP is checkable in P, so for any problem small enough to be checkable in 6 hours (which is a lot of problems, including much of NP), is now computable in no time at all. Its not a general P = NP, but its much wider in applicability than I was imagining.

HPMOR 14 continued: Comed Tea, Newcomb’s problem, science

One of the odd obsession’s of LessWrong is an old decision theory problem called Newcomb’s Paradox. It goes like this- a super intelligence that consistently predicts correctly challenges you to a game. There are two boxes, A and B. And you are allowed to take one or both boxes.

Inside box A is $10, and inside box B the super intelligence has already put $10,000 IF AND ONLY IF it predicted you will only take box B. What box should you take?

The reason this is a paradox is that one group of people (call them causal people) might decide that because the super intelligence ALREADY made its call, you might as well take both boxes. You can’t alter the past prediction.

Other people (call these LessWrongians) might say, ‘well, the super intelligence is always right, so clearly if I take box B I’ll get more money’. This camp includes LessWrongians. Yudkowsky himself had tried to formalize a decision theory that picks box B, that involved allowing causes to propagate backward in time.

A third group of people (call them ‘su3su2u1 ists’) might say “this problem is ill-posed. The idea of the super-intelligence might well be incoherent, depending on your model of how decisions are made,” Here is why- imagine human decisions can be quite noisy. For instance, what if I flip an unbiased coin to decide which box to take. Now the super-intelligence can only have had a 50/50 chance to successfully predict which box I’d take, which contradicts the premise.

There is another simple way to show the problem is probably ill-posed. Imagine another we take another super-intelligence of the same caliber as the first (call the first 1 and the second 2). 1 offers the same game to 2, and now 2 takes both boxes if it predicts that 1 put the money in box B. It takes only box B if 1 did not put the money in box B. Obviously, either intelligence 1 is wrong, or intelligence 2 is wrong, which contradicts the premise, so the idea must be inconsistent (note, you can turn any person into super-intelligence number 2 by making the boxes transparent).

Anyway, Yudkowsky has a pet decisions theory he has tried to formalize that allows causes to propagate backward in time. He likes this approach because you can get the LessWrongian answer to Newcomb every time. The problem is, his formalism has all sorts of problems with inconsistency because of the issues I raised about the inconsistency of a perfect predictor.

Why do I bring this up? Because Hariezer decides in this chapter that comed-tea MUST work by causing you to drink it right before something spit-take worthy happens. The tea predicts the humor, and then magics you into drinking it. Of course, he does no experiments to test this hypothesis at all (ironic that just a few chapters ago he lecture Hermione about only doing 1 experiment to test her idea).

So unsurprisingly perhaps, the single most used magic item in the story thus far is a manifestation of Yudkowsky’s favorite decision theory problem.

And my final note from this chapter- Hariezer drops this on us, regarding the brain’s ability to understand time travel:

Now, for the first time, he was up against the prospect of a mystery that was threatening to be permanent…. it was entirely possible that his human mind never could understand, because his brain was made of old-fashioned linear-time neurons, and this had turned out to be an impoverished subset of reality.

This seems to misunderstand the nature of mathematics and its relation to science. I can’t visualize a 4 dimensional curved space,certainly not the way I visualize 2 and 3d objects. But that doesn’t stop me from describing it and working with it as a mathematical object.

Time is ALREADY very strange and impossible to visualize. But mathematics allows us to go beyond what our brain can visualize to create notations and languages that let us deal with anything we can formalize and that has consistent rules. Its amazingly powerful.

I never thought I’d see Hariezer Yudotter, who just a few chapters back was claiming science could let us perfectly manipulate and control people (better than an imperio curse, or whatever the spell that lets you control people) argue that science/mathematics couldn’t deal with linear time.

I hope that this is a moment where in later chapters we see growth from Yudotter, and he revisits this last assumption. And I hope he does some experiments to test his comed-tea hypothesis. Right now it seems like experiments are things Hariezer asks people around him to do (so they can see things his way), but for him pure logic is good enough.

Chapter summary: I drink three glasses of scotch. Hariezer gets a time machine.

HPMOR 15: In which, once again, I want more science

I had a long post but the internet ate it earlier this week, so this is try 2. I apologize in advance, this blog post is mostly me speculating about some magi-science.

This chapter begins the long awaited lessons in magic. The topic of today’s lesson consists primarily of one thing, don’t transfigure common objects into food or drink.

Mr. Potter, suppose a student Transfigured a block of wood into a cup of water, and you drank it. What do you imagine might happen to you when the Transfiguration wore off?” There was a pause. “Excuse me, I should not have asked that of you, Mr. Potter, I forgot that you are blessed with an unusually pessimistic imagination -“ "I’m fine," Harry said, swallowing hard. "So the first answer is that I don’t know,” the Professor nodded approvingly, “but I imagine there might be… wood in my stomach, and in my bloodstream, and if any of that water had gotten absorbed into my body’s tissues - would it be wood pulp or solid wood or…” Harry’s grasp of magic failed him. He couldn’t understand how wood mapped into water in the first place, so he couldn’t understand what would happen after the water molecules were scrambled by ordinary thermal motions and the magic wore off and the mapping reversed.

We get a similar warning regarding transfiguring things into any gasses or liquids:

You will absolutely never under any circumstances Transfigure anything into a liquid or a gas. No water, no air. Nothing like water, nothing like air. Even if it is not meant to drink. Liquid evaporates, little bits and pieces of it get into the air.

Unfortunately, once again, I want the author to take it farther. Explore some actual science! What WOULD happen if that wood-water turned back into wood in your system?

So lets take a long walk off a short speculative pier together, and try to guess what might happen. First, we’ll assume magic absorbs any major energy differences and smooths over any issues at the time of transition. Otherwise when you magic in a few wood large wood molecules in place of much smaller water molecules, there will suddenly be lots of energy from the molecules repelling each other (this is called a steric mismatch) which will likely cause all sorts of problems (like a person exploding).

To even begin to answer, we have to pick a rule for the transition. Lets assume each water molecule turns into one “wood molecule” (wood is ill-defined on a molecular scale, its made up lots of shit. However, that shit is mostly long hydrocarbon chains called polysaccharides.)

So you’d drink the water, which gets absorbed pretty quickly by your body (any thats lingering in your gut unabsorbed will just turn into more fiber in your diet). After awhile, it would spread through your body, be taken up by your cells, and then these very diffuse water molecules would turn into polysaccharides. Luckily for you, your body probably knows how to deal with this, polysaccharides are hanging out all over your cells anway. Maybe somewhat surprisingly, you’d probably be fine. I think for lots of organic material, swapping out one organic molecule with another is likely to not harm you much. Of course, if the thing you swap in is poison, thats another story.

Now, I’ve cheated somewhat- I could pick another rule where you’d definitely die. Imagine swapping in a whole splinter of wood for each water molecule. You’d be shredded. The details of magic matter here, so maybe a future chapter will give us the info needed to revisit this.

What if instead of wood, we started with something inorganic like gold? If the water molecules turn into elemental gold (and you don’t explode from steric mismatches mentioned above), you’d be fine as long as the gold didn’t ionize. Elemental gold is remarkably stable, and it takes quite a bit of gold to get any heavy metal poisoning from it.

On the other hand, if it ionizes you’ll probably die. Gold salts (which split into ionic gold + other stuff in your system) have a semi-lethal doses (the dose that kills half of the people who take it) of just a few mg per kg, so a 70 kg person couldn’t survive more than 5g or so of the salt, which is even less ionic gold. So in this case, as soon as the spell wore off you’d start to be poisoned. After a few hours, you’d probably start showing signs of liver failure (jaundice, etc).

Water chemistry/physics is hard, so I have no idea if the gold atoms will actually ionize. Larger gold crystals definitely would not, and they are investigating using gold nanoparticles for medicine, which are also mostly non-toxic. However, individual atoms might still ionize.

What if we don’t drink the water? What if we just get near a liquid evaporating? Nothing much at all, as it turns out. Evaporation is a slow process, as is diffusion.

Diffusion constants are usually a few centimeters^2 per second, and diffusion is a slow process that moves forward with the square root of time (to move twice as far it takes 4 times as much time).

So even if the transformation into water lasts a full hour, a single water molecule that evaporates from the glass will travel less than 100 centimeters! So unless you are standing with your face very close to the glass, you are unlikely to encounter even a single evaporated molecule. Even with your face right near the glass, that one molecule will mostly likely just be breathed in and breathed right back out. You have a lot of anatomic dead-space in your lungs in which no exchange takes place, and the active area is optimized for picking up oxygen.

So how about transfiguring things to a gas? What happens there? Once again, this will depend on how we choose the rules of magic. When you make the gas, does it come in at room temperature and pressure? If so, this sets the density. Then you can either bring in an equal volume of gas to the original object with very few molecules, or you bring in an equal number of molecules, with a very large density.

At an equal number of molecules, you’ll get hundreds of liters of diffuse gas. Your lungs are only hold about 5 liters, so you are going to get a much smaller dose then you’d get from the water (a few percent at best), where all the molecules get taken up by your body. Also, your lungs won’t absorb most of the gas, much will get blown back out, further lowering the dose.

If its equal volume to the original object, then there will be very few gas molecules over a small area, and the diffusion argument applies- unless you get very near where you created the gas you aren’t likely at all to breathe any in.

Thus concludes a bit of speculative magi-science guess work. Sorry if I bored you.

Anyway- this chapter, I admit, intrigued me enough to spend some time thinking about what WOULD happen if something un-transfigured inside you. Not a bad chapter, really, but it again feels a tad lazy. We get some hazy worries about liquids evaporating (SCIENCE!) but no order-of-magnitude estimate about whether or not it matters (does not, unless maybe you boiled the liquid you made). There are lots of scientific ideas the author could play with, but they just get set aside.

As for the rest of the chapter, Hariezer gets shown up by Hermione, who is out-performing him and has already read her school books. A competition for grades is launched.

HPMOR 16: Shades of Ender’s game

I apologize for the longish break from HPMOR, sometimes my real job calls.

This chapter mostly comprises Hariezer’s first defense against the dark arts class. We meet the ultra-competent Quirrel (although I suppose like in the original its really the ultra-competent Voldemort) for the first time.

The lessons open with a bit of surprising anti-academic sentiment- Quirrel gives a long speech about how you needn’t learn to defend yourself against anything specific in the wizarding world because you could either magically run away or just use the instant killing spell, so the entire “Ministry-mandated” course with its “useless” textbooks is unnecessary. Of course, this comes from the mouth of the ostensible bad guy, so its unclear how much we are supposed to be creeped out by this sentiment (though Hariezer applauds).

After this, we get to the lesson. After teaching a light attack spell, Quirrel asks Hermoine (who mastered it fastest) to attack another student. She refuses, so Quirrel moves on to Malfoy who is quick to acquiesce by shooting Hermoine.

Then Quirrel puts Hariezer on the spot and things get sort of strange. When asked for unusual combat uses of everyday items, Hariezer comes up with a laundry list of outlandish ways to kill people, which leads Quirrel to observe that for Hariezer Yudotter nothing is defensive- he settles only for the destruction of his enemy. This feels very Ender’s game (Hariezer WINS, and that makes him dangerous), and sort of a silly moment.

Chapter summary: weirdly anti-academic defense against the dark arts lesson. We once more get magic, but no rules of magic.

HPMOR 17: Introducing Dumbledore, and some retreads on old ideas

This chapter opens with a little experiment with Hariezer trying to use the time turner to verify an NP-complete problem, as we discussed in a previous chapter section. Since its old ground, we won’t retread it.

From here, we move on to the first broomstick lesson, which proceeds much like the book, only with shades of elitism. Hariezer drops this nugget on us:

There couldn’t possibly be anything he could master on the first try which would baffle Hermione, and if there was and it turned out to be broomstick riding instead of anything intellectual, Harry would just die.

Which feels a bit like the complete dismissal of Ron earlier. So the anti-jock Hariezer, who wouldn’t be caught dead being good at broomsticking doesn’t get involved in racing around to try to get Neville’s remember all, instead the entire class ends up in a stand off, wands drawn. So Hariezer challenges the Slytherin who has it to a strange duel. Using his time turner in proper Bill and Ted fashion, he hides a decoy remember all and wins. Its all old stuff at this point, I’m starting to worry there is nothing new under the sun- more time turner, more Hariezer winning (in case we don’t get it, there is a conversation with McGonagall where Hariezer once more realizes he doesn’t even consider NOT winning).

AND THEN we meet Dumbledore, who is written as a lazy man’s version of insane. He’ll say something insightful, drop a Lord of the Ring’s quote and then immediately do something batshit. One moment he is trying to explain that Harry can trust him, the next he is setting a chicken on fire (yes this happens). In one baffling moment, he presents Hariezer with a big rock, and this exchange happens:

So… why do I have to carry this rock exactly?” "I can’t think of a reason, actually," said Dumbledore. "…you can’t." Dumbledore nodded. “But just because I can’t think of a reason doesn’t mean there is no reason.” "Okay," said Harry, "I’m not even sure if I should be saying this, but that is simply not the correct way to deal with our admitted ignorance of how the universe works."

Now, if someone gave you a large heavy rock and said “keep this on you, just in case” how would you begin to tell them they’re wrong? Here is Hariezer’s approach:

How can I put this formally… um… suppose you had a million boxes, and only one of the boxes contained a diamond. And you had a box full of diamond-detectors, and each diamond-detector always went off in the presence of a diamond, and went off half the time on boxes that didn’t have a diamond. If you ran twenty detectors over all the boxes, you’d have, on average, one false candidate and one true candidate left. And then it would just take one or two more detectors before you were left with the one true candidate. The point being that when there are lots of possible answers, most of the evidence you need goes into just locating the true hypothesis out of millions of possibilities - bringing it to your attention in the first place. The amount of evidence you need to judge between two or three plausible candidates is much smaller by comparison. So if you just jump ahead without evidence and promote one particular possibility to the focus of your attention, you’re skipping over most of the work.

Thank God Hariezer was able to use his advanced reasoning skills to make an analogy with diamonds in boxes to explain WHY THE IDEA THAT CARRYING A ROCK AROUND FOR NO REASON IS STUPID. This was the chapter’s rationality idea- seriously, its like Yudkowsky didn’t even try on this one.

Chapter summary: Hariezer sneers at broomstick riding, some (now standard) time turner hijinks, Hariezer meets a more insane than wise Dumbledore

HPMOR 18: What?

I wanted to discuss the weird anti-university/school-system under currents of the last few chapters, but I started into chapter 18 and it broke my brain.

This chapter is absolutely ludicrous. We meet Snape for the first time, and he behaves as you’d expect from the source material. He makes a sarcastic remark and asks Hariezer a bunch of questions Hariezer does not know the answer to.

This leads to Hariezer flipping out:

The class was utterly frozen. "Detention for one month, Potter," Severus said, smiling even more broadly. "I decline to recognize your authority as a teacher and I will not serve any detention you give." People stopped breathing. Severus’s smile vanished. “Then you will be -” his voice stopped short. "Expelled, were you about to say?" Harry, on the other hand, was now smiling thinly. "But then you seemed to doubt your ability to carry out the threat, or fear the consequences if you did. I, on the other hand, neither doubt nor fear the prospect of finding a school with less abusive professors. Or perhaps I should hire private tutors, as is my accustomed practice, and be taught at my full learning speed. I have enough money in my vault. Something about bounties on a Dark Lord I defeated. But there are teachers at Hogwarts who I rather like, so I think it will be easier if I find some way to get rid of you instead.”

Think about this- THE ONLY THINGS SNAPE HAS DONE are make a snide comment and ask Hariezer a series of questions he doesn’t know the answer to.

The situation continues to escalate, until Hariezer locks himself in a closet and uses his invisibility cloak and time turner to escape the classroom.

This leads to a meeting with the headmaster where Hariezer THREATENS TO START A NEWSPAPER CAMPAIGN AGAINST SNAPE (find a newspaper interested in the ‘some students think professor too hard on them, for instance he asked Hariezer Yudotter 3 hard questions in a row’ story)

AND EVERYONE TAKES THIS THREAT SERIOUSLY, AS IF IT COULD DO REAL HARM. HARIEZER REPEATEDLY SAYS HE IS PROTECTING STUDENTS FROM ABUSE. THEY TAKE THIS THREAT SERIOUSLY ENOUGH THAT HARIEZER NEGOTIATES A TRUCE WITH SNAPE AND DUMBLEDORE. Snape agrees to be less demanding of discipline, Hariezer agrees to apologize.

Nowhere in this chapter does Hariezer consider that he deprived other students of the damn potions lesson. In his ruminations about why Snape keeps his job, he never considers that maybe Snape knows a lot about potions/is actually a good potions teacher.

This whole chapter is basically a stupid power struggle that requires literally everyone in the chapter to behave in outrageously silly ways. Hariezer throws a temper tantrum befitting a 2 year old, and everyone else gives him his way.

On the plus side, Mcgonagall locks down Hariezer’s time turner, so hopefully that device will stop making an appearance for awhile, its been the “clever” solution to every problem for several chapters now.

One more chapter this bad and I might have to abort the project.

HPMOR 19: I CAN’T EVEN… WHAT?…

I… this… what…

So there is a lot I COULD say here, about inconsistent characterization, ridiculously contrived events,etc. But fuck it- here is the key event of this chapter: Quirrel, it turns out, is quite the martial artist (because of course he is, who gives a fuck about genre consistency or unnecessary details, PILE IN MORE “AWESOME”). The lesson he claims to have learned from martial arts (at a mysterious dojo, because of course) that Hariezer needs to learn (as evidenced by his encounter with Snape) is how to lose.

How does Quirrel teach Hariezer “how to lose”? He calls Hariezer to the front, insists Hariezer not defend himself, and then has a bunch of slytherins beat the shit out of him.

Thats right- a character who one fucking chapter ago couldn’t handle being asked three hard questions in a row (ITS ABUSE, I’ll CALL THE PAPERS) submits to being literally beaten by a gang at a teacher’s suggestion.

An 11 year old kid, at a teachers suggestion, submits to getting beaten by a bunch of 16 year olds. All of this is portrayed in a positive light.

Ideas around chapter 19

In light of the recent anon, I’m going to attempt to give the people (person?) what they want. Also, I went from not caring if people were reading this, to being a tiny bit anxious I’ll lose the audience I unexpectedly picked up. SELLING OUT.

If we ignore the literal child abuse of the chapter, the core of the idea is still somewhat malignant. Its true throughout that Hariezer DOES have a problem with “knowing how to lose,” but the way you learn to lose is by losing, not by being ordered to take a beating.

Quirrell could have challenged Hariezer to a game of chess, he could have asked questions Hariezer didn’t know the answer to (as Snape did, which prompted the insane chapter 18), etc. But the problem is the author is so invested in Hariezer being the embodiment of awesome that even when he needs to lose for story purposes, to learn a lesson, Yudkowsky doesn’t want to let Hariezer actually lose at something. Instead he gets ordered to lose, and he isn’t ordered to lose at something in his wheel house, but in the “jock-stuff” repeatedly sneered at in the story (physical confrontation)

HPMOR 20: why is this chapter called Bayes Theorem?

A return to what passes for “normal.” No child beating in this chapter, just a long, boring conversation.

This chapter opens with Hariezer ruminating about how much taking that beating sure has changes his life. He knows how to lose now, he isn’t going to become dark lord now! Quirrell quickly takes him down a peg:

"Mr. Potter," he said solemnly, with only a slight grin, "a word of advice. There is such a thing as a performance which is too perfect. Real people who have just been beaten and humiliated for fifteen minutes do not stand up and graciously forgive their enemies. It is the sort of thing you do when you’re trying to convince everyone you’re not Dark, not -“

Hariezer protests, and we get

There is nothing you can do to convince me because I would know that was exactly what you were trying to do. And if we are to be even more precise, then while I suppose it is barely possible that perfectly good people exist even though I have never met one, it is nonetheless improbable that someone would be beaten for fifteen minutes and then stand up and feel a great surge of kindly forgiveness for his attackers. On the other hand it is less improbable that a young child would imagine this as the role to play in order to convince his teacher and classmates that he is not the next Dark Lord. The import of an act lies not in what that act resembles on the surface, Mr. Potter, but in the states of mind which make that act more or less probable

How does Hariezer take this? Does he point out “if no evidence can sway your priors, your priors are too strong?” or some other bit of logic-chop Bayes-judo? Nope, he drops some nonsensical jargon:

Harry blinked. He’d just had the dichotomy between the representativeness heuristic and the Bayesian definition of evidence explained to him by a wizard.

Where is Quirrell using bayesian evidence? He isn’t, he is neglecting all evidence because all evidence fits his hypothesis. Where does the representativeness heuristic come into play? It doesn’t.

The representative heuristic is making estimates based on how typical of a class something is. i.e. show someone a picture of a stereotypical ‘nerd’ and say “is this person more likely an english or a physics grad student?” The representative heuristic says “you should answer physics.” Its a good rule-of-thumb that psychologists think is probably hardwired into us. It also leads to some well-known fallacies I won’t get into here.

Quirrell is of course doing none of that- Quirrell has a hypothesis that fits anything Hariezer could do, so no amount of evidence will dissuade him.

After this, Quirrell and Hariezer have a long talk about science (because of course Quirrell too has a fascination with space travel). This leads to some real Less Wrong stuff.

Quirrell tells us that of course muggle scientists are dangerous because

There are gates you do not open, there are seals you do not breach! The fools who can’t resist meddling are killed by the lesser perils early on, and the survivors all know that there are secrets you do not share with anyone who lacks the intelligence and the discipline to discover them for themselves!

And of course, Hariezer agrees

This was a rather different way of looking at things than Harry had grown up with. It had never occurred to him that nuclear physicists should have formed a conspiracy of silence to keep the secret of nuclear weapons from anyone not smart enough to be a nuclear physicist

Which is a sort of weirdly elitist position- after all lots of nuclear physicists are plenty dangerous. Its not intelligence that makes you less likely to drop a bomb. But this fits the general Yudkowsky/AI fear- an open research community is less important than hiding dangerous secrets. This isn’t necessarily the wrong position, but its a challenging one that merits actual discussion.

Anyone who has done research can tell you how important the open flow of ideas is for progress. I’m of the opinion that the increasing privatization of science is actually slowing us down in a lot of ways by building silos around information. How much do we retard progress in order to keep dangerous ideas out of people’s hands? Who gets to decide what is dangerous? Who decides who gets let into “the conspiracy?” Intelligence alone is no guarantee someone won’t drop a bomb, despite how obvious it seems to Quirrell and Yudotter.

After this digression about nuclear weapons, we learn from Quirrell that he snuck into NASA and enchanted the Pioneer gold plaque that will “make it last a lot longer than it otherwise would.” Its unclear to me what that wear and tear Quirrell is protecting the plaque from. Hariezer suggest that Quirrell might have snuck a magic portrait or a ghost into the plaque, because nothing makes more sense then dooming an (at least semi) sentient being to a near eternity of solitary confinement.

Anyway, partway through this chapter, Dumbledore bursts in angry that Quirrell had Hariezer beaten. Hariezer defends him, etc. The resolution is that its agreed Hariezer will start learning to protect himself from mind readers.

Chapter summary- long, mostly boring conversation, peppered with some existential risk/we need to escape the planet rhetoric. Its also called Bayes theorem despite that theorem making no appearance whatsoever.

And a note on the really weird pedagogy- we now have Quirrell who in the books is possessed by Voldemort acting as a mouthpiece for the author. This seems like a bad choice, because at some point I assume we’ll there will be a reveal, and it will turn out the reader should have trusted Quirrell.

HPMOR 21: secretive science

So this chapter begins quite strangely- Hermione is worried that she is “bad” because she is enjoying being smarter than Hariezer. She then decides that she isn’t “bad”, its a budding romance. Thats the logic she uses. But because she won the book-reading contest against Hariezer (he doesn’t flip out, it must be because he learned “how to lose”), she gets to go on a date with him. The date is skipped over.

Next we find Hariezer meeting Malfoy in a dark basement, discussing how they will go about doing science. Malfoy is written as uncharacteristically stupid, in order to be a foil once more for Hariezer, peppering the conversation with such gems as:

Then I’ll figure out how to make the experimental test say the right answer!

"You can always make the answer come out your way,” said Draco. That had been practically the first thing his tutors had taught him. “It’s just a matter of finding the right arguments.”

We get a lot of platitudes from Hariezer about how science humbles you before nature. But then we get the same ideas Quirrell suggested previously, because “science is dangerous”, they are going to run their research program as a conspiracy.

"As you say, we will establish our own Science, a magical Science, and that Science will have smarter traditions from the very start.” The voice grew hard. “The knowledge I share with you will be taught alongside the disciplines of accepting truth, the level of this knowledge will be keyed to your progress in those disciplines, and you will share that knowledge with no one else who has not learned those disciplines. Do you accept this?”

And the name of this secretive scienspiracy?

And standing amid the dusty desks in an unused classroom in the dungeons of Hogwarts, the green-lit silhouette of Harry Potter spread his arms dramatically and said, “This day shall mark the dawn of… the Bayesian Conspiracy.”

Of course. I mentioned in the previous chapter, anyone who has done science knows that its a collaborative process that requires an open exchange of ideas.

And see what I mean about the melding of ideas between Quirrell and Hariezer? Its weird to use them both as author mouthpieces. The Bayesian Conspiracy is obviously an idea Yudkowsky is fond of, and here Hariezer gets the idea largely from Quirrell just one chapter back.

HPMOR 22: science!

This chapter opens strongly enough. Hariezer decides that the entire wizarding world has probably been wrong about magic, and don’t know the first thing about it.

Hermione disagrees, and while she doesn’t outright say “maybe you should read a magical theory book about how spells are created” (such a thing must exist), she is at least somewhat down that path.

To test his ideas, Hariezer creates a single-blind test- he gets spells from a book, changes the words or the wrist motion or what not and gets Hermione to cast them. Surprisingly, Hariezer is proven wrong by this little test. For once, the world isn’t written as insane as a foil for our intrepid hero.

It seemed the universe actually did want you to say ‘Wingardium Leviosa’ and it wanted you to say it in a certain exact way and it didn’t care what you thought the pronunciation should be any more than it cared how you felt about gravity.

There are a few anti-academic snipes, because it wouldn’t be HPMOR without a little snide swipe at academia:

But if my books were worth a carp they would have given me the following important piece of advice…Don’t worry about designing an elaborate course of experiments that would make a grant proposal look impressive to a funding agency.

Weird little potshots about academia (comments like “so many bad teachers, its like 8% as bad as Oxford,” “Harry was doing better in classes now, at least the classes he considered interesting”) have been peppered throughout the chapters since Hariezer arrived at Hogwarts. Oh academia, always trying to make you learn things that might be useful, even if they are a trifle boring. So full of bad teachers, etc. Just constant little comments attacking school and academia.

Anyway, this chapter would be one of the strongest chapters, except there is a second half. In the second half, Hariezer partners with Draco to get to the bottom of wizarding blood purity.

Harry Potter had asked how Draco would go about disproving the blood purist hypothesis that wizards couldn’t do the neat stuff now that they’d done eight centuries ago because they had interbred with Muggleborns and Squibs.

Here is the thing about science, step 0 needs to be make sure you’re trying to explain a real phenomena. Hariezer knows this, he tells the story of N-rays earlier in the chapter, but completely fails to understand the point.

Hariezer and Draco have decided, based on one anecdote (the founders of Hogwarts were the best wizards ever, supposedly) that wizards are weaker today than in the past. The first thing they should do is find out if wizards are actually getting weaker. After all, the two most dangerous dark wizards ever were both recent, Grindelwald and Voldemort. Dumbledore is no slouch. Even four students were able to make the marauders map just one generation before Harry. (Incidentally, this is exactly where neoreactionaries often go wrong- they assume things are getting worse without actually checking, and then create elaborate explanations for non-existent facts).

Anyway, for the purposes of the story, I’m sure it’ll turn out that wizards are getting weaker, because Yudkoswky wrote it. But this would have been a great chance to teach an actually useful lesson, and it would make the N-ray story told earlier a useful example, and not a random factoid.

Anyway, to explain the effect they come up with a few obvious hypotheses:

1. Magic itself is fading.
2. Wizards are interbreeding with Muggles and Squibs.
3. Knowledge to cast powerful spells is being lost.
4. Wizards are eating the wrong foods as children, or something else besides blood is making them grow up weaker.
5. Muggle technology is interfering with magic. (Since 800 years ago?)
6. Stronger wizards are having fewer children. (Draco = only child? Check if 3 powerful wizards, Quirrell / Dumbledore / Dark Lord, had any children.)

They miss some other obvious ones (there is a finite amount of magic power, so increasing populations = more wizards = less power per wizard, for instance. Try to come up with your own, its easy and fun).

They come up with some ways to collect some evidence- find out what the first year curriculum was throughout Hogwarts history, and do some wizard genealogy by talking to portraits.

Still, finally some science, even if half of it was infuriating.

HPMOR 23: wizarding genetics made (way too) simple

Alright, I need to preface this: I have the average particle physicists knowledge of biology (a few college courses, long ago mostly forgotten). That said, the lagavulin is flowing, so I’m going to pontificate as if I’m obviously right, so please reblog me with corrections if I am wrong.

In this chapter, Hariezer and Draco are going to explore what I think of as the blood hypothesis- that wizardry is carried in the blood, and that intermarriage with non-magical types is diluting wizardry.

Hariezer gives Draco a brief, serviceable enough description of DNA (more like pebbles than water), He lays out two models- there are lots of wizarding genes, and the more wizard genes you have, the more powerful the wizard you are. In this case, Hariezer reasons, as powerful wizards marry less powerful wizards, or non-magical types, the frequency of the magical variant of wizard genes in the general population becomes diluted. In this model, two squibs might rarely manage to have a wizard child, but they are likely to be weaker than wizard-born wizards. Call this model 1.

The other model Hariezer lays out is that magic lies on a single recessive gene. He reasons squibs have one dominant, non-magical version, and one recessive magical version of the gene. So of kids born to squibs, 1/4 will be wizards. In this version, you either have magic or you don’t, so if wizards married the non-magical, wizards themselves could become more rare, but the power of wizards won’t be diluted. Call this model 2.

The proper test between model 1 and 2, suggests Hariezer, is to look at the children born to two squibs. If about one fourth of them are wizards, its evidence of model 2, otherwise, evidence of model 1.

There is a huge problem with this. Do you see it? Here is a hint, What other predictions does model 2 make? While you are thinking about it, read on.

Before I answer the question, I want to point out that Hariezer ignores tons of other plausible models. Here is one I just made up. Imagine, for instance, a single gene that switches magic on and off, and a whole series of other genes that make you a better wizard. Maybe some double-jointed-wrist gene allows you to move your wand in unusually deft ways. Maybe some mouth-shape gene allows you to pronounce magical sounds no one else can. In this case, magical talent can be watered down as in model 1, and wizard inheritance could still look like Mendel would suggest, as in model 2.

Alright, below I’m going to answer my query above. Soon there will be no time to figure it for yourself.

Squibs are, by definition, the non-wizard children of wizard parents. Hariezer’s model 2 predicts that squibs cannot exist. It is already empirically disproven.

Hariezer, of course, does not notice this massive problem with his favored model, and Draco’s collected genealogy suggests about 6 out of 28 squib born children were wizards, so he declares model 2 wins the test.

Draco flips out, because now that he “knows” that magic isn’t being watered down by breeding he can’t join the death eaters and his whole life is ruined,etc. Hariezer is happy that Draco has “awakened as a scientist.” (I hadn’t complained about the stilted language in awhile, just reminding you that its still there), but Draco lashes out and casts a torture spell and locks Hariezer in the dungeon. After some failed escape attempts, he once against resorts to the time turner, because even now that its locked down, its the solution to every problem.

One other thing of note- to investigate the hypothesis that really strong spells can’t be cast anymore, Hariezer tries to look up a strong spell and runs into “the interdict of Merlin” that strong spells can’t be written down, only passed from wizard to wizard.

Its looking marginally possible that it will turn out that this natural secrecy is exactly whats killing off powerful magic- its not open so ideas aren’t flourishing or being passed on. Hariezer will notice that and realize his “Bayesian Conspiracy” won’t be as effective as an open science culture, and I’ll have to take back all of my criticisms around secretive science (it will be a lesson Hariezer learns, and not an idea Hariezer endorses). It seems more likely given the author’s existential risk concerns, however, that this interdict of Merlin will be endorsed.

Some more notes regarding HPMOR

There is a line in the movie Clueless (if you aren’t familiar, Clueless was an older generation’s Mean Girls) where a woman is described as a “Monet”- in that like the painting, it looks good from afar but up close is a mess.

So I’m now nearly 25 chapters into this thing, and I’m starting to think that HPMOR is this sort of a monet- if you let yourself get carried along, it seems ok-enough. It references a lot of things that a niche group of people,myself included, like (physics! computational complexity! genetics! psychology!). But as you stare at it more, you start noticing that it doesn’t actually hang together, its a complete mess.

The hard science references are subtly wrong, and often aren’t actually explained in-story (just a jargon dump to say ‘look, here is a thing you like’).

The social science stuff fairs a bit better (its less wrong ::rimshot::), but even when its explanation is correct, its power is wildly exaggerated- conversations between Quirrell/Malfoy/Potter seem to follow scripts of the form

"Here is an awesome manipulation I’m using against you"

"My, that is an effective manipulation. You are a dangerous man"

"I know, but I also know that you are only flattering me as an attempt to manipulate me." p "My, what an effective use of Bayesian evidence that is!"

Other characters get even worse treatment, either behaving nonsensically to prove how good Harry is at manipulation (as in the chapter where Harry tells off Snape and then tries to blackmail the school because Snape asked him questions he didn’t know), OR acting nonsensically so Harry can explain why its nonsensical (“Carry this rock around for no reason.” “Thats actually the fallacy of privileging the hypothesis.”) The social science/manipulation/marketing psychology stuff is just a flavoring for conversations.

No important event in the story has hinged on any of this rationality- instead basically every conflict thus far is resolved via the time turner.

And if you strip all this out, all the wrongish science-jargon and the conversations that serve no purpose but to prove Malfoy/Quirrell/Harry are “awesome” by having them repeatedly think/tell each other how awesome they are, the story has no real structure. Its just a series of poorly paced (if you strip out the “awesome” conversations, then there are many chapters where nothing happens), disconnected events. There is no there there.

HPMOR 24: evopsych Rorschach test

Evolutionary psychology is a field that famously has a pretty poor bullshit filter. Satoshi Kanazawa once published a series of articles that beautiful people will have more female children (because beauty is more important for girls) and engineers/mathematicians will have more male children (because only men need the logic-brains). The only thing his papers proved was that he is bad at statistics (in fact, Kanazawa made an entire career out of being bad at statistics, such is the state of evo-psych).

One of the core criticisms is that for any fact observed in the world, you can tell several different evolutionary stories, and there is no real way to tell which, if any is actually true. Because of this, when someone gives you an evopsych explanation for something, its often telling you more about what they believe then it is about science or the world (there are exceptions, but they are rare).

So this chapter is a long, pretty much useless conversation between Draco and Hariezer about how they are manipulating each other and Dumbledore or whatever, but smack in the middle we get this rumination:

In the beginning, before people had quite understood how evolution worked, they’d gone around thinking crazy ideas like human intelligence evolved so that we could invent better tools.

The reason why this was crazy was that only one person in the tribe had to invent a tool, and then everyone else would use it…the person who invented something didn’t have much of a fitness advantage, didn’t have all that many more children than everyone else. [SU comment- could the inventor of an invention perhaps get to occupy a position of power within a tribe? Could that lead to them having more wealth and children?]

It was a natural guess… A natural guess, but wrong.

Before people had quite understood how evolution worked, they’d gone around thinking crazy ideas like the climate changed, and tribes had to migrate, and people had to become smarter in order to solve all the novel problems.

But human beings had four times the brain size of a chimpanzee. 20% of a human’s metabolic energy went into feeding the brain. Humans were ridiculously smarter than any other species. That sort of thing didn’t happen because the environment stepped up the difficulty of its problems a little…. [SU challenge to the reader- save this climate change evolutionary argument with an ad-hoc justification]

Ending up with that gigantic outsized brain must have taken some sort of runawayevolutionary process…And today’s scientists had a pretty good guess at what that runaway evolutionary process had been….

[It was] Millions of years of hominids trying to outwit each other - an evolutionary arms race without limit - [that] had led to… increased mental capacity.

What does his preferred explanation for the origin of intelligence (people evolved to outwit each other) say about the author?

HPMOR 24/25/26: mangled narratives

This chapter is going to be entirely about the way the story is being told in this section of chapters. There is a big meatball of a terrible idea, but I’m getting sick of that low hanging fruit, so I’ll only mention it briefly in passing.

I’m a sucker for stories about con artists. In these stories, there is a tradition of breaking with the typical chronological order of story telling- instead they show the end result of the grand plan first, followed by all the planning that went into it (or some variant of that). In that way, the audience gets to experience the climax first from the perspective of the mark, and then from the perspective of the clever grifters. Yudkowsky himself successfully employs this pattern in the first chapter with the time turner.

In this chapter, however, this pattern is badly mangled. The chapter is setting up an elaborate prank on Rita Skeeter (Draco warned Hariezer that Rita was asking questions during one of many long conversations), but jumbling the narrative accomplishes literally nothing.

Here are the events, in the order laid out in the narrative

1. Hariezer tells Draco he didn’t tell on him about the torture, and borrows some money from him

2. (this is the terrible idea meatball) Using literally the exact same logic that Intelligent Design proponents use (and doing exactly 0 experiments), Hariezer decides while thinking over breakfast:

Some intelligent engineer, then, had created the Source of Magic, and told it to pay attention to a particular DNA marker.

The obvious next thought was that this had something to do with “Atlantis”.

1. Hariezer meets with Dumbledore, and refuses to tell on Draco, says getting tortured is all part of his manipulation game.

2. Fred and George Weasley meet with a mysterious man named Flume and tells him the-boy-who-lived needs the mysterious man’s help. There is a Rtia Skeeter story mentioned that says Quirrell is secretly a death eater and is training Hariezer to be the next dark lord, a story Flume says was planted by the elder Malfoy.

3. Quirrell tells Rita Skeeter he has no dark mark, Rita ignores him.

4. Hariezer hires Fred and George (presumably with Malfoy’s money) to perpetuate a prank on Rita Skeeter- to convince her of something totally false.

5. Hariezer has lunch with Quirrell, reads a newspaper story with the headline

HARRY POTTER SECRETLY BETROTHED TO GINEVRA WEASLEY

This is the story the Weasley’s planted apparently (the prank), and apparently there was a lot of supporting evidence or something, because Quirrell is incredulous it could be done. And then Quirrell after speculating that Rita Skeeter could be capable of turning to a small animal, crushes a beetle.

So whats the problem with this narrative order? First, there is absoltuely no payoff to jumbling the chronology. The prank is left until the end, and its exactly what we expected- a false story was planted in the newspaper. It doesn’t even seem like that big a deal- just a standard gossip column story (of course, Harry and Quirrell react like its a huge, impossible-to-have-done prank, to be sure the reader knows its hard.)

Second, most of the scenes are redundant, they contain no new information whatsoever and they are therefore boring- the event covered in 3 (talking with Dumbledore) is covered in full in 1 (telling Malfoy he didn’t tell on him to Dumbledore). The events of 6 (Hariezer hiring the Weasley’s to prank for him) are completely covered in 4 (when the Weasley’s hire Flume, they tell him its for Hariezer). This chapter is twice as long as it should be, for no reason.

Third, the actual prank is never shown from either the marks or the grifter’s perspective. It happens entirely off-stage so to speak. We don’t see Rita Skeeter encountering all this amazing evidence about Hariezer’s betrothal and writing up her career making article. We don’t see Fred and George’s elaborate plan (although if I were a wizard and wanted to plant a false newspaper story, I’d just plant a false memory in a reporter).

What would have been more interesting, the actual con happening off-stage, or the long conversations about nothing that happen in these chapters? These chapters are just an utter failure. The narrative decisions are nonsensical, and everything continues to be tell, tell, tell, never show.

Also of note- Quirrell gives Hariezer Roger Bacon’s diary of magic, because of course thats a thing that exists.

HPMOR 27: answers from a psychologist

In order to answer last night’s science question, I spent today slaving on the streets, polling professionals for answers (i.e. I sent one email to an old college roommate who did a doctorate in experimental brain stuff). This will basically be a guest post.

Here is the response:

The first thing you need to know, this is called “the simulation theory of empathy.” Now that you have a magic google phrase, you can look up everything you’d want, or read on my (not so) young padawan.

You are correct that no one knows how empathy works, its too damn complicated, but what we can look at is motor control, and in motor control the smoking gun for simulation is mirror neurons. Rizzolatti and collaborators discovered the certain neurons in macaque monkey’s inferior frontal gyrus that related to the motor-vocabulary activate not only when they do a gesture, but also when they see someone else doing that same gesture. So maybe, says Rizzolatti, the same neurons responsible for action are also responsible for understanding action (action-understanding). This is not the only explanation, it could be a simple priming effect. This would be big support for simulation explanations of understanding others. Unfortunately, its not the only explanation for action-understanding. There are other areas of the macaque brain (in particular the superior temporal sulcus) that aren’t involved in action, but do appear to have some role in action[understanding.

It is not an understatement to say that this discoveryy of mirror neurons caused the entire field to lose their collective shit. For some reason, motor explanations for brain phenomena are incredibly appealing to large portions of the field, and always have been. James Woods (the old dead behaviorist, not the awesome actor) had a theory that thought itself was related to the motor-neurons that control speech. Its just something that the entire field is primed to lose their shit over. Some philosophers of the mind made all sorts of sweeping pronouncements (“mirror-neurons are responsible for the great leap forward in human evolution”, pretty sure thats a direct quote)

The problem is that the gold standard for monkey tests is to see what a lesion in that portion of the brain does. Near as anyone can tell, lesions in F5 (portion of the inferior frontal gyrus where the mirror neurons on) does not impair action-understanding.

The next, bigger problem for theories of human behavior is that there is no solid evidence of mirror neurons in humans. A bunch of fmri studies showed a bit of activity in one region, and then meta-studies suggested not that region, maybe some other region, etc. fmri studies are tricky Google dead salmon fmri.

But even if mirror neurons are involved in humans, there is really strong evidence they can’t be involved in action-understanding. The mirror proponents suggest speech is a strong trigger for suggested mirror neurons. For instance, in the speech system, we’ve known since Paul Broca (really old French guy) that lesions can destroy your ability to speak without understanding your ability to understand speech. This is a huge problem for models that link action-understanding to action, killing those neurons should destroy both.

Also,suggested human mirror neurons do not fire in regards to pantomime actions. Also in autism spectrum disorders, action-understanding is often impaired with no impairment to action.

So in summary, the simulation theory of empathy got a big resurgence after mirror neurons, but there is decently strong empirical evidence against a mirror-only theory of action-understanding in humans. That doesn’t mean mirror neurons have no role to play (though if they aren’t found in humans, it does mean they have no role to play), it just means that the brain is complicated. I think the statement you quoted to me would have been something you could read from a philosopher of mind in the late 80s or early 90s, but not something anyone involved in experiments would say. By the mid 2000s, a lot of that enthusiasm had pittered a bit. Then I left the field.

So on this particular bit of science, it looks like Yudkowsky isn’t wrong he is just presenting conjecture and hypothesis as settled science. Still I learned something here, I’d never encountered this idea before. I’ll have an actual post about chapter 27 tomorrow.

HPMOR 27: mostly retreads

This is another chapter where most of the action is stuff that has happened before, we are getting more and more retreads.

The new bit is that Hariezer is learning to defend himself from mental attacks. The goal, apparently, is to perfectly simulate someone other than yourself, in that way the mind reader learns the wrong things. This leads in to the full-throated endorsement of the simulation theory of empathy that was discussed by a professional in my earlier post. Credit where credit is due- this was an idea I’d never encountered before, and I do think HPMOR is good for some of that- if you don’t trust the presentation and google ideas as they come up, you could learn quite a bit.

We also find out Snape is a perfect mind-reader. This is an odd choice- in the original books Snape’s ability to block mind-reading was something of a metaphor for his character- you can’t know if you can trust him because he is so hard to read, his inscrutableness even fooled the greatest dark wizard ever, etc. It was, fundamentally, hid cryptic dodginess that helped the cause, but it also fermented the distrust that some of the characters in the story felt toward him.

Now for the retreads-

More pointless bitching about quidditch. Nothing was said here that wasn’t said in the earlier bitching about quidditch.

Snape enlists Hareizer’s help to fight anti-slytherin bullies (for no real reason, near as I can tell), the bullies are fought once more with con-artist style cleverness (much like in the earlier chapter with the time turner and invisitbility cloak. In this chapter, its just with an invisibility cloak).

Snape rewards Hariezer’s rescue with a conversation about Hariezer’s parents, during which Hariezer decides his mother was shallow, which upsets Snape. Its an odd moment, but the odd moments in HPMOR dialogue have piled up so high its almost not worth mentioning.

And the chapter culminates when the bullied slytherin tells Hariezer about Azkaban, pleading with Hariezer to save his parents. Of course, Hariezer can’t (something tells me he will in the near future), and we get this:

"Yeah," said the Boy-Who-Lived, "that pretty much nails it. Every time someone cries out in prayer and I can’t answer, I feel guilty about not being God."

…

The solution, obviously, was to hurry up and become God.

So another retread- Hariezer is once more making clear his motives aren’t curiosity, they are power. This was true after chapter 10, its still true now.

This is the only real action for several chapters now, unfortunately all the action feels like its already happened before in other chapters.

HPMOR 28: hacking science/map-territory

Finally we get back to some attempts to do magi-science, but its again deeply frustrating. Its more transfiguration- the only magic we have thus far explored, and it leads to a discussion of map vs. territory distinctions that is horribly mangled.

At the opening of hte chapter, instead of using science to explore magic, the new approach is to treat magic as a way to hack science itself. To that end, Hariezer tries (and fails) to transfigure something into “cure for Alzheimer’s,” and then tries (successfully) to transfigure a rope of carbon nanotubes. I guess the thought here is he can then give these things to scientists to study? Its unclear, really.

Frustrated with how useless this seems, Hermione make this odd complaint:

"Anyway," Hermione said. Her voice shook. "I don’t want to keep doing this. I don’t believe children can do things that grownups can’t, that’s only in stories."

Poor Hermione- thats the feeblest of objections, especially in a story where every character acts like they are in their late teens or twenties. Its almost as if the author was looking for some knee jerk complaint you could throw out that everyone would write-off as silly on its face.

So Hariezer decides he needs to do something adults can’t to appease Hermione. To do this, he decides to attack the constraints he knows about magic, starting with the idea that you can only transfigure a whole object, and not part of an object (a constraint I think was introduced just for this chapter?).

So Hariezer reasons: things are made out of atoms. There isn’t REALLY a whole object there,so why can’t I do part of an object? This prompted me to wonder- if you do transform part of an object, what happens at the interface? Does this whole-object constraint have something to do with the interface? I mentioned in the chapter 15 section that magicking in a lot large gold molecule into water could cause steric mismatches (just volume constraints really) with huge energy differences, hence explosions. What happens at the micro level when you take some uniform crystalline solid and try to patch on some organic material like rubber at some boundary? If you deform the (now rubbery) material, what happens when it changes back and the crystal spacing is now messed up? Could you partially transform something if you carefully worked out the interface?

It will not surprise someone who has read this far that none of these questions are asked or answered.

Instead, Hariezer thinks really hard about how atoms are real, in the process we get ruminations on the map and the territory:

But that was all in the map, the true territory wasn’t like that, reality itself had only a single level of organization, the quarks, it was a unified low-level process obeying mathematically simple rules.

This seems innocuous enough, but a fundamental mistake is being made here. For better or for worse, physics is limited in what it can tell you about the territory, it can just provide you with more accurate maps. Often it provides you with multiple, equivalent maps for the same situation with no way to choose between them.

For instance, quarks (and gluons) have this weird property- they are well defined excitations at very high energies, but not at all well-defined at low energies, where bound states become fundamental excitations. There is no such thing as a free-quark at low energy. For some problems, the quark map is useful, for many, many more problems the meson/hadron (proton,neutron,kaon,etc) map is much more useful. The same theory at a different energy scale provides a radically different map (renormalization is a bitch, and a weak coupling becomes strong).

Continuing in this vein, he keeps being unable to transform only part of an object, so he keeps trying different maps, and making the same map/territory confusion culminating in:

If he wanted power, he had to abandon his humanity, and force his thoughts to conform to the true math of quantum mechanics.

There were no particles, there were just clouds of amplitude in a multiparticle configuration space and what his brain fondly imagined to be an eraser was nothing except a gigantic factor in a wavefunction that happened to factorize,

(Side note: for Hariezer its all about power, not about curiosity, as I’ve said dozens of time now. Also, I know as much physics as anyone, and I don’t think I’ve abandoned my humanity.)

This is another example of the same problem I’m getting at above. There is no “true math of quantum mechanics.” In non-relativistic, textbook quantum mechanics, I can formulate one version of quantum mechanics on 3 space dimension 1 time dimension, and calculate things via path integrals. I can also build a large configuration space (Hilbert space) with 3 space dimensions, and 3 momentum dimensions per particle, (and one overall time dimension) and calculate things via operators on that space. These are different mathematical formulations, over different spaces, that are completely equivalent. Neither map is more appropriate than the other. Hariezer arbitrarily thinks of configuration space as the RIGHT one.

This isn’t unique to quantum mechanics, most theories have several radically different formulations. Good old newtonian mechanics has a formulation on the exact same configuration space Hariezer is thinking of.

The big point here is that the same theory has different mathematical formulations. We don’t know which is “the territory” we just have a bunch of different, but equivalent maps. Each map has its own strong suits, and its not clear that any one of them is the best way to think about all problems. Is quantum mechancis 3+1 dimensions (3 space, 1 time) or is it 6N+1 (3 space and 3 momentum + 1 time dimension)? Its both and neither (more appropriately, its just not a question that physics can answer for us).

What Hariezer is doing here isn’t separating the map and the territory, its reifying one particular map (configuration space)!

(Less important: I also find it amusing, in a physics elitist sort of way (sorry for the condescension) that Yudkowsky picks non-relativistic quantum mechanics as the final, ultimate reality. Instead of describing or even mentioning quantum field theory, which is the most low-level theory we (we being science) know of, Yudkowsky picks non-relativistic quantum mechanics, the most low-level theory HE knows.)

Anyway, despite obviously reifying a map, in-story it must be the “right” map, because suddenly he manages to transform part of an object, although he tells Hermione

Quantum mechanics wasn’t enough,” Harry said. “I had to go all the way down to timeless physics before it took.

So this is more bad pedagogy: timeless physics isn’t even a map, its the idea of a map. No one has made a decent formulation of quantum mechanics without a specified time direction (technical aside: its very hard to impose unitarity sensibly if you are trying to make time emerge from your theory, instead of being inbuilt). Its pretty far away from mainstream theory attempts, but here its presented as the ultimate idea in physics. It seems very odd to just toss in a somewhat obscure ideas as the pinnacle of physics.

Anyway, Hariezer shows Dumbledore and McGonagall his new found ability to transfigure part of an object, chapter ends.

HPMOR 29: not much here

Someone called Yudkowsky out on the questionable decision to include his pet theories as established science, so chapter 29 opens with this (why didn’t he stick this disclaimer on the chapters where the mistakes were made?):

Science disclaimers: Luosha points out that the theory of empathy in Ch. 27 (you use your own brain to simulate others) isn’t quite a known scientific fact. The evidence so far points in that direction, but we haven’t analyzed the brain circuitry and proven it. Similarly, timeless formulations of quantum mechanics (alluded to in Ch. 28) are so elegant that I’d be shocked to find the final theory had time in it, but they’re not established yet either.

He is still wrong about timeless formulations of quantum though, they aren’t more elegant, they don’t exist.

The rest of this chapter seems like its just introductory for something coming later- Hariezer, Draco and Hermione are all named as heads of Quirrell’s armies and are all trying to manipulate each other. Some complaints from Hermione that broomstick riding is jock-like and stupid, old hat by now.

There, is however, one exceptionally strange bit- apparently in this version of the world, the core plot of Prisoner of Azkaban (Scabbers the rat was really Peter Petigrew) was just a delusion that a schizophrenic Weasley brother had. Just a stupid swipe at the original book for no real reason.

HPMOR 30-31: credit where credit is due

So credit where credit is due -- these two chapters are pretty decent. We finally get some action in a chapter, there is only one bit of wrongish science, and the overall moral of the episode is a good one.

In these chapters, three teams, “armies” lead by Draco, Hariezer and Hermione, compete in a mock-battle, Quirrell’s version of a team sport. The action is more less competently written (despite things like having Neville yell “special attack”), and its more-or-less fun and quick to read. It feels a bit like a lighter-hearted version of the beginning competitions of Ender’s game (which no doubt inspired these chapters.

The overall “point” of the chapters is even pretty valuable- Hermione, who is written off as an idiot by both Draco and Hariezer splits her army and has half attack Draco and half attack Hariezer. She is seemingly wiped out almost instantly. Draco and Hariezer then fight each other nearly to death, and out pops Hermione’s army- turns out they only faked defeat. Hermione wins, and we learn that unlike Draco and Hariezer, Hermione delegated and collaborated with the rest of her team to develop strategies to win the fight. There is a (very unexpected given the tone of everything thus far) lesson about teamwork and collaboration here.

That said -- I still have nits to pick. Hariezer’s army is organized in quite possibly the dumbest possible way:

Harry had divided the army into 6 squads of 4 soldiers each, each squad commanded by a Squad Suggester. All troops were under strict orders to disobey any orders they were given if it seemed like a good idea at the time, including that one… unless Harry or the Squad Suggester prefixed the order with “Merlin says”, in which case you were supposed to actually obey.

This might seem like a good idea, but anyone who has played team sports can testify- there is a reason that you work out plays in advance, and generally have delineated roles. I assume the military has a chain of command for similar reasons, though I have never been a solider. I was hoping to see this idea for a creatively-disorganized army bite Hariezer, but it does not. There seems to be no confusion at all over orders, etc. Basically, none of what you’d expect would happen from telling an army “do what you want, disobey all orders” happens.

And it wouldn’t be HPMOR without potentially bad social science, here is today’s reference:

There was a legendary episode in social psychology called the Robbers Cave experiment. It had been set up in the bewildered aftermath of World War II, with the intent of investigating the causes and remedies of conflicts between groups. The scientists had set up a summer camp for 22 boys from 22 different schools, selecting them to all be from stable middle-class families. The first phase of the experiment had been intended to investigate what it took to start a conflict between groups. The 22 boys had been divided into two groups of 11 -

• and this had been quite sufficient.

The hostility had started from the moment the two groups had become aware of each others’ existences in the state park, insults being hurled on the first meeting. They’d named themselves the Eagles and the Rattlers (they hadn’t needed names for themselves when they thought they were the only ones in the park) and had proceeded to develop contrasting group stereotypes, the Rattlers thinking of themselves as rough-and-tough and swearing heavily, the Eagles correspondingly deciding to think of themselves as upright-and-proper.

The other part of the experiment had been testing how to resolve group conflicts. Bringing the boys together to watch fireworks hadn’t worked at all. They’d just shouted at each other and stayed apart. What had worked was warning them that there might be vandals in the park, and the two groups needing to work together to solve a failure of the park’s water system. A common task, a common enemy.

Now, I readily admit to not having read the original Robber’s Cave book, but I do have two textbooks that reference it, and Yudkowsky gets the overall shape of the study right, but fails to mention some important details. (If my books are wrong, and Yudkowsky is right, which seems highly unlikely given his track record please let me know)

Both descriptions I have suggest the experiment had 3 stages, not two. The first stage was to build up the in-groups, then the second stage was to introduce them to each other and build conflict, and then the third stage was to try and resolve the conflict. In particular, this aside from Yudkowsky originally struck me as surprising insightful:

They’d named themselves the Eagles and the Rattlers (they hadn’t needed names for themselves when they thought they were the only ones in the park)

Unfortunately, its simply not true- during phase 1 the researchers asked the groups to come up with names for themselves, and let the social norms for the groups develop on their own. The “in-group” behavior developed before they met their rival groups.

While tensions existed from first meeting, real conflicts didn’t develop until the two groups competed in teams for valuable prizes.

This stuff matters- Yudkowsky paints a picture of humans diving so easily into tribes that simply setting two groups of boys loose in the same park will cause trouble. In reality, taking two groups of boys, encouraging them to develop group habits, group names, group customs, and then setting the groups to directly competing for scarce prizes (while researchers encourage the growth of conflicts) will cause conflicts. This isn’t just a subtlety.

HPMOR 32: interlude

Chapter 32 is just a brief interlude, nothing here really, just felt the need to put this in for completeness.

HPMOR 33: it worked so well the first time, might as well try it again

This chapter has left me incredibly frustrated. After a decent chapter, we get a terrible retread of the same thing. For me, this chapter failed so hard that I’m actually feeling sort of dejected, it undid any good will the previous battle chapter had built up.

This section of chapters is basically a retread of the dueling armies just a brief section back. Unfortunately, this second battle section flubs completely a lot of the things that worked pretty well in the first battle section. There is a lot to talk about here that I think failed, so this might be long.

There is an obvious huge pacing problem here. The first battle game happens just a brief interlude before the second battle game. Instead of spreading this game out over the course of the Hogwarts school year (or at least putting a few of the other classroom episodes in between) these just get slammed together. First battle, one interlude, last battle. That means that a lot of the evolution of the game over time, how people are reacting to it, etc. is left as a tell rather than a show. A lot of this chapter is spent dealing with big changes to Hogwarts that have been developing as student’s get super-involved in this battle game, but we never see any of that.

Imagine if Ender’s game (a book fresh on my mind because of the incredibly specific references in this chapter) were structured so that you get the first battle game, and then a flash-forward to his final battle against the aliens, with Ender explaining all the strategy he learned over the rest of that year. This chapter is about as effective as that last Ender’s game battle would be.

The chapter opens with Dumbledore and McGonagall worried about the school-

Students were wearing armbands with insignia of fire or smile or upraised hand, and hexing each other in the corridors.

Loyalty to armies over house or school is tearing the school apart!

But then we turn to the army generals- apparently the new rules of the game allowed soldiers in armies to turn traitor, and its caused the whole game to spiral out of control- Draco complains:

You can’t possibly do any real plots with all this stuff going on. Last battle, one of my soldiers faked his own suicide.

Hermione agrees, everyone is losing control of their armies because of all the traitors.

"But.. wait…" I can hear you asking, "how can that make sense? Loyalty to the armies is so absolute people are hexing each other in the corridors? But at the same time, almost all the students in the armies are turning traitor and plotting against their generals? Both of those can’t be true?" I agree, you smart reader you, both of these things don’t work together. NOT ONLY IS YUDKOWSKY TELLING INSTEAD OF SHOWING, WE ARE BEING TOLD CONTRADICTORY THINGS. Yudkowsky wanted to be able to follow through on the Robber’s Cave idea he developed earlier, but he also needed all these traitors for his plot, so he tried to run in both directions at once.

Thats not the only problem with this chapter (it wouldn’t be HPMOR without misapplied science/math concepts)- it turns out Hermione is winning, so the only way for Draco and Hariezer to try to catch up is to temporarily team up, which leads to a long explanation where Hariezer explains the prisoner’s dilemma and Yudkowsky’s pet decision theory.

Here is the big problem- In the classic prisoner’s dilemma:

If my partner cooperates, I can either:

-cooperate, in which case I spend a short time in jail, and my partner spends a short time in jail

-defect, in which case I spend no time in jail, and my partner serves a long time in jail

If my partner defects, I can either:

-cooperate, in which case I spend a long time in jail, and my partner goes free

-defect, in which case I spend a long time in jail, as does my partner.

The key insight of the prisoner’s dilemma is that no matter what my partner does, defecting improves my situation. This leads to a dominant strategy where everyone defects, even though the both-defect is worse than the both-cooperate.

In the situation between Draco and Hariezer:

If Draco cooperates, Hariezer can either:

-cooperate in which case both Hariezer and Draco both have a shot at getting first or second

-defect, in which case Hariezer is guaranteed second, Draco guaranteed 3rd place

If Draco defects, Hariezer can either

-cooperate, in which case Hariezer is guaranteed 3rd, and Draco gets 2nd.

-defect, in which case Hariezer and Draco are fighting it out for 2nd and third.

Can you see the difference here? If Draco is expected to cooperate, Hariezer has no incentive to defect- both cooperate is STRICTLY BETTER than the situation where Hariezer defects against Draco. This is not at all a prisoner’s dilemma, its just cooperating against a bigger threat. All the pontificating about decision theories that Hariezer does is just wasted breath, because no one is in a prisoner’s dilemma.

After the pointless digression about the non-prisoner’s dilemma (seriously, this is getting absurd, and frustrating- I’m hard pressed to find a single science reference in this whole thing that’s unambiguously applied correctly.).

After these preliminaries, the battle begins. Unlike the light hearted, winking reference to Ender’s game of the previous chapter, Yudkowsky feels the need to make it totally explicit- they fight in the lake, so that Hariezer can use exactly the stuff he learned from Ender’s game to give him an edge. It turns the light homage of the last battle into just the setup for the beat-you-over-the-head reference this time. There is a benefit to subtlety, and assuming your reader isn’t an idiot.

Anyway, during the battle, everyone betrays everyone and the overall competition ends in a tie.

HPMOR 34-35: aftermath of the war game

These chapters contain a lot of speechifying, but in this case it actually fits, as a resolution to the battle game. Its expected and isn’t overly long.

The language, as throughout, is still horribly stilted, but I think I’m getting used to it (when Hariezer referred to Hermione as “General of Sunshine” I almost went right past it without a mental complaint). Basically, I’m likely to stop complaining about the stilted language but its still there, its always there.

Angry at the ridiculousness of the traitors, Hemione and Draco insist that if Hariezer uses traitors in his army that they will team up and destroy him. He insists he will keep using them.

Next, Quirrell gives a long speech about how much chaos the traitors were able to create, and makes an analogy to the death eaters. He insists that the only way to guard against such is essentially fascism.

Hariezer than speaks up, and says that you can do just as much damage in the hunt for traitors as traitors can do themselves, and stands up for a more open society. The themes of these speeches can be found in probably hundreds of books, but they work well enough here.

Every army leader gets a wish, Draco and Hermione decide to wish for their houses to win the house cup. In an attempt to demonstrate his argument for truth, justice and the American way, Hariezer wishes for quidditch to no longer contain the snitch. I guess nothing will rally students around an open society like one person fucking with the sport they love.

We also find out that Dumbledore helped the tie happen by aiding in the plotting(the plot was “too complicated” for any student, according to Quirrell, so it must have been Dumbledore- apparently, ‘betray everyone to keep the score close’ is a genius master plan), but we are also introduced to a mysterious cloaked stranger who was also involved but wiped all memory of his passing.

These are ok chapters, as HPMOR chapters go.

Hariezer's Arrogance

In response to some things that kai-skai has said, I started thinking about how should we view Hariezer’s arrogance. Should we view it as a character flaw? Something Hariezer will grow and overcome? I don’t think its being presented that way.

My problem with the arrogance are several:

-the author intends for Hariezer to be a teacher. He is supposed to be the master rationalist that the reader (and other characters) learn from. His arrogance makes that off-putting. If you aren’t familiar at all with the topics Hariezer happens to be discussing, you are being condescended to along with the characters in the story (although if you know the material you get to condescend to the simpletons along with Hariezer). Its just a bad pedagogical choice. You don’t teach people by putting them on the defensive.

-The arrogance is not presented by the author as a character flaw. In the story, its not a flaw to overcome, its part of what makes him “awesome.” His arrogance has not harmed him, he hasn’t felt the need to revisit it. When he thinks he knows better than everyone else, the story invariably proves him right. He hasn’t grown or been presented with a reason to grow. I would bet a great deal of money that Hariezer ends HPMOR exactly the same arrogant twerp he starts as.

-This last one is a bit of a personal reaction. Hariezer gets a lot of science wrong (I think all of it is wrong, actually, up to where I am now), and is incredibly arrogant while doing so. I’ve taught a number of classes at the college level, and I’ve had a lot of confidently, arrogantly wrong students. Hariezer’s attitude and lack of knowledge repeatedly remind me of the worst students I ever had- smart kids too arrogant to learn (and these were physics classes, where wrong or right is totally objective).

HPMOR 36/37: Christmas break/not much happens

Like all the Harry Potter books, Yudkowsky includes a Christmas break. I note that a Christmas break would make a lot of sense toward the middle of the book, not less than <1/3 of the way through. Like the original books, this is just a light bit of relaxation

Not a lot happens over break. Hariezer is a twerp who think his parents don’t respect him enough, they go to Hermione’s house for Christmas, Hariezer yells at Hermione’s parents for not respecting her intelligence enough, the parents say Hermione and Hariezer are like an old married couple (it would have been nice to see the little bonding moments in the earlier chapters). Quirrell visits Hariezer on Christmas Eve.

HPMOR 38: a cryptic conversation, not much here

This whole chapter is just a conversation between Malfoy and Hariezer. It fits squarely into the “one party doesn’t really know what the conversation is about” mold, with Hariezer being the ignorant party. Malfoy is convinced Hariezer is working with someone other than Quirrell or Dumbledore.

HPMOR 39: your transhumanism is showing

This was a rough chapter, in which primarily Hariezer and Dumbledore have an argument about death. Hariezer takes up the transhumanist position. If you aren’t familiar with the transhumanist position on death, its basically that death is bad (duh!) and that the world is full of deathists who have convinced themselves that death is good. This usually leads into the idea that some technology will save us from death (nanotech, SENS,etc), and even if they don’t we can all just freeze our corpses to be reanimated when that whole death thing gets solved. I find this position somewhat childish, as I’ll try and get to.

So, as a word of advice to future transhumanist authors who want to write literary screeds arguing against the evil deathists, FANTASY LITERATURE IS A UNIQUELY BAD CHOICE FOR ARGUING YOUR POINT. To be fair, Yudkowsky noticed this, and lampshaded it, when Hariezer says there is no afterlife, Dumbledore argues back with:

“How can you not believe it? ” said the Headmaster, looking completely flabbergasted. “Harry, you’re a wizard! You’ve seen ghosts! ” …And if not ghosts, then what of the Veil? What of the Resurrection Stone?”

i.e. how can you not believe in an afterlife with there is a literal gateway to the fucking afterlife sitting in the ministry of magic basement. Hariezer attempts to argue his way out of this, we get this story for instance:

You know, when I got here, when I got off the train from King’s Cross…I wasn’t expecting ghosts. So when I saw them, Headmaster, I did something really dumb. I jumped to conclusions. I, I thought there wasan afterlife… I thought I could meet my parents who died for me, and tell them that I’d heard about their sacrifice and that I’d begun to call them my mother and father -

"And then… asked Hermione and she said that they were just afterimages… And I should have known! I should have known without even having to ask! I shouldn’t have believed it even for all of thirty seconds!… And that was when I knew that my parents were really dead and gone forever and ever, that there wasn’t anything left of them, that I’d never get a chance to meet them and, and, and the other children thought I was crying because I was scared of ghosts

So, first point- this could have been a pretty powerful moment if Yudkowsky had actually structured the story to relate this WHEN HARIEZER FIRST MET A GHOST. Instead, the first we hear of it is this speech. Again, tell, tell, tell, never show

Second point- what exactly does Hariezer assume is being “afterimaged?” Clearly some sort of personality, something not physical is surviving in the wizarding world after death. If fighting death is this important to Hariezer, why hasn’t he even attempted to study ghosts yet? (full disclosure, I am an atheist personally. However, if I lived in a world WITH ACTUAL MAGIC, LITERAL GHOSTS, a stone that resurrects the dead, and a FUCKING GATEWAY TO THE AFTERLIFE I might revisit that position).

Here is Hariezer’s response to the gateway to the afterlife:

That doesn’t even sound like an interesting fraud,” Harry said, his voice calmer now that there was nothing there to make him hope, or make him angry for having hopes dashed. “Someone built a stone archway, made a little black rippling surface between it that Vanished anything it touched, and enchanted it to whisper to people and hypnotize them.”

Do you see how incurious Hariezer is? If someone told me there was a LITERAL GATEWAY TO THE AFTERLIFE I’d want to see it. I’d want to test it, see it. Can we try to record and amplify the whispers? Are things being said?

Why do they think its a gateway to the afterlife? Who built it? Minimally, this could have lead to a chapter where Hariezer debunks wizarding spiritualists like a wizard-world Houdini. (Houdini spent a great deal of his time exposing mediums and psychics who ‘contacted the dead’ as frauds.) I’m pretty sure I would have even enjoyed a chapter like that.

In the context of the wizarding world, there is all sorts of non-trivial evidence for an afterlife that simply doesn’t exist in the real world. Its just a bad choice to present these ideas in the context of this story.

Anyway, ignoring what a bad choice it is to argue against an afterlife in the context of fantasy fiction, lets move on:

Dumbledore presents some dumb arguments so that Hariezer can seem wise. Hariezer tells us death is the most frightening thing imaginable, its not good,etc. Basically, death is scary, no one should have to die. If we had all the time imaginable we would actually use it. Pretty standard stuff, Dumbledore drops the ball presenting any real arguments.

So I’ll take up Dumbledore’s side of the argument. I have some bad news for Hariezer’s philosophy. You are going to die. I’m going to die. Everyone is going to die. It sucks, and its unfortunate, sure, but there is no way around it. Its not a choice! We aren’t CHOOSING death. Even if medicine can replace your body (which doesn’t seem likely in my lifetime), the sun will explode some day. Even if we get away from the solar system, eventually we’ll run out of free energy in the universe.

But you do have one choice regarding death- you can accept that you’ll die someday, or you can convince yourself there is some way out. Convince yourself that if you say the right prayers, or in the Less Wrong case, work on the right decision theory to power an AI you’ll get to live forever. Convince yourself that if you give a life insurance policy to the amateur biologists that run croynics organizations you’ll be reanimated.

The problem with the second choice is that there is an opportunity cost- time spent praying or working on silly decision theories is time that you aren’t doing things that might matter to other humans. We accept death to be more productive in life. Stories about accepting death aren’t saying death is good they are saying death is inevitable.

Edit: I take back a bit about cognitive dissonance that was here.

HPMOR 40: short follow up to 39

Instead of Dumbledore’s views, in this chapter we get Quirrell’s view of death. He agrees with Hariezer, unsurprisingly.

HPMOR 41: another round of Quirrell's battle game

It seems odd that AFTER the culminating scene, the award being handed out, and the big fascist vs. freedom speechifying that we have yet another round of Quirrell’s battle game.

Draco and Hermione are now working together against Hariezer. Through a series of circumstances, Draco has to drop Hermione off a roof to win.

Edit: I also point out that we don’t actually get details of the battle in this time, it opens with

Only a single soldier now stood between them and Harry, a Slytherin boy named Samuel Clamons, whose hand was clenched white around his wand, held upward to sustain his Prismatic Wall.

We then get a narrator summary of the battle that had lead up that moment. Again, tell, tell, tell never show.

HPMOR 42: is there a point to this?

Basically an extraneous chapter, but one strange detail at the end.

So in this chapter, Hariezer is worried that its his fault that in the battle last chapter. Hermione got dropped off a roof. Hermione agrees to forgive him as long as he lets Draco drop him off the same roof.

He takes a potion to help him fall slowly and is dropped, but so many young girls try to summon him to their arms (yes, this IS what happens) that he ends up falling, luckily Remus Lupin is there to catch him.

Afterwards, Remus and Hariezer talk. Hariezer learns that his father was something of a bully. And, for some reason, that Peter Petigrew and Sirius Black were lovers. Does anyone know what the point of making Petigrew and Black lovers would be?

Conversations

My girlfriend: “What have you been working on over there?”

Me: “Uhhhh… so…. there is this horrible Harry Potter fan fiction… you know, when people on the internet write more stories about Harry Potter? Yea, that. Anyway, this one is pretty terrible so I thought I’d read it and complain about it on the internet…. So I’m listening to me say this out loud and it sounds ridiculous, but.. well, it IS ridiculous… but…”

HPMOR 43-46: Subtle Metaphors

These chapters actually moved pretty decently. When Yudkowsky isn’t writing dialogue, his prose style can actually be pretty workman-like. Nothing that would get you to stop and marvel at the word play, but it keeps the pace brisk and moving.

Now, in JK Rowling’s original books, it always seemed to me that the dementors were a (not-so-subtle) nod to depression. They leave people wallowing in their worst memories, low energy, unable to remember the happy thoughts,etc.

In HPMOR, however, Hariezer (after initially failing to summon a patronus) decides that the dementors really represent death. You see in HPMOR, instead of relieving their saddest, most depressing memories the characters just see a bunch of rotting corpses when the dementors get near.

This does, of course, introduce new questions? What does it mean that the dementors guard Azkaban? Why don’t the prisoner’s instantly die? Why doesn’t a dementor attack just flat-out kill you?

Anyway, apparently the way to kill death is to just imagine that someday humans will defeat death, in appropriately Carl Sagan-esque language:

The Earth, blazing blue and white with reflected sunlight as it hung in space, amid the black void and the brilliant points of light. It belonged there, within that image, because it was what gave everything else its meaning. The Earth was what made the stars significant, made them more than uncontrolled fusion reactions, because it was Earth that would someday colonize the galaxy, and fulfill the promise of the night sky.

Would they still be plagued by Dementors, the children’s children’s children, the distant descendants of humankind as they strode from star to star? No. Of course not. The Dementors were only little nuisances, paling into nothingness in the light of that promise; not unkillable, not invincible, not even close.

Once you know this, your patronus becomes a human, and kills the dementor. Get it THE PATRONUS IS HUMANS (represented in this case by a human) and THE DEMENTOR IS DEATH. Humans defeat death. Very subtle.

Another large block of chapters with no science.

HPMOR 47: Racism is Bad

Nothing really objectionable here, just more conversations and plotting.

Hariezer spends much of this chapter explaining to Draco that racism is bad, and that a lot of pure bloods probably hate mudbloods because it gives them a chance to feel superior. Hariezer suggests these racist ideas are poisoning slytherin.

We also find out that Draco and his father seem to believe that Dumbledore burned Draco’s mother alive. This is clearly a departure from the original books. Hariezer agrees to take as an enemy whoever killed Draco’s mother. Feels like it’ll end up being more plots-within-plots stuff.

Another chapter with no science explored. We do find out Hariezer speaks snake language.

HPMOR 48: Utilitarianism

This chapter is actually solid as far as these things go. After learning he can talk to snakes Hariezer begins to wonder if all animals are sentient, after all snakes can talk. This has obvious implications for meat eating.

From there, he begins to wonder if plants might be sentient, in which case he wouldn’t be able to eat anything at all. This leads him to the library for research.

He also introduces scope insensitivity and utilitarianism, even though it isn’t really required at all to explain his point to Hermione. Hermione asks why he is freaking out, and instead of answering “I don’t want to eat anything that thinks and talks,” he says stuff like

"Look, it’s a question of multiplication, okay? There’s a lot of plants in the world, if they’renot sentient then they’re not important, but if plants are people then they’ve got more moral weight than all the human beings in the world put together. Now, of course your brain doesn’t realize that on an intuitive level, but that’s because the brain can’t multiply. Like if you ask three separate groups of Canadian households how much they’ll pay to save two thousand, twenty thousand, or two hundred thousand birds from dying in oil ponds, the three groups will respectively state that they’re willing to pay seventy-eight, eighty-eight, and eighty dollars. No difference, in other words. It’s called scope insensitivity.

Is that really the best way to describe his thinking? Why say something with 10 words while several hundred will do. What does scope insensitivity have to do with the idea “I don’t want to eat things that talk and think?”

Everything below here is unrelated to HPMOR and has more to do with scope insensitivity as a concept:

Now, because I have taught undergraduates intro physics, I do wonder (and have in the past)- is Kahneman’s scope insensitivity related to the general innumeracy of most people? i.e. how many people who hear that question just mentally replace literally any number with “a big number”?

The first time I taught undergraduates I was surprised to learn that most of the students had no ability to judge if their answers seemed plausible. I began adding a question “does this answer seem order of magnitude correct?” I’d also take off more points for answers that were the wrong order of magnitude, unless the student put a note saying something like “I know this is way too big, but I can’t find my mistake.”

You could ask a question about a guy throwing a football, and answers would range from 1 meter/second all the way to 5000 meters/second. You could ask a question about how far someone can hit a baseball and answers would similarly range from a few meters to a few kilometers. No one would notice when answers were wildly wrong. Lest someone think this is a units problem (Americans aren’t used to metric units), even if I forced them to convert to miles per hour, miles, or feet students couldn’t figure out if the numbers were the right order of magnitude.

So I began to give a few short talks on what I thought as basic numeracy. Create mental yardsticks (the distance from your apartment to campus might be around a few miles, the distance between this shitty college town and the nearest actual city might be around a few hundred miles,etc). When you encounter unfamiliar problems, try to relate it back to familiar ones. Scale the parameters in equations so you have dimensionless quantities * yardsticks you understand. And after being explicitly taught most of the students got better at understanding the size of numbers.

Since I began working in the business world I’ve noticed that most people never develop that skill. Stick a number in a sentence and people just mentally run right over it, you might as well have inserted some klingon phrases. Some of the better actuaries do have some nice numerical intuition, but a surprising number don’t. They can calculate, but they don’t understand what the calculations are really telling them, like Searle’s chinese room but with numbers.

In Kahneman’s scope neglect questions, there are big problems with innumeracy- if you ask people how much they’d spend on X where X is any charity that seems importantish, you are likely to get an answer of around $100. In some sense, it is scope neglect, in another sense you just max out people’s generosity/spending cash really quickly.

If your rephrase it to “how much should the government spend” you hit general innumeracy problems, and you also hit general innumeracy problems when you specify large, specific numbers of birds.

I suspect Kahneman would have gotten different results had he asked his questions varying questions as: “what percentage of the federal government’s wildlife budget should be spent preventing disease for birds in your city?” vs. "what percentage of the federal government’s wildlife budget should be spent preventing disease for birds in your state?" vs. "what percentage of the federal government’s wildlife budget should be spent preventing disease for birds in the whole country?" (I actually ran this experiment on a convenience sample of students in a 300 level physics class several years ago and got 5%,8% and 10% respectively, but the differences weren’t significant, though the trend was suggestive.)

I suspect the problem isn’t that “brains can’t multiply” so much as “most people are never taught how to think about numbers.”

If anyone knows of further literature on this, feel free to pass it my way.

HPMOR 49: not much here

I thought I posted something about this last weekend, I think tumblr are it. So this will be particularly light. Hariezer notices that Quirrell knows too much (phrased as “his priors are too good”) but hasn’t yet put it together that Quirrell.

There is also (credit where credit is due) a clever working in of the second book into Yudkowsky’s world. The “interdict of Merlin” Yudkowsky invented prevents wizards from writing spells down, so Slytherin’s basilisk was placed in Hogwarts to pass spells on to “the heir of Slytherin.” Voldemort learned those secets and then killed the basilisk, so Hariezer has no shortcut to powerful spells.

HPMOR 50: In Need of an Editor

So this is basically a complete rehash again- it fits into the “Hariezer uses the time turner and the invisibility cloak to solve bullying” mold we’ve already seen a few times. The time turner + invisibility cloak is the solution to all problems, and when Yudkowsky needs a conflict, he throws in bullying. I think we’ve seen this exact conflict with this exact solution at least three other times.

In this chapter, its Hermione being bullied, he protects her by creating an alibi with his timer turner, dressing in an invisibility cloak, and whispering some wisdom in the bullies ear. Because most bullies just need the wisdom of an eleven year old whispered into their ear.

HPMOR 51-63: Economy of Language

So this block of chapters is roughly the length of the Maltese Falcon or the first Harry Potter book, probably 2/3 the length of the Hobbit. This one relatively straight-forward episode of this story is the length of The Maltese Falcon. Basically, the ratio of things-happening/words-written is terrible.

This chapter amounts to a prison break- Quirrell tells Hariezer that Bellatrix Black was innocent, so they are going to break her out. Its a weird section, given how Black escaped in the original novels (i.e. the dementors sided with the dark lord, so all he had to do was go to the dementors say “you are on my side, please let out bellatrix black, and everyone else while you are at it.”)

The plan is to have Hariezer use his patronus while Quirrell travels in snake form in his pouch. They’ll replace Bellatrix with a corpse, so everyone will just think she is dead. It becomes incredibly clear upon meeting Bellatrix that she wasn’t “innocent” at all, though she might be not guilt in the by-reason-of-insanity sense.

This doesn’t phase Hariezer, they just keep moving forward with the plan, which goes awry pretty quickly when an auror stumbles on them. Quirrell ties to kill the auror. Hariezer tries to block the killing spell and ends up knocking out Quirrell and turning his patronus off, and the plan goes to hell.

To escape, Hariezer first scares the dementors off by threatening to blast them with his uber-patronus (even death is apparently scared of death in this story). Then Quirrell wakes up, and with Quirrell’s help he transfigures a hole in the wall, and transfigures a rocket which he straps to his broomstick, and out they fly. The rocket goes so fast the aurors can’t keep up.

Its a decent bit of action in a story desperately needing a bit of action, but its marred by excessive verbosity. We have huge expanses of Hariezer talking with Quirrell, Hariezer talking to himself, Hariezer thinking about dementors, etc. Instead of a tense, taught 50 pages we get a turgid 300.

After they get to safety, Quirrell and Hariezer discuss the horror that is Azkaban. Quirrell tells Hariezer that only a democracy could produce such a torturous prison. A dark lord like Voldemort would have no use for it once got bored:

You know, Mr. Potter, if He-Who-Must-Not-Be-Named had come to rule over magical Britain, and built such a place as Azkaban, he would have built it because he enjoyed seeing his enemies suffer. And if instead he began to find their suffering distasteful, why, he would order Azkaban torn down the next day.

Hariezer doesn’t take up the pro-democracy side, and only time will tell if he goes full-on reactionary like Quirrell by the end of our story. By the end, Hariezer is ruminating on the Milgram experiment, although I don’t think its really applicable to the horror of Azkaban (its not like the dementors are “just following orders”- they live to kill).

Hariezer then uses his time turner to go back to right before the prison breakout, the perfect alibi to the perfect crime.

Dumbledore and Mcgonagall suspect Hariezer played a part in the escape, because of the use of the rocket. They ask Hariezer to use his time turner to send a message back in time (which he wouldn’t be able to do it if he had already used his turner to hide his crime).

Hariezer solves this through the time-turner-ex-machina of Quirrell knowing someone else with a time turner, because when Yudkowsky can’t solve a problem with a time turner, he solves it with two time turners.

HPMOR 64/65: respite

Chapter 64 is again “omake” so I didn’t read it.

Chapter 65 appears to be a pit-stop before another long block of chapters. Hariezer is chaffing that he has been confined to Hogwarts in order to protect him from the dark lord, so he and Quirrell are thinking of hiring a play-actor to pretend to be Voldemort, so that Quirrell can vanquish him.

These were a brief respite between the huge 12- chapter block I just got through and another giant 12 chapter block. Its looking like the science ideas are slowing down in these long chapter blocks, as the focus shifts to action. youzicha has suggested a lot of the rest will be Hariezer cleverly “hacking” his way out of situations, like the rocket in the previous 12 chapter block. The sweet spot for me has been discussing the science presented in these chapters, so between the expected lack of science and the increasing length of chapter blocks, expect slower updates.

HPMOR 66-77: absolutely, appallingly awful

There is a general problem with fanfiction (although usually not in serial fiction where things tend to stay a bit more focused for whatever reason), where the side/B-plots are written entirely in one pass instead of intertwined along side the main plot. Instead of being a pleasant diversion, the side-plot piles up in one big chunk. This is one such side-plot.

Also worth noting these chapters combine basically everything I dislike about HPMOR into one book-length bundle of horror. It was honest-to-god work to continue to power through this section. So this will be just a sketch of this awful block of chapters.

We opened with another superfluous round of the army game, in which nothing notable really happens other than some character named Daphne challenges Neville to “a most ancient duel” WHICH IS APPARENTLY A BATTLE WITH LIGHTSABERS. My eyes rolled so hard I almost had a migraine, and this was the first chapter of the block.

After the battle, Hermoine becomes concerned that women are underrepresented among heros of the wizarding world, and starts a “Society for the Promotion of Heroic Equality for Witches” or SPHEW. They star with a protest in front of Dumbledore’s office and then decide to heroine it up and put an end to bullying. You see, in the HPMOR world, bullying isn’t a question of social dynamics, or ostracizing kids. Bullying is coordinated ambushes of kids in hallways by groups of older kids, and an opportunity for “leveling-up.” The way to fight bullies in this strange world is to engage in pitched wizard-battles in the hallways (having fought an actual bully in reality as a middle schooler I can tell you that at least for me “fight back” doesn’t really solve the problem in any way). In this world, the victims of the bullying are barely mentioned and don’t have names.

And of course, the authority figures like McGonagall don’t even really show up during all of this. Students are constantly attacking each other in the hallways and no one is doing anything about it. Because the way to make your characters seem “rational” is to make sure the entire world is insane.

Things quickly escalate until 44 bullies get together to ambush the eight girls in SPHEW. A back of the envelope calculation suggests Hogwarts has maybe 300 students. So we are to expect slightly more than 10% of the population of students are the sort of “get together and plot an ambush” bullies that maybe you find in 90s highschool TV shows. Luckily, Hariezer had asked Quirrell to protect the girls, so disguised Quirrell takes down the 44 bullies.

We get a “lesson” (lesson in this context means ‘series of insanely terrible ideas’) on “heroic responsibility” in the form of Hariezer lecturing to Harmoine .

The boy didn’t blink. “You could call it heroic responsibility, maybe,” Harry Potter said. “Not like the usual sort. It means that whatever happens, no matter what, it’s always your fault… Following the school rules isn’t an excuse, someone else being in charge isn’t an excuse, even trying your best isn’t an excuse. There just aren’t any excuses, you’ve got toget the job done no matter what.”… Being a heroine means your job isn’t finished until you’ve done whatever it takes to protect the other girls, permanently.”

You know a good way to solve bullying? Expel the bullies. You know who has the power to do that? McGonagall and Dumbledore. A school is a system and has procedures in place to deal with problems. The proper response is almost always “tell an authroity figure you trust.” Being “rational” is knowing when to trust the system to do its job.

In this case, Yudkowsky hasn’t even pulled his usual trick of writing the system as failing- no one even attempts to tell an authority figure about the bullying and no authority figure engages with it, besides Quirrell who engages by disguising himself and attacking students, and Snape who secretly (unknown even to SPHEW) directs SPHEW to where the bullies will be. The system of school discipline stops existing for this entire series of chapters.

We get a final denouement between Hariezer and Dumbledore where the bullying situation is discussed by referencce to Ghandi’s passive resistance in India, WW2 and Churchill, and the larger wizarding war that feel largely overwrought because it was bullying. Big speeches about how Hermoine has been put in danger, etc ring empty because it was bullying. Yes, being bullied is traumatic (sometimes life-long traumatic), but its not WORLD WAR traumatic.

I also can’t help but note the irony that the block’s action largely started on Hermoine’s attempt to “self-actualize” by behaving more heroically, and ends with Dumbledore and Hariezer discussing whether it was the doing the right thing to let Hermoine play her silly little game.

Terrible things in HPMOR

• Lack of science: I have no wrong science to complain about, because these chapters have no science references at all really.
• the world/characters behave in silly ways as a foil for the characters: the authority figures don’t do anything to prevent the escalating bullying/conflict, aside from Snape and Quirrell who get actively involved. The bullying itself isn’t an actual social dynamic, its just general “conflict” to throw at the characters.
• Time turner/invisibility cloak solves all problems: in a slight twist, Snape gives a time turner to a random student and uses her to pass messages to SPHEW so they can find and attack bullies
• Superfluous retreads of previous chapters: the army battle that starts its off, much of the bullying is retread. There are several separate bully-fights in this block of chapters.
• Horrible pacing: this whole block of chapters is a B-plot roughly the length of an entire book.
• Stilted language. Everyone refers to the magic light sabers as “the most ancient blade” every time they reference it

Munchkinism

I’ve been meaning make a post like this for several weeks, since yxoque reminded me of the idea of the munchkin. jadagul mentioned me in a post today that reminded me I had never made it. Anyway:

I grew up playing Dungeons and Dragons, which was always an extremely fun way to waste a middle school afternoon. The beauty of Dungeons and Dragons is that it provides structure for a group of kids to sit around and tell a shared story as a group. The rules of the game are flexible, and one of the players acts as a living rule-interpreter to guide the action and keep the story flowing.

Somehow, every Dungeons and Dragons community I’ve ever been part of (middle school, highschool and college) had the same word for a particularly common failure mode of the game, and that word was munchkin, or munchkining (does anyone know if there was a gaming magazine that used this phrase?). The failure is simple - people get wrapped up in the letter of the rules, instead of the spirit, and start building the most powerful character possible instead of a character that makes sense as a role. Instead of story flow, the game gets bogged down in dice rolls and checks so that the munchkins can demonstrate how powerful they are. Particularly egregious munchkins have been known to cheat on their character creation rolls to boost all their abilities. With one particular group in highschool, I witnessed one particularly hot-headed munchkin yell at everyone else playing the game when the dungeon master (the human rule interpreter) slightly modified a rule and ended up weakening the muchkin’s character.

The frustrating thing about HPMOR is that Hariezer is designed, as yxoque pointed out, to be a munchkin- using science to exploit the rules of the magical world (which could be an interesting question), but because Yudkowsky is writing the rules of magic as he goes, Hariezer is essentially cheating at a game he is making up on the fly.

All of the cleverness isn’t really cleverness- its easy to find loopholes in the rules you yourself create as you go especially if you created them to have giant loopholes.

In Azkaban, Hariezer uses science to escape by transfiguring himself a rocket. This only makes sense because for some unknown reason magic brooms aren’t as fast as rockets.

In one of his army games, Hariezer uses gloves with gecko setae to climb down a wall, because for some reason broomsticks aren’t allowed. For some reason, there is no ‘grip a wall’ spell.

Yudkowsky isn’t bound by the handful of constraints in Rowling’s world (where Dementors represent depression, not death), hell he doesn’t even stick to his own constraints. In Hariezer’s escape from Azkaban he violates literally the only constraint he had laid down (don’t transfigure objects into something you plan to burn).

Every other problem in the story is solved by using the time turner as a deus ex machina. Even when plot constraints mean Hariezer’s time turner can’t be used, Yudkowsky just introduces another time turner rather than come up with a novel and clever solution for his characters.

Hariezer’s plans in HPMOR work only because the other characters become temporarily dumb to accommodate his “rationality” and because the magic is written around the idea of him succeeding.

"Genre savvy"

So a lot of people have asked me to take a look at the Yudkowsky writing guide, and I will eventually (first I have to finish HPMOR ,which is taking forever because I’m incredibly bored with it, but I HAVE MADE A COMMITMENT- hopefully more HPMOR live blogging after Thanksgiving).

But I did hit something that also applies to HPMOR, and a lot of other stories. Yudkowsky advocated that characters “have read the books you’ve read” so they can solve those problems. One of my anonymous asked used the phrase “genre savvy” for this- and google lead me to TV tropes page. The problem with this idea is that as soon as you insert a genre savvy character, your themes shift, much like having a character break the fourth wall. Suddenly your story is about stories. Your story is now a commentary on the genre/genre conventions.

Now, there are places where this can work fairly well- those Scream movies, for instance, were supposed to (at least in part) ABOUT horror movies as much as they WERE horror movies. Similarly, every fan-fiction is (on some level) a commentary on the original works, so “genre savvy” fan fiction self-inserts aren’t nearly as bad an idea as they could be.

HOWEVER (and this is really important)- MOST STORIES SHOULD NOT BE ABOUT STORIES IN THE ABSTRACT/GENRE/GENRE CONVENTIONS, and this means it is a terrible idea having characters that constantly approach things on a meta level “this is like in this fiction book I read” . If you don’t have anything interesting to say about the actual genre conventions, then adding a genre savvy character is almost certainly going to do you more harm then good. If you are bored with a genre convention, you’ll almost certainly get more leverage out of subverting it (if you lead both the character AND the reader to expect a zig, and instead they get a zag it can liven things up a bit) then by sticking in a genre-savvy character.

Sticking in a genre-savvy character just says “look at this silly convention!” and then when that convention is used anyway, it just feels like the writer being a lazy hipster. Sure your reader might get a brief burst of smugness “he/she’s right, all those genre books ARE stupid! Look how smart I am!” but you aren’t really moving your story forward. You are critiquing lazy conventions while also trying to use them.

If you don’t like the conventions of a genre, don’t write in that genre, or subvert them to make things more interesting. Or simply refuse to use those conventions all together, go your own way.

HPMOR 78: action without consequences

A change of tactics- this is chapter is part of another block of chapters, but I’m having trouble getting through it, so I’m going to write in installments chapter by chapter, instead of a dump on a 12 chapter block again.

This chapter is another installment of Quirrell’s battle game. This time, the parents are in the stands, which becomes important when Hermione out-magics Draco.

Afterwards, Draco is upset because his father saw him getting out magiced by a mud blood. This causes Draco, in an effort to save face or get revenge or something, to send a note to lure Hermione to meet him alone. Then, cut to the next morning- Hermione is arrested for the attempted murder of Draco. So thats it for the chapter summary.

But I want to use this chapter to touch on something that has bothered me about this story- most of the action is totally without stakes or consequences for the characters. As readers we don’t care what happens. In the case for the Quirrell battle game, the prize for victory was already handed out at the Christmas break, none of the characters have anything on the line, and the story doesn’t really act like winning or losing has real consequences for anyone involved. A lot is happening, but its ultimately boring.

The same thing happened in the anti-bullying chapters. Most of the characters being victimized lack names or personalities. Hermione and team aren’t defending characters we care about and like, they are fighting the abstract concept of bullying (and the same is largely true of Hariezer’s forays into fighting bullies.)

Part of this is because of the obvious homage to Ender’s game, without understanding Ender’s game was doing something very different- the whole point of Ender’s Game is that the series of games absolutely do feel low stakes. Even when Ender kills another kid, its largely shrugged off as Ender continuing to win (which is the first sign something a bit deeper is happening). It supposed to feel game-y so the reader rides along with Ender and doesn’t viscerally notice the genocide happening. The contrast between the real world stakes and the games being played is the point of the story. Where Ender’s game failed for me is after the battles- we don’t feel Ender’s horror at learning what happened. Sure Ender becomes speaker for the dead, but the book doesn’t make us feel Ender’s horror the same way we ride along with the game stuff. I think this is why so many people I know largely missed the point of the book and walked away with “War games are awesome!” (SCROLL DOWN FOR Fight Club FOOTNOTE THAT WAS MAKING THIS PARAGRAPH TOO LONG) But I digress- if your theme isn’t something to do with the the connection between war and games and the way people perceive violence vs games, etc, turning down the emotional stakes and the consequences for the characters make your story feel like reading a video game play-by-play, which is horribly boring.

If you cut out all the Quirell game chapters after chapter 35, no one would notice- there is nothing at stake.

ALSO- this chapter has an example of what I’ll call “DM munchkining” i.e. its easy to Munchkin when you write the rules. Hariezer is looking for powerful magic to aid him in battle, and starts reading up on potion making. He needs a way to make potions in the woods without magical ingredients, so he deduces by reading books that you don’t really need a magical ingredient, you get out from a potion ingredient what went in to making it. So Hariezer makes a potion with acorns that gets back all the light that went in to creating the acorn via photosynthesis. My point here is that this rule was created in this chapter entirely to be exploited by Hariezer in this battle. In a previous battle chapter, Hariezer exploits the fact that metal armor can block spells, a rule created specifically for that chapter to be exploited. Its not munchkining, its calvinball.

FOOTNOTE: This same problem happens with Fight Club. The tone of the movie builds up Tyler Durden as this awesome dude and the tone doesn’t shift when Ed Norton’s narrator character starts to realize how fucked everything is. So you end up with this movie thats suppose to be satirical but no one notices. They rebel against a society they find dehumanizing, BY CREATING A SOCIETY WHERE THEY LITERALLY HAVE NO NAMES, but the tone is strong enough that people are like “GO PROJECT MAYHEM! WE SHOULD START A FIGHT CLUB!”

HPMOR 79

This chapter continues on from 78. Hermione has been arrested for murder, but Hariezer now realizes in a sudden insight that she has given a false memory.

Hariezer also realizes this is how the Weasley twins planted Rita Skeeter’s false news story- they simply memory charmed Rita. Of course, this opens up more questions then it solves- if false memory charming can be done with such precision, wouldn’t there be a rash of manipulations of this type? Its such an obvious manipulation technique that chapters 24-26 with the Fred and George “caper” was written in a weirdly non-linear style to try to make it seem more mysterious.

Anyway, Hariezer tells the adults who start investigating who might have memory charmed Hermione (you’d think wizard police would do some sort of investigation but its HPMOR, so the world needs to be maximally silly as a foil to Hariezer).

And then he has a discussion with the other kids who are bad mouthing Hermione:

Professor Quirrell isn’t here to explain to me how stupid people are, but I bet this time I can get it on my own. People do something dumb and get caught and are given Veritaserum. Not romantic master criminals, because they wouldn’t get caught, they would have learned Occlumency. Sad, pathetic, incompetent criminals get caught, and confess under Veritaserum, and they’re desperate to stay out of Azkaban so they say they were False-Memory-Charmed. Right? So your brain, by sheer Pavlovian association, links the idea of False Memory Charms to pathetic criminals with unbelievable excuses. You don’t have to consider the specific details, your brain just pattern-matches the hypothesis into a bucket of things you don’t believe, and you’re done. Just like my father thought that magical hypotheses could never be believed, because he’d heard so many stupid people talking about magic. Believing a hypothesis that involves False Memory Charms is low-status.

This sort of thing bothers the hell out me. Not only is cloying elitism creeping in, but in HPMOR as in the real world, arguments regarding “status” are just thinly disguised ad-hominems. True or not true, they aren’t really attacking an argument, just the people making them.

After all, if we fall back on the “Bayesian conspiracy” confessing to a crime/having a memory of a crime is equal evidence for having done the crime and having been false memory charmed, so all the action here is in the prior. CLAIMING a false memory charm is evidence of nothing at all.

So, if the base rate of false memory charms is so low that its laughable and “low status,” then the students are correctly using Bayesian reasoning.

Although Hariezer may point out that they aren’t taking into account evidence about what sort of person Hermione is, but if the base rates of false memory charms are really so low that is unlikely to matter much- after all Hariezer doesn’t have any specific positive evidence she was false memory charmed, and she has been behaving strangely toward Draco for awhile (which Hariezer suggests is a symptom of the way the perpetrator went about the false memory charm, but could just as easily be evidence she did it- the action is still in the prior).

Similarly, his father didn’t believe in magic because it SHOULDN’T have been believed- until the story begins he has supposedly lived his whole life in our world- where magic is quite obviously not a real thing, regardless of “status. “

OF COURSE- if the world were written as a non-silly place, the base rate for false memory charms would be through the roof and everyone would say “yea, she was probably false memory charmed! Who just blurts out a confession?” and the wizard cops would just do their job.

Remember when

Way back in chapter 20 something, Quirrell gave Hariezer Roger Bacon’s magic diary, and it was going to jump start his investigation of the rules of magic? And then it was literally never mentioned again? The aptly named Checkov’s Roger Bacon’s Magi-science Diary probably applies here.

HPMOR 80

Apparently in the wizarding world, the way a trial is conducted involves a bunch of politicians voting if someone is guilty or innocent, so in this chapter the elder Malfoy uses his influence to convict Hermione. Not much to this chapter really.

BUT in some asides, we do get some flirting with neoreaction:

At that podium stands an old man, with care-lined face and a silver beard that stretches down below his waist; this is Albus Percival Wulfric Brian Dumbledore… Karen Dutton bequeathed the Line to Albus Dumbledore… each wizard passing it to their chosen successor, back and back in unbroken chain to the day Merlin laid down his life. That (if you were wondering) is how the country of magical Britain managed to elect Cornelius Fudge for its Minister, and yet end up with Albus Dumbledore for its Chief Warlock. Not by law (for written law can be rewritten) but by most ancient tradition, the Wizengamot does not choose who shall preside over its follies. Since the day of Merlin’s sacrifice, the most important duty of any Chief Warlock has been to exercise the highest caution in their choice of people who are both good and able to discern good successors.

And we get the PC/NPC distinction used by Hariezer to separate himself from the sheeple:

The wealthy elites of magical Britain have collective force, but not individual agency; their goals are too alien and trivial for them to have personal roles in the tale. As of now, this present time, the boy neither likes nor dislikes the plum-colored robes, because his brain does not assign them enough agenthood to be the subjects of moral judgment. He is a PC, and they are wallpaper.

Hermione is convicted and Hariezer is sad he couldn’t figure out something to do about it (he did try to threaten the elder Malfoy to no avail).

HPMOR 81

Our last chapter ended with Hermione in peril- she was found guilty of the attempted murder of Draco! How will Hariezer get around this one?

Luckily the way the wizard world justice system works is fucking insane- being found guilty puts Hermione in the Malfoy’s “blood debt.” So Hariezer tells Malfoy:

By the debt owed from House Malfoy to House Potter!…I’m surprised you’ve forgotten…surely it was a cruel and painful period of your life, laboring under the Imperius curse of He-Who-Must-Not-Be-Named, until you were freed of it by the efforts of House Potter. By my mother, Lily Potter, who died for it, and by my father, James Potter, who died for it, and by me, of course.

So Hariezer wants the blood debt transferred to him so he can decide Hermione’s fate (what a convenient and ridiculous way to handle a system of law and order).

But blood debts don’t transfer in this stupid world, instead you also have to pay money. So Malfoy demands something like twice the money in Hariezer’s vault. Hariezer waffles a bit, but decides to pay. Because the demand is such a large sum, this will involve going into debt to the Malfoys.

And then things get really stupid- Dumbledore says, as guardian of Hariezer’s vault he won’t let the transaction happen.

I’m - sorry, Harry - but this choice is not yours - for I am still the guardian of your vault.”

“What? ” said Harry, too shocked to compose his reply.

"I cannot let you go into debt to Lucius Malfoy, Harry! I cannot! You do not know - you do not realize -"

So… here is a question- if Hariezer is going to go into a lot of debt to pay Malfoy how does blocking him access to his money help avoid the debt? Wouldn’t Hariezer just take out a bigger loan from Malfoy?

Anyway, despite super rationality, Hariezer doesn’t think through how stupid Dumbledore’s threat is. Hariezer instead threatens to destroy Azkaban if Dumbledore won’t let him pay Malfoy, so Dumbledore relents.

Malfoy tries to weasel out of this nebulous blood debt arrangement because the rules of wizard justice change on the fly, but Hermione swears allegiance to House Potter and that prevents Malfoy’s weasel attempt.

I acknowledge the debt, but the law does not strictly oblige me to accept it in cancellation,” said Lord Malfoy with a grim smile. “The girl is no part of House Potter; the debt I owe House Potter is no debt to her…

And Hermione, without waiting for any further instructions, said, the words spilling out of her in a rush, “I swear service to the House of Potter, to obey its Master or Mistress, and stand at their right hand, and fight at their command, and follow where they go, until the day I die.”

The implications here are obvious- if you saved all of magical britain from a dark lord, and literally everyone owes you a “blood debt” you are totally above the law. Hariezer should just steal the money he owes Malfoy from some other magical families.

HPMOR 82

So the trial is wrapped up, but to finish off the section we get a long discussion between Dumbledore and Hariezer.

First, credit where credit is due there is an atypical subversion here- now its Dumbledore attempting to give a rationality lesson to Hariezer, and Hariezer agrees that he is right. Its an attempt to mix up the formula a bit, and I appreciate it even if the rest of this chapter is profoundly stupid.

So what is the rationality lesson here?

“Yes," Harry said, "I flinched away from the pain of losing all the money in my vault. But Idid it! That’s what counts! And you -” The indignation that had faltered out of Harry’s voice returned. “You actually put a price on Hermione Granger’s life, and you put it below a hundred thousand Galleons!”

"Oh?" the old wizard said softly. "And what price do you put on her life, then? A million Galleons?"

"Are you familiar with the economic concept of ‘replacement value’?" The words were spilling from Harry’s lips almost faster than he could consider them. "Hermione’s replacement value is infinite! There’s nowhere I can go to buy another one!”

Now you’re just talking mathematical nonsense, said Slytherin. Ravenclaw, back me up here?

"Is Minerva’s life also of infinite worth?" the old wizard said harshly. "Would you sacrifice Minerva to save Hermione?"

"Yes and yes," Harry snapped. "That’s part of Professor McGonagall’s job and she knows it."

"Then Minerva’s value is not infinite," said the old wizard, "for all that she is loved. There can only be one king upon a chessboard, Harry Potter, only one piece that you will sacrifice any other piece to save. And Hermione Granger is not that piece. Make no mistake, Harry Potter, this day you may well have lost your war."

Basically, the lesson is this- you have to be willing to put a value on human life, even if it seems profane. Its actually a good lesson and very important to learn. If everyone was more familiar with this, the semi frequent GOVERNMENT HEALTHCARE IS DEATH PANELS panic would never happen. Although I’d add a caveat- anyone who has worked in healthcare does this so often that we start to make a mistake the other way (forgetting that underneath the numbers are actual people).

Anyway, to justify the rationality lesson further we get a reference to some of Tetlock’s work (note: I’m unfamiliar with the work cited here, so I’m taking Yudkowsky at his word- if you’ve read the rest of my hpmor stuff, you know this is dangerous).

You’d already read about Philip Tetlock’s experiments on people asked to trade off a sacred value against a secular one, like a hospital administrator who has to choose between spending a million dollars on a liver to save a five-year-old, and spending the million dollars to buy other hospital equipment or pay physician salaries. And the subjects in the experiment became indignant and wanted to punish the hospital administrator for even thinking about the choice. Do you remember reading about that, Harry Potter? Do you remember thinking how very stupid that was, since if hospital equipment and doctor salaries didn’t also save lives, there would be no point in having hospitals or doctors? Should the hospital administrator have paid a billion pounds for that liver, even if it meant the hospital going bankrupt the next day?

To bring it home, we find out that Voldemort captured Dumbledore’s brother and demanded ransom, and Mad Eye counseled thusly

"You ransom Aberforth, you lose the war," the man said sharply. "That simple. One hundred thousand Galleons is nearly all we’ve got in the war-chest, and if you use it like this, it won’t be refilled. What’ll you do, try to convince the Potters to empty their vault like the Longbottoms already did? Voldie’s just going to kidnap someone else and make another demand. Alice, Minerva, anyone you care about, they’ll all be targets if you pay off the Death Eaters. That’s not the lesson you should be trying to teach them."

So instead of ransoming Aberforth he burned Lucious Malfoy’s wife alive (or at least convinced the death eaters that he did). That way they would think twice about targeting him.

I think the rationality lesson is fine and dandy, just one problem- this situation is not at all like the hospital administrator in the example given. The problem here is that the idea of putting a price on a human life is only a useful concept in day-to-day reality where money has some real meaning. In an actual war, even one against a sort of weird guerrilla army of dark wizards money only becomes useful if you can exchange it for more resources, and in the wizard war resources means wizards.

Ask yourself this- would a death eater target someone close to Dumbledore even if there was no possibility of ransom? OF COURSE THEY WOULD- the whole point is defeating Dumbledore, the person standing against them. Voldemort wouldn’t ask for ransom, because its a stupid thing to do- he would kill Aberforth and send pieces of him to Dumbledore by owl. This idea that ransoming makes targets of all of Dumbledore’s allies is just idiotic- they are already targets.

Next, ask yourself this- does Voldemort have any use for money? Money is an abstract, useful because we can exchange for useful things. But its pretty apparent that Voldemort doesn’t really need money- he has no problem killing, taking and stealing. The parts of magical Britain that are willing to stand up to him won’t sell his death eaters goods at any price, and the rest are so scared they’ll give him anything for free.

Meanwhile, Dumbledore is leading a dedicated resistance- basically a volunteer army. He doesn’t need to buy people’s time, they are giving it freely! Mad Eye himself notes that he could ask the Longbottoms or the Potters to empty their vaults and they would. What the resistance needs isn’t money, its people willing to fight. So in the context of this sort of war, and able fighting man like Aberforth is worth basically infinite money- money is common and useless and people willing to stand up to Voldemort are in extremely tight supply.

It would have made a lot more sense to have Voldemort ask for prisoner exchange or something like that. Aberforth in exchange for Bellatrix Black. Then both sides would be trading value for value. But then the Tetlock reference wouldn’t be nearly as on-the-nose.

At least this chapter makes clear the reason for the profoundly stupid wizard justice system and the utterly absurd blood-debt-punishment system. The whole idea was to arrange things so Hariezer could be asked to pay a ransom to Luscious Malfoy, so the reader can learn about Tetlock’s research/putting a price on lives,etc.

At least I only have like 20 chapters of this thing left.

Name of the Wind bitching

Whelp, Kvothe’s “awesomeness” has totally overwhelmed the narrative. Kvothe now has several “awesome” skills- he plays music so well that he was spontaneously given 7 talons (which is 7 times the standard Rothfuss unit for “a lot of money”). He plays music for money in a low-end tavern.

He is a journeyman artificer, which means he can make magic lamps and what not and sell them for cash. He is brilliant enough that he could easily tutor students. He has two very wealthy friends who he could borrow from.

AND YET he is constantly low on cash. To make this seem plausible, the book is weighed down by super long exposition in which Kvothe explains to the reader why all these obvious ways to make money aren’t working for him. When Kvothe isn’t explaining it to the reader directly, we cut to the framing story where Kvothe is explaining it to his two listeners. The book is chock-full of these paragraphs that are like “I know this is really stupid but here is why it actually makes sense.” Removing all this justification/exposition would probably cut the length of the book by at least 1/4.

I could look past all of this if we were meeting other interesting characters at wizard school, but that isn’t happening. Kvothe has two good friends among the students, Wil and Sim. I’ve read countless paragraphs of banter between them and Kvothe, but I don’t know what they study, or really anything about them other than one has an accent.

Another character Auri, a homeless girl who is that fantasy version of “mentally ill” that just makes people extra quirky, became friends with Kvothe off screen. Literally, we find out she exists after she has already listened to Kvothe playing music for days. She shows up for a scene then vanishes again for awhile.

And we get a love interest who mostly just banters wittily with Kvothe and then vanishes. After pages of witty banter, Kvothe will then remind the reader he is shy around women (despite, you know, having just wittily bantered for pages, because that’s how characterization works in this novel).

HPMOR bitching

Much like my previous name of the wind complaints, HPMOR is heavy with exposition- and for a similar reason. Hariezer is too “awesome” which leads to heavy-handed exposition (if for slightly different reasons than name of the wind).

The standard rule of show,don’t tell implies that the best way to teach your audience something in a narrative is to have your characters learn from experience. Your characters need to make a mistake, or have something backfire. That way they can come out the other side stronger, having learned something. If you don’t trust your audience to have gotten the lesson, you can cap it off with some exposition outlining exactly what you want to learn, but the core of the lesson should be taught by the characters experience.

But Yudkowsky inserted a Hariezer fully-equipped with the “methods of rationality.” So we get lots of episodes that set-up a conflict, and then Hariezer has a huge dump of exposition that explains why its not really a problem because rationality-judo, and the tension drains away. It would be far better to have Hariezer learn over time, so the audience can learn along with him.

So Hariezer isn’t going to grow, he is just going to exposition dump most of his problems away. We can at least watch him explore the world, right? After all, Yudkowsky has put a “real scientist” into Hogwarts so we can finally see what material you actually learn at wizard school! All that academic stuff missing from the original novels! NOPE- we haven’t had a single class in the last 60 chapters. Hariezer isn’t even learning magic in a systematic way.

I really, really don’t see what people see in this. The handful of chapters I found amusing feel like an eternity ago, it ran off the rails dozens of chapters ago! People sell the story as “using the scientific method in JK Rowling’s universe” but a more accurate description would be “it starts as using the scientific method in JK rowling’s universe, but stops doing that around chapter 25 or so. Then mostly its just about a war game, with some political maneuvering.”

HPMOR 83-84

These are just rehashes of things we’ve already been presented with (so many words, so little content). The other students still think Hermione did it (although this is written in an akward tell rather than show style- Hariezer tells Hermione what is going on, rather than Hermione or the reader experiencing it). We get gems of cloying elitism like this:

Hermione, you’ve told me a lot of times that I look down too much on other people. But if I expected too much of them - if I expected people to get things right - I really would hate them, then. Idealism aside, Hogwarts students don’t actually know enough cognitive science to take responsibility for how their own minds work. It’s not their fault they’re crazy.

There is one bit of new info- as part of this investigation of the attempted murder of Draco, I guess Quirrell was investigated, and the aurors seem to think he is some missing wizard lord or something. This is totally superfluous, I assume we all know Quirrell is Voldemort. I’m hoping this doesn’t turn into a plot line.

And finally, Quirrell tries to convince Hermione to leave and go to a wizard school where people don’t think she tried to kill someone. This is fine, but in part of it, Quirrell gives us this gem on being a hero:

Long ago, long before your time or Harry Potter’s, there was a man who was hailed as a savior. The destined scion, such a one as anyone would recognize from tales, wielding justice and vengeance like twin wands against his dreadful nemesis…

"In all honesty…I still don’t understand it. They should have known that their lives depended on that man’s success. And yet it was as if they tried to do everything they could to make his life unpleasant. To throw every possible obstacle into his way. I was not naive, Miss Granger, I did not expect the power-holders to align themselves with me so quickly - not without something in it for themselves. But their power, too, was threatened; and so I was shocked how they seemed content to step back, and leave to that man all burdens of responsibility. They sneered at his performance, remarking among themselves how they would do better in his place, though they did not condescend to step forward.”

So… the people seem mostly to rally around Dumbledore. He has a position of power and influence because of his dark-wizard vanquishing deeds. There aren’t a lot of indications people are actively attempting to make Dumbledore’s life unpleasant, he has the position he wants, turned down the position of minister of magic,etc. People are mostly in awe of Dumbledore.

But there is some other hero, we are supposed to believe, who society mocked? I can’t help but draw parallels to Friendly AI research here…

HPMOR 85

A return to my blogging obsession of old (which has been a slog for at least 20 chapters now, but if there is one thing that is true of all phds- we finish what we fucking start, even if it’s an awful idea).

This chapter is actually not so bad, mostly Hariezer just reflecting on the difficulty of weighing his friends lives against “the cause” as Dumbledore suggested he failed to do with Hermione in her trial a few chapters ago.

There are some good bits. For instance, this interesting bit about bow and arrow’s in Australia:

A year ago, Dad had gone to the Australian National University in Canberra for a conference where he’d been an invited speaker, and he’d taken Mum and Harry along. And they’d all visited the National Museum of Australia, because, it had turned out, there was basically nothing else to do in Canberra. The glass display cases had shown rock-throwers crafted by the Australian aborigines - like giant wooden shoehorns, they’d looked, but smoothed and carved and ornamented with painstaking care. In the 40,000 years since anatomically modern humans had migrated to Australia from Asia, nobody had invented the bow-and-arrow. It really made you appreciate how non-obvious was the idea of Progress.

I always thought the fact that Australians (and lot of small islanders) lost the bow and arrow (which is interesting! They had it and then they forgot about it!) was an interesting observation about the power of sharing ideas and the importance of large groups for creativity. Small, isolated populations seem to lose the ability to innovate. Granted, almost all of my knowledge about this comes from one anthropology course I only half remember.

And of course there are always some sections that filled me with rage-

Even though Muggle physics explicitly permitted possibilities like molecular nanotechnology or the Penrose process for extracting energy from black holes, most people filed that away in the same section of their brain that stored fairy tales and history books, well away from their personal realities:

Molecular nanotechnology is just the words that sci-fi authors (and Eric Drexler) use for magic. And the nearest black hole is probably something like 2000 light years away. The reason people treat this stuff as far from their personal reality is exactly the same reason Yudkowsky treats it as far from his personal reality- IT IS. Black holes are neat, and GR is a ton of fun, but we aren’t going to be engineering with black holes in my lifetime.

No surprise, then, that the wizarding world lived in a conceptual universe bounded - not by fundamental laws of magic that nobody even knew - but just by the surface rules of known Charms and enchantments…Even if Harry’s first guess had been mistaken, one way or another it was still inconceivable that the fundamental laws of the universe contained a special case for human lips shaping the phrase ‘Wingardium Leviosa’. …What were theultimate possibilities of invention, if the underlying laws of the universe permitted an eleven-year-old with a stick to violate almost every constraint in the Muggle version of physics? You know what would be awesome? IF YOU GOT AROUND TO DOING SOME EXPERIMENTS AND EXPLORING THIS IDEA. The absolute essence of science is NOT asking these questions, it’s deciding to try to find out the fucking answers! You can’t be content to just wonder about things, you have to put the work in! Hariezer’s wonderment never gets past the stoned-college-kid wondering aloud and into ACTUAL exploration, and its getting really frustrating. YOU PROMISED ME YOU WERE GOING TO USE THE SCIENTIFIC METHOD TO LEARN THE SECRETS OF MAGIC. WAY BACK IN THE EARLY CHAPTERS.

Anyway, towards the end of the ruminations, Fawkes visits Hariezer and basically offers to take him to Azkaban to try to take out the evil place. Hariezer (probably wisely) decides not to go. And the chapter ends.

HPMOR 86

I just realized I have like 145 followers (HOLY SHIT!) and they probably came for the HPMOR thing. So I better keep the updates rolling!

Anyway, this chapter is basically Hariezer and friends (Dumbledore, Snape, Mcgonagall, Mad-eye Moody) all trying to guess who might have been responsible for trying to frame Hermione. No real conclusions are drawn, not much to see her.

A few notable things here- magic apparently works by the letter of the law, rather than the spirit:

You say someone with the Dark Mark can’t reveal its secrets to anyone who doesn’t already know them. So to find out how the Dark Mark operates, write down every way you can imagine the Dark Mark might work, then watch Professor Snape try to tell each of those things to a confederate - maybe one who doesn’t know what the experiment is about - I’ll explain binary search later so that you can play Twenty Questions to narrow things down - and whatever he can’t say out loud is true. His silence would be something that behaves differently in the presence of true statements about the Mark, versus false statements, you see.

Luckily, Voldemort thought of the test, thus freeing Snape to tell how the mark actually works:

The Dark Lord was no fool, despite Potter’s delusions. The moment such a test is suspected, the Mark ceases to bind our tongues. Yet I could not hint at the possibility, but only wait for another to deduce it.

Why not just make sure the death eaters don’t actually know the secrets of the mark? Seems like memory spells are everywhere already, and it would be way easier than this silly logic puzzle.

Finding out the secrets of the dark mark prompts Hariezer to try a Bayesian estimate of whether Voldemort is actually dangerous. I repeat that for emphasis:

Harry Potter, first year of Hogwarts who has only really succeeded at 1 thing in his learn-the-science-of-magic plan (partial transfiguration), and who knows he is not the most dangerous wizard at Hogwarts (Quirrel, Dumbledore), wonders whether Voldemort could possibly be a threat.

Here are some of the things he considers:

Harry had been to a convocation of the Wizengamot. He’d seen the laughable ‘security precautions’, if you could call them that, guarding the deepest levels of the Ministry of Magic. They didn’t even have the Thief’s Downfall which goblins used to wash away Polyjuice and Imperius Curses on people entering Gringotts … [if it] took you more than ten years to fail to overthrow the government of magical Britain, it meant you were stupid. But might they have some other precautions? Maybe they use some sort of secret precautions Harry himself doesn’t yet know about yet? Or might the wizards of the Wizengamot be pretty powerful in their own right?

There were hypotheses where the Dark Lord was smart and the Order of the Phoenixdidn’t just instantly die, but those hypotheses were more complicated and ought to get complexity penalties. After the complexity penalties of the further excuses were factored in, there would be a large likelihood ratio from the hypotheses ‘The Dark Lord is smart’ versus ‘The Dark Lord was stupid’ to the observation, ‘The Dark Lord did not instantly win the war’. That was probably worth a 10:1 likelihood ratio in favor of the Dark Lord being stupid… but maybe not 100:1. You couldn’t actually say that ‘The Dark Lord instantly wins’ had a probability of more than 99 percent, assuming the Dark Lord started out smart; the sum over all possible excuses would be more than .01.

Dude, do you even Bayesian? Probability the dark mark still works if Voldemort is dead. ~0 (everyone who knows magic thinks that the mark still existing is proof he is still out there). Given that Voldemort is alive, probability he successfully complete some sort of immortality ritual ~1. Probability someone who completed an immortality ritual knows more magic than (and therefore is a threat to) Hariezer Yudotter ~1.

So given that the dark mark is still around, Voldie is crazy dangerous, regardless of priors or base rates.

It’s helpful to look at where the information is, instead of trying to estimate the probability Voldemort could have instantly killed some of the most powerful wizards on the fucking planet.

Anyway….

OH, another thing that happens- Hariezer challenges Mad-eye to a little mini duel. Guess how he solves the problem of winning against Mad-eye? Any ideas? What could he use? I’ll give you a hint, it rhymes with time turner. This story really should be called Harry Potter And The Method of Time Turners. Seriously- time turners solve basically all the problems in this book. Anyway, he goes to Flitwick, learns a bending spell, and then time turners back into the room to pop Moody.

It’s not actually a bad scene, there is a bit of action and it moves pretty quickly. The problem is that the time turner solution is so damn boring at this point.

Also, we find out in this chapter that everyone believes Quirrell is really somebody named David Monroe whose family was killed by Voldemort and who was a leader during the war against Voldemort.

So we have some potential possibilities-

1. Voldemort was impersonating/half-invented the personality of David Monroe in order to play both sides during the war. After all, all of Monroe’s family was killed but him. Maybe all of Monroe’s family was killed, including him, and Voldemort started impersonating the dead guy. This could be a neat dynamic I guess. Could “Voldemort” have been a Watchmen style plan to unite magical Britain against a common threat that went awry for Monroe/Riddle? Quirrell really did get body snatched in this scenario. We could imagine an ending here where Monroe/Riddle are training Potter to be the leader of magical Britain that Monroe/Riddle wanted to be.

2. Monroe was a real dude, Voldemort body-snatched him, and now you’ve got Monroe brain fighting Voldemort brain inside. For some reason, they are impersonating Quirrell?

If its not the first scenario, I’m going to be sort of annoyed, because scenario 2 doesn’t provide us with much reason for the weird Monroe bit- you could just give Quirrell all of Monroe’s backstory.

Anyway, 86 chapters to go, I think this damn thing is going to clock in around 120 when all is said and done. ::sigh:: Time for a scotch.

HPMOR 87: skinner box your friends

Hariezer is worried Hermione will be uncomfortable around him after the trial. So what is his solution?

"I was going to give you more space," said Harry Potter, "only I was reading up on Critch’s theories about hedonics and how to train your inner pigeon and how small immediate positive and negative feedbacks secretly control most of what we actually do, and it occurred to me that you might be avoiding me because seeing me made you think of things that felt like negative associations, and I really didn’t want to let that run any longer without doing something about it, so I got ahold of a bag of chocolates from the Weasley twins and I’m just going to give you one every time you see me as a positive reinforcement if that’s all right with you -"

Now, this idea of positive/negative reinforcement is an old one, and goes back to probably the psychologists associated with behaviorism (BF Skinner, Pavlov, etc).

The weird thing is, there is no “Critch” I can find associated with the behaviorists, or really any of the stuff attributed above. I also emailed my psych friend, who also has never heard of it (but “it’s not really my field at all”). I’m thinking there is like a 90% chance that Yudkowsky just invented a scientist here? Why not just say BF Skinner, or Pavlov here? WHAT IS GOING ON HERE?

Anyway, Hermione and Hariezer are brainstorming ways to make money when they get into an argument because Hariezer has been sciencing with Draco:

"You were doing SCIENCE with him? " "Well -" "You were doing SCIENCE with him? You were supposed to be doing science with ME! " Hermione, I get it. You wanted to figure out how magic works you’ve got some curiosity about the world. And now you think Hariezer kept his program going, but cut you out of the big discoveries, will leave you off the publications. But I’ve got news for you, girl, he hasn’t been doing science WITH ANYONE for like 60 chapters now. HE JUST FORGOT ABOUT IT.

Anyway, this argument blows up, and Hariezer explains puberty:

But even with all that weird magical stuff letting me be more adult than I should be, I haven’t gone through puberty yet and there’s no hormones in my bloodstream and my brain is physically incapable of falling in love with anyone. So I’m not in love with you! I couldn’t possibly be in love with you!

And then drops some evopsych

and besides I’ve been reading about evolutionary psychology, and, well, there are all these suggestions that one man and one woman living together happily ever afterward may be more the exception rather than the rule, and in hunter-gatherer tribes it was more often just staying together for two or three years to raise a child during its most vulnerable stages - and, I mean, considering how many people end up horribly unhappy in traditional marriages, it seems like it might be the sort of thing that needs some clever reworking - especially if we actually do solve immortality To the story’s credit, this works about as well as you’d expect and Hermione storms off.

I think the evopsych dropping could have been sort of funny if it were played more for laughs (Hariezer’s inept way of calming Hermione down), but here it just seems like a way to shoehorn this bit of evopsych into the story.

The final scene in the chapter is played for laughs, with another student coming over after seeing Hermione storm off and saying “Witches! go figure, huh?”

HPMOR 88: in which I complain about a lack of time turners

The problem with solving every problem in your story with time turners is that it becomes incredibly conspicuous when you don’t solve a problem with time turners.

In this chapter, the bit of canon from book 1 with the troll in the dungeon is introduced- someone comes running into the dining hall yelling troll. Luckily, Quirrell has the students well prepared:

Without any hesitation, the Defense Professor swung smoothly on the Gryffindor table and clapped his hands with a sound like a floor cracking through. "Michelle Morgan of House Gryffindor, second in command of Pinnini’s Army," the Defense Professor said calmly into the resulting quiet. "Please advise your Head of House." Michelle Morgan climbed up onto her bench and spoke, the tiny witch sounding far more confident than Minerva remembered her being at the start of the year. “Students walking through the hallways would be spread out and impossible to defend. All students are to remain in the Great Hall and form a cluster in the center… not surrounded by tables, a troll would jump right over tables… with the perimeter defended by seventh-year students. From the armies only, no matter how good they are at duelling, so they don’t get in each other’s lines of fire.”

So everyone will be safe from troll, but WAIT- Hariezer realizes Hermione is missing. What does he do? Does he commit himself to time turning himself a message telling him where Hermione is (to be fair, the time is noon, and the earliest he can reach with a time turner is 3pm. However he knows of another student who uses a time turner and is willing to send messages with it, from the post Azkaban escape. He also knows other powerful wizards use time turners, so he could ask one of them to pass the message,etc).

I suspect we are approaching an important plot moment that time turnering would somehow break. Maybe we finally get a Quirrell reveal? Anywho, it’s jarring to not see Hariezer go immediately for the time turner. Instead he tries to enlist the aid of other students (and not ask if anyone has a time turner).

Anyway, Hariezer decides they need to go look for her as fast as possible- but then

The witch he’d named turned from where she’d been staring steadily out at the perimeter, her expression aghast for the one second before her face closed up. "The Deputy Headmistress ordered us all to stay here, Mr. Potter." It took an effort for Harry to unclench his teeth. “Professor Quirrell didn’t say that and neither did you. Professor McGonagall isn’t a tactician, she didn’t think to check if we had missing students and she thought it was a good idea to start marching students through the hallways. But Professor McGonagall understands after her mistakes are pointed out to her, you saw how she listened to you and Professor Quirrell, and I’m certain that she wouldn’t want us to just ignore the fact that Hermione Granger is out there, alone -“

So Hariezer flags this as

Harry’s brain flagged this as I’m talking to NPCs again and he spun on his heel and dashed back for the broomstick.

Yes, Hariezer, in this world you are talking to NPCs- characters Yudkowsky wrote in, entirely to be stupid so that you can appear brilliant.

Anyway, he rushes off the Weasley twins to go find Hermione, and just as he finds her the chapter ends. I look forward to tuning in next time for the thrilling conclusion.

HPMOR 89: grimdark

There will be spoilers ahead. Although if you cared about spoilers why are you reading this?

So I thought the plot moment we were leading up to was a Quirrell reveal and I was dead wrong (a pun, because Hermione dies). By the time Hariezer arrives, Hermione has already been troll smashed (should have used the time turner,bro).

A brief battle scene ensues in which the Weasleys fail to be very effective, andHariezer kills the troll by floating his father’s rock (which he has been wearing in a ring) into the trolls mouth and then letting it go back to its original size, which pops the troll head.

Hermione then utters her final words “not your fault” and then dies. Hariezer is obviously upset by this.

Not a bad chapter really, even though it required a sort “rationality failure” involving the time turners to get here. Normally I wouldn’t care about this sort of thing, but the fact that basically every problem thus far was solved with time turners makes it very hard to suspend my disbelief here. It feels a touch to much like characters are doing things just to make the plot happen (and not following their ‘natural’ actions).

I fear the next ten chapters will be just reflections on this (instead of things happening).

HPMOR 90: Hariezer's lack of self reflection

Brief note- it’s mardi gras, and I’m about as over served as I ever have been. I LIKE HOW OVER SERVED AS A PHRASE BLAMES THE BARTENDER AND NOT ME. THIS IS A THEME FOR THIS CHAPTER. Anyway, hopefullly this will not lack my usual (non) eloquence.

This chapter begins what appears to be a 9 part section on Hariezer trying to cope with the death of his friend.

As the chapter opens, Hariezer cools Hermione’s body to try to preserve it. I guess that will slow decay, but probably not by enough to matter.

And then Hariezer gets understandably mopey. Everyone is concerned he is withdrawing from the world, so Mcgonagall goes to talk to him and we get this bit:

"Nothing I could have done? " Harry’s voice rose on the last word. "Nothing I could have…Or if I’d just approached the whole problem from a different angle - if I’d looked for a student with a Time-Turner to send a message back in time..

It’s the one in bold that is especially troubling because the time turner is seriously what Hariezer always turns to (TURNS TO! GET IT! IT’S AN AWFUL PUN). When your character is defined by his munchkining ability to solve problems via time turner, and the one time he doesn’t go for the time turner a major plot point happens, it’s jarring to the reader. Almost as if characters are behaving entirely to make the plot happen…

Anyway,

She was aware now that tears were sliding down her cheeks, again. “Harry - Harry, you have to believe that this isn’t your fault!” "Of course it’s my fault. There’s no one else here who could be responsible for anything." "No! You-Know-Who killed Hermione!" She was hardly aware of what she was saying, that she hadn’t screened the room against who might be listening. "Not you! No matter what else you could’ve done, it’s not you who killed her, it was Voldemort! If you can’t believe that you’ll go mad, Harry!" "That’s not how responsibility works, Professor." Harry’s voice was patient, like he was explaining things to a child who was certain not to understand. He wasn’t looking at her anymore, just staring off at the wall to her right side. "When you do a fault analysis, there’s no point in assigning fault to a part of the system you can’t change afterward

So keep this in mind- Hariezer says it’t no use blaming anyone but himself, because he can’t change their actions. This seems like a silly NPC/PC distinction- no one can change their past actions, but everyone can learn how they could have improved things.

"All right, then," Harry said in a monotone. "I tried to do the sensible thing, when I saw Hermione was missing and that none of the Professors knew. I asked for a seventh-year student to go with me on a broomstick and protect me while we looked for Hermione. I asked for help. I begged for help. And nobody helped me. Because you gave everyone an absolute order to stay in one place or they’d be expelled, no excuses…. So when something you didn’t foresee happened and it would’ve made perfect sense to send out a seventh-year student on a fast broom to look for Hermione Granger, the students knew you wouldn’t understand or forgive. They weren’t afraid of the troll, they were afraid of you. The discipline, the conformity, the cowardice that you instilled in them delayed me just long enough for Hermione to die. Not that I should’ve tried asking for help from normal people, of course, and I will change and be less stupid next time. But if I were dumb enough to allocate responsibility to someone who isn’t me, that’s what I’d say."

What exactly does Hariezer think she should have said here? If a fire had broken out in the meal hall does Hariezer think that everyone would have stayed in the cafeteria and burned to death out of fear of Mcgonagall? Also, it certainly sounds as if Hariezer has plenty of blame for people not himself. ”I only blame me, but also you suck in the following ways…”

But normal people don’t choose on the basis of consequences, they just play roles. There’s a picture in your head of a stern disciplinarian and you do whatever that picture would do, whether or not it makes any sense….People like you aren’t responsible for anything, people like me are, and when we fail there’s no one else to blame.” I AM THE ONLY PC, YOU ARE ALL NPC. I AM THE ONLY FULL HUMAN. TREMBLE BEFORE MY AGENTYNESS. I get that Harizer is mourning, but is their any more condescending way to mourn? ”Everything is my fault because you aren’t all even fully human?” You are a fucking twerp Hariezer, even when you mourn.

His hand darted beneath his robes, brought forth the golden sphere that was the Ministry-issued protective shell of his Time Turner. He spoke in a dead, level voice without any emphasis. “This could’ve saved Hermione, if I’d been able to use it. But you thought it was your role to shut me down and get in my way. No, Hariezer, you were told THERE WERE RULES and you violated them. You yourself have said that time travel can be dangerous and you were using it because Snape asked questions you didn’t know the answer to, and really to solve any trivial problem. You broke the rules, and it locked your time turner down when you might have really wanted it. Total boy-who-cried-wolf situation, and yet its conspicuously absent from your discussion above- you blame yourself in lots of ways, but not in this way.

Unable to speak, she brought forth her wand and did so, releasing the time-keyed enchantment she’d laced into the shell’s lock.

The only lessons learned from this are other character “updating towards” the idea that Hariezer Yudotter is always right. And he fails when other people have prevented his natural PC based awesomenes.

Anyway, Mcgonagall sends in the big guns (Quirrell) to try to talk to Hariezer, which leads Hariezer to say to him:

The boy’s voice was razor-sharp. “I’d forgotten there was someone else in Hogwarts who could be responsible for things.”

And later in the conversation:

"You did want to save her. You wanted it so strongly that you made some sort of actual effort. I suppose your mind, if not theirs, would be capable of that."

So you see- it’s clearly not about assigning himself all the blame (because he can only change his own actions), it’s about separating the world into ‘real people’ and ‘NPCs’ Only real people can get any blame for anything, everyone else is just window dressing. Maybe it’s a pet peeve, but I react in abhorrence to this “you aren’t even human enough to share some blame” schtick.

8 more chapters in this fucking section.

HPMOR 91

Total retread of the last chapter. Hariezer is still blaming himself, Snape tries to talk to him. They bring his parents in to try to talk to him. Nothing here really.

HPMOR 92

Really, still nothing here. Quirrell is also concerned about Hariezer, but as before his concern seems less than totally genuine. I fear this arc is basically just a lot of retreads.

HPMOR 93

Still very little going on in these chapters…

So Mcgonagall completes the transformation she began two chapters ago, and realizes rules are for suckers and Hariezer is always right

"I am ashamed," said Minerva McGonagall, "of the events of this day. I am ashamed that there were only two of you. Ashamed of what I have done to Gryffindor. Of all the Houses, it should have been Gryffindor to help when Hermione Granger was in need, when Harry Potter called for the brave to aid him. It was true, a seventh-year could have held back a mountain troll while searching for Miss Granger. And you should have believed that the Head of House Gryffindor," her voice broke, "would have believed in you. If you disobeyed her to do what was right, in events she had not foreseen. And the reason you did not believe this, is that I have never shown it to you. I did not believe in you. I did not believe in the virtues of Gryffindor itself. I tried to stamp out your defiance, instead of training your courage to wisdom.

Maybe I’m projecting too much of canon McGonagall onto my reading of this one in this fanfic, but has she really been stamping out all defiance and overly stern? Would any student really have believed they would have expelled for trying to help find a missing student in a dire situation?

Hariezer certainly wasn’t expelled (or punished in anyway) for his experimenting with transfiguration/discovering partial transfiguration. He was punished for flaunting his time turner despite explicit instructions not to… But in a school for magic users, that is probably a necessity.

Also, Hermione’s body has gone missing. I suspect Hariezer is cryonicsing it.

HPMOR 94

This is the best chapter of this “reflect on what just happened” giant block of chapters, but that’s not saying much.

Hariezer might not have taken Hermione’s body, but seems unconcerned that it’s missing (maybe he took it to freeze the brain, maybe Voldie took it to resurrect Hermione or brain upload her or something). That’s the only real thing of merit that happens in this chapter (a conversation between Dumbledore and Hariezer, a conversation between Neville and Hariezer).

Hariezer has finally convinced himself that Voldemort is smart, which leads to this rumination

Okay, serious question. If the enemy is that smart, why the heck am I still alive? Is it seriously that hard to poison someone, are there Charms and Potions and bezoars which can cure me of literally anything that could be slipped into my breakfast? Would the wards record it, trace the magic of the murderer? Could my scar contain the fragment of soul that’s keeping the Dark Lord anchored to the world, so he doesn’t want to kill me? Instead he’s trying to drive off all my friends to weaken my spirit so he can take over my body? It’d explain the Parselmouth thing. The Sorting Hat might not be able to detect a lich-phylactery-thingy. Obvious problem 1, the Dark Lord is supposed to have made his lich-phylactery-thingy in 1943 by killing whatshername and framing Mr. Hagrid. Obvious problem 2, there’s no such thing as souls.

So, all the readers are already on board this train, because they’ve read the canon novel, so I guess it’s nice that the “super rationalist” is considering it (although Voldemort is smart, therefore I have a Voldemort fragment trying to possess me is a huge leap. You didn’t even Bayes that shit bro).

But seriously, “there’s no such thing as souls?” SO DON’T CALL IT A SOUL, CALL IT A MAGIC RESURRECTION FRAGMENT. Are we really getting hung up on semantics?

These chapter are intensely frustrating because any “rising action” in this story (we are nearing the conclusion after all) is blunted because after anything happens, we need 10 chapters for everyone to talk about everything and digest the events. The ratio of words/plot is ridiculously huge.

We do maybe get a bit of self-reflection when Neville tries to blame himself for Hermione’s death:

"Wow," the empty air finally said. "Wow. That puts a pretty different perspective on things, I have to say. I’m going to remember this the next time I feel an impulse to blame myself for something. Neville, the term in the literature for this is ‘egocentric bias’, it means that you experience everything about your own life but you don’t get to experience everything else that happens in the world. There was way, way more going on than you running in front of me. You’re going to spend weeks remembering that thing you did there for six seconds, I can tell, but nobody else is going to bother thinking about it. Other people spend a lot less time thinking about your past mistakes than you do, just because you’re not the center of their worlds. I guarantee to you that nobody except you has even consideredblaming Neville Longbottom for what happened to Hermione. Not for a fraction of a second. You are being, if you will pardon the phrase, a silly-dilly. Now shut up and say goodbye."

It would be nice for Hariezer to more explicitly use this to come to terms with his own grieving (instead of insisting on “heroic responsibility” for himself a few sections back, and also insisting it’s McGonagall’s fault for trying to enforce rules, and now insisting that blaming yourself is egocentric bias). I hope this is Hariezer realizing that he shouldn’t blame himself, and growing a bit, but fear this is Hariezer suggesting that Neville isn’t important enough to blame.

Anyway, Hariezer insists that Neville leave for awhile to help keep him safe.

HPMOR 95

So the chapter opens with more incuriousness, which is the rest of the chapter in miniature:

Harry had set the alarm upon his mechanical watch to tell him when it was lunchtime, since he couldn’t actually look at his wrist, being invisible and all that. It raised the question of how his eyeglasses worked while he was wearing the Cloak. For that matter the Law of the Excluded Middle seemed to imply that either the rhodopsin complexes in his retina were absorbing photons and transducing them to neural spikes, or alternatively, those photons were going straight through his body and out the other side, but not both. It really did seem increasingly likely that invisibility cloaks let you see outward while being invisible yourself because, on some fundamental level, that was how the caster had - not wanted - but implicitly believed - that invisibility should work.

This would be an excellent fucking question to explore, maybe via some experiments. But no. I’ve totally given up on this story exploring the magic world in any detail at all. Anyway, Hariezer skips straight from “I wonder how this works” to “it must work this way, how could we exploit it”

Whereupon you had to wonder whether anyone had tried Confunding or Legilimizing someone into implicitly and matter-of-factly believing that Fixus Everythingus ought to be an easy first-year Charm, and then trying to invent it. Or maybe find a worthy Muggleborn in a country that didn’t identify Muggleborn children, and tell them some extensive lies, fake up a surrounding story and corresponding evidence, so that, from the very beginning, they’d have a different idea of what magic could do.

This skips all the interesting hard work of science.

The majority of the chapter is a long discussion between Quirrell and Hariezer where Quirrell tries to convince Hariezer not to try to raise the dead. It’s too dangerous, may end the universe, etc.

Lots of discussion about how special Quirrell and Hariezer are because only they would even think to fight death,etc. It’s all a boring retread of ideas already explored in earlier chapters,etc.

It reads a lot like any discussion of cryonics with a cryonics true believer:

The Defense Professor’s voice was also rising. “The Transfiguration Professor is reading from a script, Mr. Potter! That script calls for her to mourn and grieve, that all may know how much she cared. Ordinary people react poorly if you suggest that they go off-script. As you already knew!”

Also, it’s sloppy world building- do we really think no wizards in the HPMOR universe have spent time investigating death/spells to reverse aging/spells to deal with head injuries,etc?

THERE IS A RESURRECTION STONE AND A LITERAL GATEWAY TO THE AFTERLIFE IN THE BASEMENT OF THE MINISTRY OF MAGIC. Maybe Hariezer’s FIRST FUCKING STOP if he wanted to investigate bringing back the dead SHOULD BE THAT GATE. Maybe some scientific experiments?

It’s like the above incuriousness with the invisibility cloak (and the typical transhumanist approach to science)- assume all the problems are solved and imagine what the world be like, how dangerous that power might be. This is no way to explore a question. It’s not even producing a very interesting story.

Quirrell assumes Hariezer might end the world even though he has shown 0 aptitude with any magic even approaching dangerous…

HPMOR 96: more of the same

Remus takes Hariezer to Godric’s Hollow to try to cheer him up or whatever.

Hariezer discovers the Potter’s family motto is apparently the passage from Corinthians:

The Last Enemy to Be Destroyed is Death

Hariezer is super glad that his family has a long history of trying to end death, and (at least) realizes that other wizards have tried. Of course, the idea of actually looking at their research doesn’t fucking occur to him because this story is very silly.

We get this rumination from Hariezer on the Peverell’s ‘deathly hallows’ from the books:

Hiding from Death’s shadow is not defeating Death itself. The Resurrection Stone couldn’t really bring anyone back. The Elder Wand couldn’t protect you from old age.

HOW THE FUCK DO YOU KNOW THE RESURRECTION STONE CAN’T BRING ANYONE BACK? HAVE YOU EVEN SEEN IT?

Step 1- assume that the resurrection stone doesn’t work because you can’t magically bring back the dead

Step 2- decide you want to magically resurrect the dead

Step 3- never revisit step 1.

SCIENCE!

GO INVESTIGATE THE DOORWAY TO THE AFTERLIFE! GO TALK TO PEOPLE ABOUT THE RESURRECTION STONE! DO SOME FUCKING RESEARCH! ”I’m going to resurrect the dead by thinking really hard about how much death sucks and doing nothing else.”

HPMOR 97: plot points resolved arbitrarily

Next on the list to talk with Hariezer regarding Hermione’s death? The Malfoys who call Hariezer to Gringotts under the pretense of talking about Hariezer’s debt.

On the way in he passes a goblin, which prompts this

If I understand human nature correctly - and if I’m right that all the humanoid magical species are genetically human plus a heritable magical effect -

How did you come to that conclusion Hariezer? What did you do to study it? Did you just make it up with no justification whatsoever? This story confuses science jargon for science.

Anyway, Lucious is worried that he’ll be blamed for Hermione’s death (although given that it has already been established that the wizard court votes exactly as he wants it to I’m not sure why he is worried about it) so he agrees to cancel Hariezer’s debt and return all his money if Hariezer swears Lucious didn’t have anything to do with the troll that killed Hermione.

This makes very little sense- why would anyone listen to Hariezer on this? Hariezer doesn’t actually know that the Malfoys weren’t involved. If he is asked “how do you know?” he’ll have to say “I don’t.” If he Bayesed that shit, the Malfoys should be near the fucking top of the suspect list…

Anyway, the Malfoys try to convince Hariezer that Dumbledore killed Hermione as some sort of multi-level plot.

I’m so bored.

HPMOR 98: this block is nearly over!

The agreement put in place in the previous chapter is enacted.

Malfoy announces to Hogwarts that Hermione was innocent. Hariezer says there is no ill will between the Potters and the Malfoys. Why did we even need this fucking scene?

Through backstage maneuvering by Hariezer and Malfoy,the Hogwarts board of governors enacts some rules for safety of students (travel in packs, work together,etc). Why they needed the maneuvering I don’t know (just ask McGonagall to implement whatever rules you want. No effort required).

Also, Neville was sent away from Hogwarts like.. three chapters ago. But now he is in Hogwarts and stands up to read some of the rules? And Draco, who was closer to Hariezer, returns to Hogwarts? This makes no sense given Hariezer’s fear for his friends? ”No one is safe! Wait, I changed my mind even though nothing has happened.”

There was also a surreal moment where the second worst thing I’ve ever read referenced the first:

"Remind me to buy you a copy of the Muggle novel Atlas Shrugged,"

HPMOR 99

This chapter is literally one sentence long. Unicorn died at Hogwarts. Why not just slap it into the previous chapter?

HPMOR 100

Remember that mysterious bit about the unicorns dying? That merited a whole one-sentence chapter? Luckily, it’s totally resolved in this chapter.

Borrowing a scene from canon, we have Draco and some slytherin palls (working to fix the school) investigating the forest with Hagrid as part of a detention. This leads to a variant of an old game theory/cs joke:

Meself,” Hagrid continued, “I think we might ‘ave a Parisian hydra on our ‘ands. They’re no threat to a wizard, yeh’ve just got to keep holdin’ ‘em off long enough, and there’s no way yeh can lose. I mean literally no way yeh can lose so long’s yeh keep fightin’. Trouble is, against a Parisian hydra, most creatures give up long before. Takes a while to cut down all the heads, yeh see.” "Bah," said the foreign boy. "In Durmstrang we learn to fight Buchholz hydra. Unimaginably more tedious to fight! I mean literally, cannot imagine. First-years not believe us when we tell them winning is possible! Instructor must give second order, iterate until they comprehend."

This time, it’s just Draco and friends in detention, no Hariezer/

When Draco encounters the unicorn killer, all of a sudden Hariezer and aurors come riding in to save the day:

After Captain Brodski had learned that Draco Malfoy was in the Forbidden Forest, seemingly in the company of Rubeus Hagrid, Brodski had begun inquiring to find out who had authorized this, and had still been unable to find out when Draco Malfoy had missed check-in. Despite Harry’s protests, the Auror Captain, who was authorized to know about Time-Turners, had refused to allow deployment to before the time of the missed check-in; there were standard procedures involving Time. But Brodski had given Harry written orders allowing him to go back and deploy an Auror trio to arrive one second after the missed check-in time.

So… why does Hariezer come with the aurors? For what purpose? He is always talking about avoiding danger,etc so why ride into danger when the battle wizards will probably be enough?

Anyway, we all know its Quirrell killing unicorns, so I’ll skip to the Hariezer/Quirrell interaction:

The use of unicorn’s blood is too well-known.” "I don’t know it," Harry said. "I know you do not," the Defense Professor said sharply. "Or you would not be pestering me about it. The power of unicorn’s blood is to preserve your life for a time, even if you are on the very verge of death."

And then

"And why -" Harry’s breath hitched again. "Why isn’t unicorn’s blood standard in healer’s kits, then? To keep someone alive, even if they’re on the very verge of dying from their legs being eaten?" "Because there are permanent side effects," Professor Quirrell said quietly. "Side effects? Side effects? What kind of side effect is medically worse than DEATH? " Harry’s voice rose on the last word until he was shouting. "Not everyone thinks the same way we do, Mr. Potter. Though, to be fair, the blood must come from a live unicorn and the unicorn must die in the drinking. Would I be here otherwise?" Harry turned, stared at the surrounding trees. “Have a herd of unicorns at St. Mungos. Floo the patients there, or use portkeys.” "Yes, that would work."

So do you remember a few chapters back when Hariezer was worried about eating plants or animals that might be conscious (after he learned snake speech)?

He knows literally nothing about unicorns here, nothing about what the side effects are,etc. I know lots of doctors who have living wills because they aren’t ok with the side effects of certain life-preserving treatments.

This feels again like canon is fighting the transhumanist message the author wants to insert.

HPMOR 101

Still in the woods, Hariezer encounters a centaur who tries to kill him, because he divines that Hariezer is going to make all the stars die.

There are some standard anti-astrology arguments, which again seems to be fighting the actual situation because the centaurs successfully use astrology to divine things.

We get this:

"Cometary orbits are also set thousands of years in advance so they shouldn’t correlate much to current events. And the light of the stars takes years to travel from the stars to Earth, and the stars don’t move much at all, not visibly. So the obvious hypothesis is that centaurs have a native magical talent for Divination which you just, well, project onto the night sky."

There are so, so many other hypothesis Hariezer. Maybe starlight has a magical component that waxes and wanes as stars align into different magical symbols are some such. The HPMOR scientific method:

observation -> generate 1 hypothesis -> assume you are right -> it turns out that you are right.

Quirrell saves Hariezer and I guess in the aftermath Filch and Hagrid both get sacked (we aren’t actually shown this, instead Dumbledore and Hariezer have a discussion about it, because why show when you can have characters talk about! So much more interesting!)

Anyway, Dumbeldore is a bit sad by the loss of Filch and especially Hagrid, but Hariezer says

"Your mistake," Harry said, looking down at his knees, feeling at least ten percent as exhausted as he’d ever been, "is a cognitive bias we would call, in the trade, scope insensitivity. Failure to multiply. You’re thinking about how happy Mr. Hagrid would be when he heard the news. Consider the next ten years and a thousand students taking Magical Creatures and ten percent of them being scalded by Ashwinders. No one student would be hurt as much as Mr. Hagrid would be happy, but there’d be a hundred students being hurt and only one happy teacher."

First “in the trade”? Really?

Anyway, Hariezer isn’t multiplying in the obvious tangible benefits of an enthusiastic teacher who really knows his shit regarding magical creatures. Yes, more students will be scalded but its because there will be SUPER AWESOME LESSONS WHERE KIDS COULD BE SCALDED!

In the balance, I think Hariezer was right about Filch and Dumbledore was right about Hagrid.

Anyway, thats it for this chapter, its a standard “chapter where people do nothing that talk.”

Harry Potter and the Methods of Expository Dialogue.

HPMOR 102: open borders and death spells

Quirrell is still dying, Hariezer brings him a unicorn he turned into a stone.

We learn how horcruxes work in this world:

Only one who doess not believe in common liess will reasson further, ssee beneath obsscuration, realisse how to casst sspell. Required murder iss not ssacrificial ritual at all. Ssudden death ssometimes makess ghosst, if magic burssts and imprintss on nearby thing. Horcrux sspell channelss death-bursst through casster, createss your own ghosst insstead of victim’ss, imprintss ghosst in sspecial device. Ssecond victim pickss up horcrux device, device imprintss your memoriess into them. But only memoriess from time horcrux device wass made. You ssee flaw?”

Wait? A ghost has all the memories of the person who died? Why isn’t Hariezer reading everything he can about how these imprints work? If the Horcrux can transfer ghost-like stuff into a person, could you return any ghost to a new body? I feel like Hariezer just says “I’m going to end death! Humanity should end death! I can’t believe no one is trying to end death!” But he isn’t actually doing anything about it himself.

Also, if that is how a horcrux works WHY THE FUCK WOULD VOLDEMORT PUT ONE ON A PIONEER PROBE? The odds of that encountering people again are pretty much nill. At least we’ve learned horcruxes aren’t conscious- I had assumed Voldemort had condemned one of his copies to an eternity of isolation.

We also learn that in HPMOR world

There is a second level to the Killing Curse. Harry’s brain had solved the riddle instantly, in the moment of first hearing it; as though the knowledge had always been inside him, waiting to make itself known. Harry had read once, somewhere, that the opposite of happiness wasn’t sadness, but boredom; and the author had gone on to say that to find happiness in life you asked yourself not what would make you happy, but what would excite you. And by the same reasoning, hatred wasn’t the true opposite of love. Even hatred was a kind of respect that you could give to someone’s existence. If you cared about someone enough to prefer their dying to their living, it meant you were thinking about them. It had come up much earlier, before the Trial, in conversation with Hermione; when she’d said something about magical Britain being Prejudiced, with considerable and recent justification. And Harry had thought - but not said - that at least she’d been let into Hogwarts to be spat upon. Not like certain people living in certain countries, who were, it was said, as human as anyone else; who were said to be sapient beings, worth more than any mere unicorn. But who nonetheless wouldn’t be allowed to live in Muggle Britain. On that score, at least, no Muggle had the right to look a wizard in the eye. Magical Britain might discriminate against Muggleborns, but at least it allowed them inside so they could be spat upon in person. What is deadlier than hate, and flows without limit? "Indifference," Harry whispered aloud, the secret of a spell he would never be able to cast; and kept striding toward the library to read anything he could find, anything at all, about the Philosopher’s Stone.

So standard open borders stuff, not worth spending time with.

But I want to talk about the magic here- apparently you can only cast the killing curse at people you hate, and toward people you are indifferent toward. So you can’t kill your loved ones! Big limitation!

Also, Hariezer “99% of the fucking planet is NPCs” Yudotter isn’t indifferent to anyone? I call BS.

HPMOR 103: very punny

Credit where credit is due, this whole chapter sets up a pretty clever pun.

The students take an exam, and then receive their final “battle magic” grades. Hermione is failed because she made the mistake of dying. Hariezer gets an exceeds expectations, which Quirrell informs Hariezer “It is the same grade… that I received in my own first year.”

Get it? He marked him as an equal.

HPMOR 104: plot threads hastily tied up/also some nonsense

So this chapter opens with a quidditch game, in an attempt to wrap up an earlier plot thread- Quirrell’s reward for his battle game (a reward given out back in chapter 34 or so, and literally never mentioned again until this chapter) was that slytherin and ravenclaw would tie for the house cup and Hogwarts would stop playing quidditch with the snitch.

Going into this game, Hufflepuff is in the lead for house cup “by something like five hundred points.” Quirrell is out of commission with his sickness, but the students have taken matters into their own hands- it appears the plan is just to not catch the snitch?

It was at eight pm and six minutes, according to Harry’s watch, when Slytherin had just scored another 10 points bringing the score to 170-140, when Cedric Diggory leapt out of his seat and shouted “Those bastards!” "Yeah!" cried a young boy beside him, leaping to his own feet. "Who do they think they are, scoring points?" "Not that!" cried Cedric Diggory. "They’re - they’re trying to steal the Cup from us! " "But we’re not in the running any more for -" "Not the Quidditch Cup! The House Cup!"

What? It’s totally unclear to me how this is supposed to work. In the books, as I remember it, points were awarded for winning quidditch games NOT for simply scoring points within a quidditch game? Winning 500 to 500 will just result in some fixed amount of points going to the winner.

Also, there appears to be a misunderstanding of quidditch:

The game had started at six o’ clock in the afternoon. A typical game would have gone until seven or so, at which point it would have been time for dinner. No, as I recall, games go on for days not one hour. I think the books mention a game lasting longer than a month. No one would be upset at a game where the snitch hasn’t been caught in a few hours.

Basically, this whole thing feels really ill-conceived.

Luckily, the chapter pivots away from the quidditch game pretty quickly, Hariezer gets a letter from himself.

Beware the constellation, and help the watcher of stars and by the wise and the well-meaning. in the place that is prohibited and bloody stupid. Pass unseen by the life-eaters’ confederates, Six, and seven in a square,

I note that Hariezer established way back when somewhere that he has a system in place to communicate with himself, with secret codes for his notes to make sure they really are for him. I’m too lazy to dig this back up, but I definitely remember reading it. Probably in chapter 13 with the time travel game?

Anyway, apparently Hariezer has forgotten this (I hope this comes up and it’s not just a weird problem introduced for no reason?) because this turns out to be a decoy note from Quirrell to lure him to the forbidden corridor. After a whole bunch of people all show up at the forbidden corridor at the same time, and some chaos breaks out, Hariezer and Quirrell are the last men standing, which leads to this:

An awful intuition had come over Harry, something separate from all the reasoning he’d done so far, an intuition that Harry couldn’t put into words; except that he and the Defense Professor were very much alike in certain ways, and faking a Time-Turned message was just the sort of creative method that Harry himself might have tried to bypass all of a target’s protections - … And Professor Quirrell had known a password that Bellatrix Black had thought identified the Dark Lord and his presence gave the Boy-Who-Lived a sense of doom and his magic interacted destructively with Harry’s and his favorite spell was Avada Kedavra and and and … Harry’s mouth was dry, even his lips were trembling with adrenaline, but he managed to speak. “Hello, Lord Voldemort.” Professor Quirrell inclined his head in acknowledgement, and said, “Hello, Tom Riddle.”

We also indirectly find out that Quirrell killed Hermione (but we already knew that), although he did it by controlling professor Sprout (I guess to throw off the scent if he got caught?)

Anyway, this pivotal plot moment seems to rely entirely on the fact that Hariezer forgot his own coded note system?

HPMOR 105

So Quirrell gets Hariezer to cooperate with him, by threatening students, and offering to resurrect Hermione if he gets the philosopher’s stone

And know this, I have taken hostages. I have already set in motion a spell that will kill hundreds of Hogwarts students, including many you called friends. I can stop that spell using the Stone, if I obtain it successfully. If I am interrupted before then, or if I choose not to stop the spell, hundreds of students will die.

Hariezer does manage to extract a concession:

Agreed,” hissed Professor Quirrell. “Help me, and you sshall have ansswerss to your quesstions, sso long ass they are about passt eventss, and not my planss for the future. I do not intend to raisse my hand or magic againsst you in future, sso long ass you do not raisse your hand or magic againsst me. Sshall kill none within sschool groundss for a week, unlesss I musst. Now promisse that you will not attempt to warn againsst me or esscape. Promisse to put forth your own besst efforts toward helping me to obtain the Sstone. And your girl-child friend sshall be revived by me, to true life and health; nor sshall me or mine ever sseek to harm her.” A twisted smile. “Promisse, boy, and the bargain will be sstruck.”

So coming up we’ll get one of those chapters where the villain explains everything. Always a good sign when the villain does apparently nothing for 90 or so out of 100 chapters, and then explains the significance of everything at the very end.

HPMOR 106

Not much happens here, Quirrell kills the 3 headed cerberus to get past the first puzzle. When Hariezer points out that might have alerted someone, Quirrell is all “eh, I fucked all the wards up.”

So I guess more time to go before we get the villain monologue chapter.

HPMOR 107

Still no villain monologue. Quirrell and Hariezer encounter the other puzzles from the book, and Quirrell blasts them to death with magic fire rather than actually solve them.

However, Quirrell has some random reasons to not blast apart the potion room (he respects Snape or something, blah,bah). Anyway, apparently this means he’ll have to make a long and complicated potion, which will give Quirrell and Hariezer some time to talk.

Side note: credit where credit is due, I again notice these chapters flow much better, and have a much smoother writing style. There is some wit in the way that Quirrell just hulk-smashes all the puzzles (although stopping at Snape;s puzzle seems like a contrived way to drop the monologue we know is coming next chapter or so into the story) When things are happening, HPMOR can be decently written.

HPMOR 108: monologue

So we get the big “explain everything” monologue, and it’s kind of a let down?

The first secret we get- Hariezer is indeed a copy of Voldemort (which was just resolving some dramatic irony, we all knew this because we read the books). In a slight twist, we find out that he was intentionally a horcrux-

It occurred to me how I might fulfill the Prophecy my own way, to my own benefit. I would mark the baby as my equal by casting the old horcrux spell in such fashion as to imprint my own spirit onto the baby’s blank slate… I would arrange with the other Tom Riddle that he should appear to vanquish me, and he would rule over the Britain he had saved. We would play the game against each other forever, keeping our lives interesting amid a world of fools.

But apparently creating the horcrux created some sort of magic resonance and killed his body. But he had somehow made true-immortal horcruxes. Unfortunately, he had put them stupid places like on the pioneer probe or in volcanos where people would never touch them, so he never managed to find a host (remember when I complained about that a few chapters back?)

Hariezer does point out that Voldemort should have tested the new horcrux spell. He suggests Voldie failed to do so because he doesn’t think about doing nice things, but Voldie could have just horcruxed someone, killed them to test it, then destroyed the horcrux, then killed them for real. Not nice, pretty straightforward. It feels like this is going to be Voldemort’s weakness that gets exploited.

We find out that the philosopher’s stone makes transfigurations permanent, which I guess is a minor twist on the traditional legend? Really, just a specific way of making it work- in the legends it can transmute metals, heal sickness, bring dead plants back to life, let you make homunculi,etc.

In HPMOR, powerful magical artifacts can’t have been produced recently, because lore is lost of whatever, so we get a grimdark history of the stone, involving a Hogwarts student seducing professor Baba Yaga to trick her into taking her virginity so she could steal the stone. Really incidental to anything. Anyway, Flamel, who stole the stone, is both man and woman and uses the stone to transmute back and forth, and apparently gave Dumbledore power to fight Grindlewald.

Quirrell killed Hermione (duh) because

I killed Miss Granger to improve your position relative to that of Lucius Malfoy, since my plans did not call for him to have so much leverage over you.

I don’t think this actually makes much sense at all? It’s pretty clear Voldie plans to kill Hariezer as soon as this is over, so why should he care about Malfoy at all in this? I had admittedly assume he killed Hermione to help his dark side take over Hariezer or something.

Apparently they raided Azkaban to find out where Black had hidden Quirrell’s wand.

Also, as expected, Voldemort was both Monroe and Voldemort and was playing both sides in order to gain political power. He wanted to get political power because he was afraid muggles would destroy the world.

Basically, every single reveal is basically what you’d expect from the books. Harry Potter and The Obvious Villain Monologue.

The only open question is why the Hariezer-crux, given how that spell is supposed to work, didn’t have any of Voldemort’s memories up until that time? I expect we are supposed to chalk it up to “the spell didn’t quite work because of the resonance that blew everything up” or whatever.

HPMOR 109: supreme author wank

We get to the final mirror, and we get this bit of author wank:

Upon a wall of metal in a place where no one had come for centuries, I found written the claim that some Atlanteans foresaw their world’s end, and sought to forge a device of great power to avert the inevitable catastrophe. If that device had been completed, the story claimed, it would have become an absolutely stable existence that could withstand the channeling of unlimited magic in order to grant wishes. And also - this was said to be the vastly harder task - the device would somehow avert the inevitable catastrophes any sane person would expect to follow from that premise. The aspect I found interesting was that, according to the tale writ upon those metal plates, the rest of Atlantis ignored this project and went upon their ways. It was sometimes praised as a noble public endeavor, but nearly all other Atlanteans found more important things to do on any given day than help. Even the Atlantean nobles ignored the prospect of somebody other than themselves obtaining unchallengeable power, which a less experienced cynic might expect to catch their attention. With relatively little support, the tiny handful of would-be makers of this device labored under working conditions that were not so much dramatically arduous, as pointlessly annoying. Eventually time ran out and Atlantis was destroyed with the device still far from complete. I recognise certain echoes of my own experience that one does not usually see invented in mere tales.”

Get it? It’s friendly AI and we are all living in Atlantis! And Yud is bravely toiling away in obscurity to save us all! (Note: toiling in obscurity in this context means soliciting donations to continue running one of the least productive research organizations in existence.)

Anyway, after this bit of wankery, Voldie and Hariezer return to the problem of how to get the stone. The answer turns out to be by confounding himself into thinking that he is Dumbledore wanting the stone back after Voldemort has been defeated.

I point out that the book’s original condition where the way to get the stone was to not want the stone was vastly more clever. Don’t think of elephants, and all that.

Anyway, after Voldemort gets the stone, Dumbledore shows up.

HPMOR 110

Apparently Dumbledore was going to use the mirror as a trap to banish Voldemort. But when he saw Hariezer was with him, Dumbledore sacrificed himself to save Hariezer. So now Dumbledore is banished somewhere.

So I read chapter 114

And I kind of don’t get how Hariezer’s solution worked? He turned the ground near his wand into spider silk, but how did he get it around everyone’s neck? Why did he make it spider silk before turning it into nanowire?

HPMOR 111

So in this chapter, Hariezer is stripped of wand, pouch and time turner, and Voldemort has the philosopher’s stone. It’s looking pretty bad for our hero.

Voldemort walks Hariezer to an altar he has prepared,does some magic stuff, and a new shiny body appears for him. So now he has fully resurrected.

Voldemort then brings back Hermione (Hariezer HAD stolen the remains). I note that I don’t think Hariezer has actually done anything in this entire story- his first agenda- study magic using science- was a complete bust, and then his second agenda was a big resolution to bring back Hermione, but Voldemort did it for him. Voldemort also crossed Hermione with a troll and unicorn (creating a class Trermionecorn) so she is now basically indestructible. Why didn’t Voldemort do this to his own body? No idea. Why would someone obsessed with fighting death and pissed as hell about how long it took him to reclaim his old body think to give Hermione wolverine-level indestructibility but not himself? Much like the letter Hariezer got to set this up, it’s always bad when characters have to behave out of character in order to set the plot up.

Anyway, to bring Hermione back Voldie gives Hariezer his wand back so he can hit her with his super-patronus. So he now has a wand.

Voldemort then asks Hariezer for Roger Bacon’s diary (which he turns into a Hermione horcrux), which prompts Hariezer to say

I tried translating a little at the beginning, but it was going slowly -” Actually, it had been excruciatingly slow and Harry had found other priorities.

Yep, doing experiments to discover magic, despite being the premise of the first 20ish chapters then immediately stopped being any priority at all. Luckily it shows up here as an excuse for Hariezer to get his pouch back (he retrieved it to get the diary).

While Voldemort is distracted making the horcrux, Hariezer whips a gun out of the pouch and shoots Voldemort. Something tells me it didn’t work.

HPMOR 112

Not surprisingly, the shots did nothing to Voldemort who apparently can create a wall of dirt faster than a bullet travels. Apparently it was a trick, because Voldemort had created some system where no Tom Riddle could kill any other Tom Riddle unless the first had attacked him. Somehow, Hariezer hadn’t been bound to it, and now neither of them are.

I don’t know why Yudkowsky introduced this curse and then had it broken immediately? It would have been more interesting to force Hariezer to deal with Voldemort without taking away his immortality. Actually, given all the focus on memory charms in the story, it’s pretty clear that when he achieves ultimate victory Hariezer will do it with a memory charm- turning Tom Riddle into an immortal amnesiac or implanting other memories in him (so he is an immortal guy who works in a restaurant or something).

In retaliation, Voldemort hits him with a spell that takes everything but his glasses and wand, so he is naked in a graveyard, and a bunch of death eaters teleport in (37 of them).

HPMOR 113

This is a short chapter, Hariezer is still in peril. As Voldemort’s death eaters pop in, one of them tries to turn against him but Voldemort kills him dead.

And then he forces Hariezer to swear an oath to not try to destroy the world- Voldemort’s plan is apparently to resurrect Hermione, force Hariezer to not destroy the world, and then to kill him. (I note that if he did things in a slightly different order, he’d have already won…)

So this chapter ends with Hariezer surrounded by death eaters, all with wands pointed at him, ready to kill him, and we get this (sorry, it’s a long quote).

This is your final exam. Your solution must at least allow Harry to evade immediate death, despite being naked, holding only his wand, facing 36 Death Eaters plus the fully resurrected Lord Voldemort. 12:01AM Pacific Time (8:01AM UTC) on Tuesday, March 3rd, 2015, the story will continue to Ch. 121. Everyone who might want to help Harry thinks he is at a Quidditch game. he cannot develop wordless wandless Legilimency in the next 60 seconds. the Dark Lord’s utility function cannot be changed by talking to him. the Death Eaters will fire on him immediately. if Harry cannot reach his Time-Turner without Time-Turned help - then the Time-Turner will not come into play. Harry is allowed to attain his full potential as a rationalist, now in this moment or never, regardless of his previous flaws. if you are using the word ‘rational’ correctly, is just a needlessly fancy way of saying ‘the best solution’ or ‘the solution I like’ or ‘the solution I think we should use’, and you should usually say one of the latter instead. (We only need the word ‘rational’ to talk about ways of thinking, considered apart from any particular solutions.) if you know exactly what a smart mind would do, you must be at least that smart yourself….

The issue here is that literally any solution, no matter how outlandish will work if you can design the rules to make the munchkin work.

Hariezer could use partial transfiguration to make an atomic lattice under the earth with a few atoms of anti-matter under each death eater to blow up each of them.

He could use the fact that Voldemort apparently cast broomstick spells on his legs to levitate Voldemort. And after levitating Voldemort away convince the death eaters he is even more powerful than Voldie.

He could use the fact that Tom Riddle controls the dark mark to burn all the death eaters where they stand- Voldemort can’t kill Hariezer with magic because of the resonance, and a simple shield should stop bullets.

He could partially transfigure a dead man’s switch of sorts, so that if he dies fireworks go up and the quidditch game gets disrupted- he knows it didn’t, so he knows he won’t die.

In my own solution I posted earlier, he could talk his way out using logic-ratio-judo. Lots of options other than my original posted solution here- convince them that killing Hariezer makes the prophecy come true because of science reasons,etc.

He could partially transfigure poison gas or knock out gas.

Try to come up with your own outlandish solution. The thing is, all of these work because we don’t really know the rules of HPMOR magic- we have a rough outline, but there is still a lot of author flexibility. So there is that.

HPMOR 114: solutions to the exam

Hariezer transfigures part of his wand into carbon nanotubes that he makes into a net around each death eater’s head and around voldemort’s hands.

He uses it to decapitate all the death eaters (killing Malfoy’s father almost certainly) and taking Voldemort’s hand’s off.

Voldemort then charges at him but Hariezer hits him with a stunning spell.

I note that this solution is as outlandish as any.

It WAS foreshadowed in the very first chapter, but that doesn’t make it less outlandish.

HPMOR 115-116-117

Throughout the story, whenever there is action we then have 5-10 chapters where everyone digests what happened. These chapters all fit in that vein.

Hariezer rearranges the scene of his victory to put Voldemort’s hands around Hermione’s neck and then rigs a big explosion. He then time turners back to the quidditch game (why the hell did he have time left on his time turner?)

Anyway, when he gets back to the game he does a whole “I CAN FEEL THE DARK LORD COMING” thing, and says the Hermione followed him back. I guess the reason for this plot is just to keep him out of it/explain how Hermione came back? You’d think he could have just hung around the scene of the crime, waited ‘till he was discovered and explain what happened?

Then in chapter 117, Mcgonagall explains to the school what was found- Malfoy’s father is dead, Hermione is alive, etc.

So after the big HPMOR reveal

It sort of feels like HPMOR is just the overly wordy gritty Potter reboot with some science stuff slapped on to the first 20 chapters or so.

Like, Potter is still a horcrux, Voldemort still wanted to take take over the wizard world and kill the muggles, etc.

Even the anti-death themes have fallen flat because of “show, don’t tell”- Dumbledore was a “deathist” but he was on the side of not killing all the muggles, Voledmort actually defeated death but that position rides along side the kill-everyone ethos, and Hariezer’s resolution to end death apparently was going to blow up the entire world. So the characters might argue the positions, but for reasons I don’t comprehend, actually following through is shown as a terrible idea.

I read the rest of HPMOR/113 puzzle

I read HPMOR, and will put chapter updates when I have time, but I wanted to put down my version of how Hariezer will get out of the predicament at the end of 113. I fear if I put this down after the next chapter is released, and if it’s correct, people will say I looked ahead.

Anyway, the way that this would normally be solved in HPMOR is simple the time turner- Hariezer would resolve to go and find Mcgonagall or Bones or whoever when this was all over and tell them to time turner into this location and bring the heat. Or whatever. But that has been disallowed by the rules.

But I think Yud is setting this up as a sort of “AI box” experiment, because he has his obsessions and they show up time and time again. So the solution is simply to convince Voldemort to let him go. How? In a version of Roko’s basilisk he needs to convince Voldemort that they might be in a simulation- i.e. maybe they are still looking in the mirror. Hasn’t everything gone a little too well since they first looked in? Dumbledore was vanquished, bringing back Hermione was practically easy, every little thing has been going perfectly. Maybe the mirror is just simulating what he wants to see?

So two ways to go from here- Hariezer is also looking in the mirror (and he has also gotten what he wanted, Hermione being brought back) so he might be able to test this just by wishing for something.

Or, Hariezer can convince Voldemort that the only way to know for sure is for Voldemort to fail to get something he wants, and the last thing he wants is for Hariezer to die.

HPMOR 118

The story is still in resolution mood, and I want to point out one thing that this story does right that the original books failed at- which is an actual resolution. The one big failure of the original Harry Potter books, in my mind, was that after Voldemort was defeated, we flashed immediately to the epilogue. No funeral for the departed (no chance to say goodbye to the departed Fred Weasley,etc).

Of course, in the HPMOR style, there is a huge resolution after literally every major even in the story, so it’s at least in part a stopped clock situation.

This chapter is Quirrell’s funeral, which is mostly a student giving a long eulogy (remember, Hariezer dressed things up to make it look like Quirrell died fighting Voldemort, which is sort of true, but not the Quirrell anyone knew.)

HPMOR 119

Still in resolution mode.

Hariezer comes clean with (essentially) the order of the Phoenix members, and tells them about how Dumbledore is trapped in the mirror. This leads to him receiving some letters Dumbledore left.

We find out that Dumbledore has been acting to fulfill a certain prophecy that Hariezer plays a role in-

Yet in your case, Harry, and in your case alone, the prophecies of your apocalypse have loopholes, though those loopholes be ever so slight. Always ‘he will end the world’, not ‘he will end life’.

So I guess he’ll bring in the transhumanist future.

Hariezer has also been given Dumbledore’s place, which Amelia Bones is annoyed at, so he makes her regent until he is old enough.

We also get one last weird pointless rearrangement of the canon books- apparently Peter Pettigrew was one of those shape shifting wizards and somehow got tricked into looking like Sirius Black. So the wrong person has been locked up in Azkaban. I don’t really “get” this whole Sirius/Peter were lovers/Sirius was evil/the plot of book 3 was a schizophrenic delusion running thread (also, Hariezer deduces, with only the evidence that there is a Black in prison and a dead Black among the death eaters, that Peter Pettigrew was a shapeshifter, that Peter immitated black, and that Peter is the one in Azkaban.)

And Hariezer puts a plan in place to open a hospital using the philsophers stone, so BAM, death is defeated, at least in the wizarding world. Unless it turns out the stone has limits or something.

HPMOR 120

More resolution.

Hariezer comes clean to Draco about how the death eaters died. (why did he go to the effort of the subterfuge, if he was going to come clean to everyone afterwards? It just added a weird layer to all this resolution).

Draco is sad his parents are dead. BUT, surprise- as I predicted way back when, Dumbledore only faked Narcissa Malfoy’s death and put her in magic witness protection.

I think one of the things I strongly dislike about HPMOR is that there doesn’t seem to be any joy...

I think one of the things I strongly dislike about HPMOR is that there doesn’t seem to be any joy purely in the discovery. People have fun playing the battle games, or fighting bullies with time turners, or generally being powerful, but no one seems to have fun just trying to figure things out.

For some reason (the reason is that I have a fair amount of scotch in me actually), my brain keeps trying to put together an imprecise metaphor to old SNES rpgs- a friend of mine in grade school loved FF2, but he always went out of his way to find all the powerups and do all the side quests,etc. This meant he was always powerful enough to smash boss fights in one or two punches. And I always hated that- what is the fun in that? What is the challenge? When things got too easy, I started running from all the random encounters and stopped buying equipment so that the boss battles were more fun.

And HPMOR feels like playing the game the first way- instead of working hard at the fun part (discovery), you get to just use Aristotle’s method (Harry Potter and Methods of Aristotelian Science) and slap an answer down. And that answer makes you more powerful- you can time turner all your problems away like shooing a misquito with a flamethrower, when a dementor shows up you get to destroy it just by thinking hard- no discovery required. The story repeatedly skips the fun part- the struggle, the learning, the discovery.

HPMOR 121

Snape leaves hogwarts, thus completing an arc I don’t think I ever cared about.

HPMOR 122: the end of the beginning

So unlike the canon books, the end of HPMOR sets it up more as an origin story than a finished adventure. After the canon books, we get the impression Harry, Ron and Hermione settled into peaceful wizard lives. After HPMOR, Hariezer has set up a magical think tank to solve the problem of friendly magic, with Hermione as his super-powered, indestructible lab assistant (tell me again how Hariezer isn't a self insert?) , and we get the impression the real work is just starting. He also has the idea to found CFAR:

It would help if Muggles had classes for this sort of thing, but they didn't. Maybe Harry could recruit Daniel Kahneman, fake his death, rejuvenate him with the Stone, and put him in charge of inventing better training methods...

We also learn that a more open society of idea sharing is an idea so destructive that Hariezer's vow to end the world wouldn't let him do it:

Harry could look back now on the Unbreakable Vow that he'd made, and guess that if not for that Vow, disaster might have already been set in motion yesterday when Harry had wanted to tear down the International Statute of Secrecy.

So secretive magiscience lead by Hariezer (with Hermione as super-powered "Sparkling Unicorn Princess" side kick) will save the day, sometime in the future.

An acquaintance of mine, let’s call him Mike, is looking for work after getting laid off from a contract role at Microsoft, which has happened to a lot of people I know. Like me, Mike has 11 years in industry. Unlike me, he doesn't know a lot of folks at trendy companies, so I passed his resume around to some engineers I know at companies that are desperately hiring. My engineering friends thought Mike's resume was fine, but most recruiters rejected him in the resume screening phase.

When I asked why he was getting rejected, the typical response I got was:

1. Tech experience is in irrelevant tech
2. "Experience is too random, with payments, mobile, data analytics, and UX."
3. Contractors are generally not the strongest technically

This response is something from a recruiter that was relayed to me through an engineer; the engineer was incredulous at the response from the recruiter. Just so we have a name, let's call this company TrendCo. It's one of the thousands of companies that claims to have world class engineers, hire only the best, etc. This is one company in particular, but it's representative of a large class of companies and the responses Mike has gotten.

Anyway, (1) is code for “Mike's a .NET dev, and we don't like people with Windows experience”.

I'm familiar with TrendCo's tech stack, which multiple employees have told me is “a tire fire”. Their core systems top out under 1k QPS, which has caused them to go down under load. Mike has worked on systems that can handle multiple orders of magnitude more load, but his experience is, apparently, irrelevant.

(2) is hard to make sense of. I've interviewed at TrendCo and one of the selling points is that it's a startup where you get to do a lot of different things. TrendCo almost exclusively hires generalists but Mike is, apparently, too general for them.

(3), combined with (1), gets at what TrendCo's real complaint with Mike is. He's not their type. TrendCo's median employee is a recent graduate from one of maybe five “top” schools with 0-2 years of experience. They have a few experienced hires, but not many, and most of their experienced hires have something trendy on their resume, not a boring old company like Microsoft.

Whether or not you think there's anything wrong with having a type and rejecting people who aren't your type, as Thomas Ptacek has observed, if your type is the same type everyone else is competing for, “you are competing for talent with the wealthiest (or most overfunded) tech companies in the market”.

If you look at new grad hiring data, it looks like FB is offering people with zero experience > $100k/ salary, $100k signing bonus, and $150k in RSUs, for an amortized total comp > $160k/yr, including $240k in the first year. Google's package has > $100k salary, a variable signing bonus in the $10k range, and $187k in RSUs. That comes in a bit lower than FB, but it's much higher than most companies that claim to only hire the best are willing to pay for a new grad. Keep in mind that compensation can go much higher for contested candidates, and that compensation for experienced candidates is probably higher than you expect if you're not a hiring manager who's seen what competitive offers look like today.

By going after people with the most sought after qualifications, TrendCo has narrowed their options down to either paying out the nose for employees, or offering non-competitive compensation packages. TrendCo has chosen the latter option, which partially explains why they have, proportionally, so few senior devs -- the compensation delta increases as you get more senior, and you have to make a really compelling pitch to someone to get them to choose TrendCo when you're offering $150k/yr less than the competition. And as people get more experience, they're less likely to believe the part of the pitch that explains how much the stock options are worth.

Just to be clear, I don't have anything against people with trendy backgrounds. I know a lot of these people who have impeccable interviewing skills and got 5-10 strong offers last time they looked for work. I've worked with someone like that: he was just out of school, his total comp package was north of $200k/yr, and he was worth every penny. But think about that for a minute. He had strong offers from six different companies, of which he was going to accept at most one. Including lunch and phone screens, the companies put in an average of eight hours apiece interviewing him. And because they wanted to hire him so much, the companies that were really serious spent an average of another five hours apiece of engineer time trying to convince him to take their offer. Because these companies had, on average, a ⅙ chance of hiring this person, they have to spend at least an expected (8+5) * 6 = 78 hours of engineer time¹. People with great backgrounds are, on average, pretty great, but they're really hard to hire. It's much easier to hire people who are underrated, especially if you're not paying market rates.

I've seen this hyperfocus on hiring people with trendy backgrounds from both sides of the table, and it's ridiculous from both sides.

On the referring side of hiring, I tried to get a startup I was at to hire the most interesting and creative programmer I've ever met, who was tragically underemployed for years because of his low GPA in college. We declined to hire him and I was told that his low GPA meant that he couldn't be very smart. Years later, Google took a chance on him and he's been killing it since then. He actually convinced me to join Google, and at Google, I tried to hire one of the most productive programmers I know, who was promptly rejected by a recruiter for not being technical enough.

On the candidate side of hiring, I've experienced both being in demand and being almost unhireable. Because I did my undergrad at Wisconsin, which is one of the 25 schools that claims to be a top 10 cs/engineering school, I had recruiters beating down my door when I graduated. But that's silly -- that I attended Wisconsin wasn't anything about me; I just happened to grow up in the state of Wisconsin. If I grew up in Utah, I probably would have ended up going to school at Utah. When I've compared notes with folks who attended schools like Utah and Boise State, their education is basically the same as mine. Wisconsin's rank as an engineering school comes from having professors who do great research which is, at best, weakly correlated to effectiveness at actually teaching undergrads. Despite getting the same engineering education you could get at hundreds of other schools, I had a very easy time getting interviews and finding a great job.

I spent 7.5 years in that great job, at Centaur. Centaur has a pretty strong reputation among hardware companies in Austin who've been around for a while, and I had an easy time shopping for local jobs at hardware companies. But I don't know of any software folks who've heard of Centaur, and as a result I couldn't get an interview at most software companies. There were even a couple of cases where I had really strong internal referrals and the recruiters still didn't want to talk to me, which I found funny and my friends found frustrating.

When I could get interviews, they often went poorly. A typical rejection reason was something like “we process millions of transactions per day here and we really need someone with more relevant experience who can handle these things without ramping up”. And then Google took a chance on me and I was the second person on a project to get serious about deep learning performance, which was a 20%-time project until just before I joined. We built the fastest deep learning system in the world. From what I hear, they're now on the Nth generation of that project, but even the first generation thing we built had better per-rack performance and performance per dollar than any other production system out there for years (excluding follow-ons to that project, of course).

While I was at Google I had recruiters pinging me about job opportunities all the time. And now that I'm at boring old Microsoft, I don't get nearly as many recruiters reaching out to me. I've been considering looking for work² and I wonder how trendy I'll be if I do. Experience in irrelevant tech? Check! Random experience? Check! Contractor? Well, no. But two out of three ain't bad.

My point here isn't anything about me. It's that here's this person³ who has wildly different levels of attractiveness to employers at various times, mostly due to superficial factors that don't have much to do with actual productivity. This is a really common story among people who end up at Google. If you hired them before they worked at Google, you might have gotten a great deal! But no one (except Google) was willing to take that chance. There's something to be said for paying more to get a known quantity, but a company like TrendCo that isn't willing to do that cripples its hiring pipeline by only going after people with trendy resumes, and if you wouldn't hire someone before they worked at Google and would after, the main thing you know is that the person is above average at whiteboard algorithms quizzes (or got lucky one day).

I don't mean to pick on startups like TrendCo in particular. Boring old companies have their version of what a trendy background is, too. A friend of mine who's desperate to hire can't do anything with some of the resumes I pass his way because his group isn't allowed to hire anyone without a degree. Another person I know is in a similar situation because his group has a bright-line rule that causes them to reject people who aren't already employed.

Not only are these decisions non-optimal for companies, they create a path dependence in employment outcomes that causes individual good (or bad) events to follow people around for decades. You can see similar effects in the literature on career earnings in a variety of fields⁴.

Thomas Ptacek has this great line about how “we interview people whose only prior work experience is "Line of Business .NET Developer", and they end up showing us how to write exploits for elliptic curve partial nonce bias attacks that involve Fourier transforms and BKZ lattice reduction steps that take 6 hours to run.” If you work at a company that doesn't reject people out of hand for not being trendy, you'll hear lots of stories like this. Some of the best people I've worked with went to schools you've never heard of and worked at companies you've never heard of until they ended up at Google. Some are still at companies you've never heard of.

If you read Zach Holman, you may recall that when he said that he was fired, someone responded with “If an employer has decided to fire you, then you've not only failed at your job, you've failed as a human being.” A lot of people treat employment status and credentials as measures of the inherent worth of individuals. But a large component of these markers of success, not to mention success itself, is luck.

Solutions?

I can understand why this happens. At an individual level, we're prone to the fundamental attribution error. At an organizational level, fast growing organizations burn a large fraction of their time on interviews, and the obvious way to cut down on time spent interviewing is to only interview people with "good" qualifications. Unfortunately, that's counterproductive when you're chasing after the same tiny pool of people as everyone else.

Here are the beginnings of some ideas. I'm open to better suggestions!

Moneyball

Billy Beane and Paul Depodesta took the Oakland A's, a baseball franchise with nowhere near the budget of top teams, and created what was arguably the best team in baseball by finding and “hiring” players who were statistically underrated for their price. The thing I find really amazing about this is that they publicly talked about doing this, and then Michael Lewis wrote a book, titled Moneyball, about them doing this. Despite the publicity, it took years for enough competitors to catch on enough that the A's strategy stopped giving them a very large edge.

You can see the exact same thing in software hiring. Thomas Ptacek has been talking about how they hired unusually effective people at Matasano for at least half a decade, maybe more. Google bigwigs regularly talk about the hiring data they have and what hasn't worked. I believe they talked about how focusing on top schools wasn't effective and didn't turn up employees that have better performance years ago, but that doesn't stop TrendCo from focusing hiring efforts on top schools.

Training / mentorship

You see a lot of talk about moneyball, but for some reason people are less excited about… trainingball? Practiceball? Whatever you want to call taking people who aren't “the best” and teaching them how to be “the best”.

This is another one where it's easy to see the impact through the lens of sports, because there is so much good performance data. Since it's basketball season, if we look at college basketball, for example, we can identify a handful of programs that regularly take unremarkable inputs and produce good outputs. And that's against a field of competitors where every team is expected to coach and train their players.

When it comes to tech companies, most of the competition isn't even trying. At the median large company, you get a couple days of “orientation”, which is mostly legal mumbo jumbo and paperwork, and the occasional “training”, which is usually a set of videos and a set of multiple-choice questions that are offered up for compliance reasons, not to teach anyone anything. And you'll be assigned a mentor who, more likely than not, won't provide any actual mentorship. Startups tend to be even worse! It's not hard to do better than that.

Considering how much money companies spend on hiring and retaining "the best", you'd expect them to spend at least a (non-zero) fraction on training. It's also quite strange that companies don't focus more or training and mentorship when trying to recruit. Specific things I've learned in specific roles have been tremendously valuable to me, but it's almost always either been a happy accident, or something I went out of my way to do. Most companies don't focus on this stuff. Sure, recruiters will tell you that "you'll learn so much more here than at Google, which will make you more valuable", implying that it's worth the $150k/yr pay cut, but if you ask them what, specifically, they do to make a better learning environment than Google, they never have a good answer.

Process / tools / culture

I've worked at two companies that both have effectively infinite resources to spend on tooling. One of them, let's call them ToolCo, is really serious about tooling and invests heavily in tools. People describe tooling there with phrases like “magical”, “the best I've ever seen”, and “I can't believe this is even possible”. And I can see why. For example, if you want to build a project that's millions of lines of code, their build system will make that take somewhere between 5s and 20s (assuming you don't enable LTO or anything else that can't be parallelized)⁵. In the course of a regular day at work you'll use multiple tools that seem magical because they're so far ahead of what's available in the outside world.

The other company, let's call them ProdCo pays lip service to tooling, but doesn't really value it. People describing ProdCo tools use phrases like “world class bad software” and “I am 2x less productive than I've ever been anywhere else”, and “I can't believe this is even possible”. ProdCo has a paper on a new build system; their claimed numbers for speedup from parallelization/caching, onboarding time, and reliability, are at least two orders of magnitude worse than the equivalent at ToolCo. And, in my experience, the actual numbers are worse than the claims in the paper. In the course of a day of work at ProdCo, you'll use multiple tools that are multiple orders of magnitude worse than the equivalent at ToolCo in multiple dimensions. These kinds of things add up and can easily make a larger difference than “hiring only the best”.

Processes and culture also matter. I once worked on a team that didn't use version control or have a bug tracker. For every no-brainer item on the Joel test, there are teams out there that make the wrong choice.

Although I've only worked on one team that completely failed the Joel test (they scored a 1 out of 12), every team I've worked on has had glaring deficiencies that are technically trivial (but sometimes culturally difficult) to fix. When I was at Google, we had really bad communication problems between the two halves of our team that were in different locations. My fix was brain-dead simple: I started typing up meeting notes for all of our local meetings and discussions and taking questions from the remote team about things that surprised them in our notes. That's something anyone could have done, and it was a huge productivity improvement for the entire team. I've literally never found an environment where you can't massively improve productivity with something that trivial. Sometimes people don't agree (e.g., it took months to get the non-version-control-using-team to use version control), but that's a topic for another post.

Programmers are woefully underutilized at most companies. What's the point of hiring "the best" and then crippling them? You can get better results by hiring undistinguished folks and setting them up for success, and it's a lot cheaper.

Conclusion

When I started programming, I heard a lot about how programmers are down to earth, not like those elitist folks who have uniforms involving suits and ties. You can even wear t-shirts to work! But if you think programmers aren't elitist, try wearing a suit and tie to an interview sometime. You'll have to go above and beyond to prove that you're not a bad cultural fit. We like to think that we're different from all those industries that judge people based on appearance, but we do the same thing, only instead of saying that people are a bad fit because they don't wear ties, we say they're a bad fit because they do, and instead of saying people aren't smart enough because they don't have the right pedigree… wait, that's exactly the same.

See also: developer hiring and the market for lemons

Thanks to Kelley Eskridge, Laura Lindzey, John Hergenroeder, Kamal Marhubi, Julia Evans, Steven McCarthy, Lindsey Kuper, Leah Hanson, Darius Bacon, Pierre-Yves Baccou, Kyle Littler, Jorge Montero, Sierra Rotimi-Williams, and Mark Dominus for discussion/comments/corrections.

The book starts with a story about a time Margaret Hamilton brought her young daughter with her to NASA, back in the days of the Apollo program. During a simulation mission, her daughter caused the mission to crash by pressing some keys that caused a prelaunch program to run during the simulated mission. Hamilton submitted a change request to add error checking code to prevent the error from happening again, but the request was rejected because the error case should never happen.

On the next mission, Apollo 8, that exact error condition occurred and a potentially fatal problem that could have been prevented with a trivial check took NASA’s engineers 9 hours to resolve.

This sounds familiar -- I’ve lost track of the number of dev post-mortems that have the same basic structure.

This is an experiment in note-taking for me in two ways. First, I normally take pen and paper notes and then scan them in for posterity. Second, I normally don’t post my notes online, but I’ve been inspired to try this by Jamie Brandon’s notes on books he’s read. My handwritten notes are a series of bullet points, which may not translate well into markdown. One issue is that my markdown renderer doesn’t handle more than one level of nesting, so things will get artificially flattened. There are probably more issues. Let’s find out what they are! In case it's not obvious, asides from me are in italics.

Chapter 1: Introduction

Everything in this chapter is covered in much more detail later.

Two approaches to hiring people to manage system stability:

Traditional approach: sysadmins

• Assemble existing components and deploy to produce a service
• Respond to events and updates as they occur
• Grow team to absorb increased work as service grows
• Pros
  • Easy to implement because it’s standard
  • Large talent pool to hire from
  • Lots of available software
• Cons
  • Manual intervention for change management and event handling causes size of team to scale with load on system
  • Ops is fundamentally at odds with dev, which can cause pathological resistance to changes, which causes similarly pathological response from devs, which reclassify “launches” as “incremental updates”, “flag flips”, etc.

Google’s approach: SREs

• Have software engineers do operations
• Candidates should be able to pass or nearly pass normal dev hiring bar, and may have some additional skills that are rare among devs (e.g., L1 - L3 networking or UNIX system internals).
• Career progress comparable to dev career track
• Results
  • SREs would be bored by doing tasks by hand
  • Have the skillset necessary to automate tasks
  • Do the same work as an operations team, but with automation instead of manual labor
• To avoid manual labor trap that causes team size to scale with service load, Google places a 50% cap on the amount of “ops” work for SREs
  • Upper bound. Actual amount of ops work is expected to be much lower
• Pros
  • Cheaper to scale
  • Circumvents devs/ops split
• Cons
  • Hard to hire for
  • May be unorthodox in ways that require management support (e.g., product team may push back against decision to stop releases for the quarter because the error budget is depleted)

I don’t really understand how this is an example of circumventing the dev/ops split. I can see how it’s true in one sense, but the example of stopping all releases because an error budget got hit doesn’t seem fundamentally different from the “sysadmin” example where teams push back against launches. It seems that SREs have more political capital to spend and that, in the specific examples given, the SREs might be more reasonable, but there’s no reason to think that sysadmins can’t be reasonable.

Tenets of SRE

• SRE team responsible for latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning

Ensuring a durable focus on engineering

• 50% ops cap means that extra ops work is redirected to product teams on overflow
• Provides feedback mechanism to product teams as well as keeps load down
• Target max 2 events per 8-12 hour on-call shift
• Postmortems for all serious incidents, even if they didn’t trigger a page
• Blameless postmortems

2 events per shift is the max, but what’s the average? How many on-call events are expected to get sent from the SRE team to the dev team per week?

How do you get from a blameful postmortem culture to a blameless postmortem culture? Now that everyone knows that you should have blameless postmortems, everyone will claim to do them. Sort of like having good testing and deployment practices. I’ve been lucky to be on an on call rotation that’s never gotten paged, but when I talk to folks who joined recently and are on call, they have not so great stories of finger pointing, trash talk, and blame shifting. The fact that everyone knows you’re supposed to be blameless seems to make it harder to call out blamefulness, not easier.

Move fast without breaking SLO

• Error budget. 100% is the wrong reliability target for basically everything
• Going from 5 9s to 100% reliability isn’t noticeable to most users and requires tremendous effort
• Set a goal that acknowledges the trade-off and leaves an error budget
• Error budget can be spent on anything: launching features, etc.
• Error budget allows for discussion about how phased rollouts and 1% experiments can maintain tolerable levels of errors
• Goal of SRE team isn’t “zero outages” -- SRE and product devs are incentive aligned to spend the error budget to get maximum feature velocity

It’s not explicitly stated, but for teams that need to “move fast”, consistently coming in way under the error budget could be taken as a sign that the team is spending too much effort on reliability.

I like this idea a lot, but when I discussed this with Jessica Kerr, she pushed back on this idea because maybe you’re just under your error budget because you got lucky and a single really bad event can wipe out your error budget for the next decade. Followup question: how can you be confident enough in your risk model that you can purposefully consume error budget to move faster without worrying that a downstream (in time) bad event will put you overbudget? Nat Welch (a former Google SRE) responded to this by saying that you can build confidence through simulated disasters and other testing.

Monitoring

• Monitoring should never require a human to interpret any part of the alerting domain
• Three valid kinds of monitoring output
  • Alerts: human needs to take action immediately
  • Tickets: human needs to take action eventually
  • Logging: no action needed
  • Note that, for example, graphs are a type of log

Emergency Response

• Reliability is a function of MTTF (mean-time-to-failure) and MTTR (mean-time-to-recovery)
• For evaluating responses, we care about MTTR
• Humans add latency
• Systems that don’t require humans to respond will have higher availability due to lower MTTR
• Having a “playbook” produces 3x lower MTTR
  • Having hero generalists who can respond to everything works, but having playbooks works better

I personally agree, but boy do we like our on call heros. I wonder how we can foster a culture of documentation.

Change management

• 70% of outages due to changes in a live system. Mitigation:
  • Implement progressive rollouts
  • Monitoring
  • Rollback
• Remove humans from the loop, avoid standard human problems on repetitive tasks

Demand forecasting and capacity planning

• Straightforward, but a surprising number of teams/services don’t do it

Provisioning

• Adding capacity riskier than load shifting, since it often involves spinning up new instances/locations, making significant changes to existing systems (config files, load balancers, etc.)
• Expensive enough that it should be done only when necessary; must be done quickly
  • If you don’t know what you actually need and overprovision that costs money

Efficiency and performance

• Load slows down systems
• SREs provision to meet capacity target with a specific response time goal
• Efficiency == money

Chapter 2: The production environment at Google, from the viewpoint of an SRE

No notes on this chapter because I’m already pretty familiar with it. TODO: maybe go back and read this chapter in more detail.

Chapter 3: Embracing risk

• Ex: if a user is on a smartphone with 99% reliability, they can’t tell the difference between 99.99% and 99.999% reliability

Managing risk

• Reliability isn’t linear in cost. It can easily cost 100x more to get one additional increment of reliability
  • Cost associated with redundant equipment
  • Cost of building out features for reliability as opposed to “normal” features
  • Goal: make systems reliable enough, but not too reliable!

Measuring service risk

• Standard practice: identify metric to represent property of system to optimize
• Possible metric = uptime / (uptime + downtime)
  • Problematic for a globally distributed service. What does uptime really mean?
• Aggregate availability = successful requests / total requests
  • Obv, not all requests are equal, but aggregate availability is an ok first order approximation
• Usually set quarterly targets

Risk tolerance of services

• Usually not objectively obvious
• SREs work with product owners to translate business objectives into explicit objectives

Identifying risk tolerance of consumer services

TODO: maybe read this in detail on second pass

Identifying risk tolerance of infrastructure services

Target availability

• Running ex: Bigtable
  • Some consumer services serve data directly from Bigtable -- need low latency and high reliability
  • Some teams use bigtable as a backing store for offline analysis -- care more about throughput than reliability
• Too expensive to meet all needs generically
  • Ex: Bigtable instance
  • Low-latency Bigtable user wants low queue depth
  • Throughput oriented Bigtable user wants moderate to high queue depth
  • Success and failure are diametrically opposed in these two cases!

Cost

• Partition infra and offer different levels of service
• In addition to obv. benefits, allows service to externalize the cost of providing different levels of service (e.g., expect latency oriented service to be more expensive than throughput oriented service)

Motivation for error budgets

No notes on this because I already believe all of this. Maybe go back and re-read this if involved in debate about this.

Chapter 4: Service level objectives

Note: skipping notes on terminology section.

• Ex: Chubby planned outages
  • Google found that Chubby was consistently over its SLO, and that global Chubby outages would cause unusually bad outages at Google
  • Chubby was so reliable that teams were incorrectly assuming that it would never be down and failing to design systems that account for failures in Chubby
  • Solution: take Chubby down globally when it’s too far above its SLO for a quarter to “show” teams that Chubby can go down

What do you and your users care about?

• Too many indicators: hard to pay attention
• Too few indicators: might ignore important behavior
• Different classes of services should have different indicators
  • User-facing: availability, latency, throughput
  • Storage: latency, availability, durability
  • Big data: throughput, end-to-end latency
• All systems care about correctness

Collecting indicators

• Can often do naturally from server, but client-side metrics sometimes needed.

Aggregation

• Use distributions and not averages
• User studies show that people usually prefer slower average with better tail latency
• Standardize on common defs, e.g., average over 1 minute, average over tasks in cluster, etc.
  • Can have exceptions, but having reasonable defaults makes things easier

Choosing targets

• Don’t pick target based on current performance
  • Current performance may require heroic effort
• Keep it simple
• Avoid absolutes
  • Unreasonable to talk about “infinite” scale or “always” available
• Minimize number of SLOs
• Perfection can wait
  • Can always redefine SLOs over time
• SLOs set expectations
  • Keep a safety margin (internal SLOs can be defined more loosely than external SLOs)
• Don’t overachieve
  • See Chubby example, above
  • Another example is making sure that the system isn’t too fast under light loads

Chapter 5: Eliminating toil

Carla Geisser: "If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow."

• Def: Toil
  • Not just “work I don’t want to do”
  • Manual
  • Repetitive
  • Automatable
  • Tactical
  • No enduring value
  • O(n) with service growth
• In surveys, find 33% toil on average
  • Numbers can be as low as 0% and as high as 80%
  • Toil > 50% is a sign that the manager should spread toil load more evenly
• Is toil always bad?
  • Predictable and repetitive tasks can be calming
  • Can produce a sense of accomplishment, can be low-risk / low-stress activities

Section on why toil is bad. Skipping notetaking for that section.

Chapter 6: Monitoring distributed systems

• Why monitor?
  • Analyze long-term trends
  • Compare over time or do experiments
  • Alerting
  • Building dashboards
  • Debugging

As Alex Clemmer is wont to say, our problem isn’t that we move too slowly, it’s that we build the wrong thing. I wonder how we could get from where we are today to having enough instrumentation to be able to make informed decisions when building new systems.

Setting reasonable expectations

• Monitoring is non-trivial
• 10-12 person SRE team typically has 1-2 people building and maintaining monitoring
• Number has decreased over time due to improvements in tooling/libs/centralized monitoring infra
• General trend towards simpler/faster monitoring systems, with better tools for post hoc analysis
• Avoid “magic” systems
• Limited success with complex dependency hierarchies (e.g., “if DB slow, alert for DB, otherwise alert for website”).
  • Used mostly (only?) for very stable parts of system
• Rules that generate alerts for humans should be simple to understand and represent a clear failure

Avoiding magic includes avoiding ML?

• Lots of white-box monitoring
• Some black-box monitoring for critical stuff
• Four golden signals
  • Latency
  • Traffic
  • Errors
  • Saturation

Interesting examples from Bigtable and Gmail from chapter not transcribed. A lot of information on the importance of keeping alerts simple also not transcribed.

The long run

• There’s often a tension between long-run and short-run availability
• Can sometimes fix unreliable systems through heroic effort, but that’s a burnout risk and also a failure risk
• Taking a controlled hit in short-term reliability is usually the better trade

Chapter 7: Evolution of automation at Google

• “Automation is a force multiplier, not a panacea”
• Value of automation
  • Consistency
  • Extensibility
  • MTTR
  • Faster non-repair actions
  • Time savings

Multiple interesting case studies and explanations skipped in notes.

Chapter 8: Release engineering

• This is a specific job function at Google

Release engineer role

• Release engineers work with SWEs and SREs to define how software is released
  • Allows dev teams to focus on dev work
• Define best practices
  • Compiler flags, formats for build ID tags, etc.
• Releases automated
• Models vary between teams
  • Could be “push on green” and deploy every build
  • Could be hourly builds and deploys
  • etc.
• Hermetic builds
  • Building same rev number should always give identical results
  • Self-contained -- this includes versioning everything down the compiler used
  • Can cherry-pick fixes against an old rev to fix production software
• Virtually all changes require code review
• Branching
  • All code in main branch
  • Releases are branched off
  • Fixes can go from master to branch
  • Branches never merged back
• Testing
  • CI
  • Release process creates an audit trail that runs tests and shows that tests passed
• Config management
  • Deceptively simple, can cause instability
• Many possible schemes (all involve storing config in source control and having strict config review)
• Use mainline for config -- config maintained at head and applied immediately
  • Originally used for Borg (and pre-Borg systems)
  • Binary releases and config changes decoupled!
• Include config files and binaries in same package
  • Simple
  • Tightly couples binary and config -- ok for projects with few config files or where few configs change
• Package config into “configuration packages”
  • Same hermetic principle as for code
• Release engineering shouldn’t be an afterthought!
  • Budget resources at beginning of dev cycle

Chapter 9: Simplicity

• Stability vs. agility
  • Can make things stable by freezing -- need to balance the two
  • Reliable systems can increase agility
  • Reliable rollouts make it easier to link changes to bugs
• Virtue of boring!
• Essential vs. accidental complexity
  • SREs should push back when accidental complexity is introduced
• Code is a liability
  • Remove dead code or other bloat
• Minimal APIs
  • Smaller APIs easier to test, more reliable
• Modularity
  • API versioning
  • Same as code, where you’d avoid misc/util classes
• Releases
  • Small releases easier to measure
  • Can’t tell what happened if we released 100 changes together

Chapter 10: Altering from time-series data

Borgmon

• Similar-ish to Prometheus
• Common data format for logging
• Data used for both dashboards and alerts
• Formalized a legacy data format, “varz”, which allowed metrics to be viewed via HTTP
  • To view metrics manually, go to http://foo:80/varz
• Adding a metric only requires a single declaration in code
  • low user-cost to add new metric
• Borgmon fetches /varz from each target periodically
  • Also includes synthetic data like health check, if name was resolved, etc.,
• Time series arena
  • Data stored in-memory, with checkpointing to disk
  • Fixed sized allocation
  • GC expires oldest entries when full
  • conceptually a 2-d array with time on one axis and items on the other axis
  • 24 bytes for a data point -> 1M unique time series for 12 hours at 1-minute intervals = 17 GB
• Borgmon rules
  • Algebraic expressions
  • Compute time-series from other time-series
  • Rules evaluated in parallel on a threadpool
• Counters vs. gauges
  • Def: counters are non-decreasing
  • Def: can take any value
  • Counters preferred to gauges because gauges can lose information depending on sampling interval
• Altering
  • Borgmon rules can trigger alerts
  • Have minimum duration to prevent “flapping”
  • Usually set to two duration cycles so that missed collections don’t trigger an alert
• Scaling
  • Borgmon can take time-series data from other Borgmon (uses binary streaming protocol instead of the text-based varz protocol)
  • Can have multiple tiers of filters
• Prober
  • Black-box monitoring that monitors what the user sees
  • Can be queried with varz or directly send alerts to Altertmanager
• Configuration
  • Separation between definition of rules and targets being monitored

Chapter 11: Being on-call

• Typical response time
  • 5 min for user-facing or other time-critical tasks
  • 30 min for less time-sensitive stuff
• Response times linked to SLOs
  • Ex: 99.99% for a quarter is 13 minutes of downtime; clearly can’t have response time above 13 minutes
  • Services with looser SLOs can have response times in the 10s of minutes (or more?)
• Primary vs secondary on-call
  • Work distribution varies by team
  • In some, secondary can be backup for primary
  • In others, secondary handles non-urgent / non-paging events, primary handles pages
• Balanced on-call
  • Def: quantity: percent of time on-call
  • Def: quality: number of incidents that occur while on call

This is great. We should do this. People sometimes get really rough on-call rotations a few times in a row and considering the infrequency of on-call rotations there’s no reason to expect that this should randomly balance out over the course of a year or two.

• Balance in quantity
  • >= 50% of SRE time goes into engineering
  • Of remainder, no more than 25% spent on-call
• Prefer multi-site teams
  • Night shifts are bad for health, multi-site teams allow elimination of night shifts
• Balance in quality
  • On average, dealing with an incident (incl root-cause analysis, remediation, writing postmortem, fixing bug, etc.) takes 6 hours.
  • => shouldn’t have more than 2 incidents in a 12-hour on-call shift
  • To stay within upper bound, want very flat distribution of pages, with median value of 0
• Compensation -- extra pay for being on-call (time-off or cash)

Chapter 12: Effective troubleshooting

No notes for this chapter.

Chapter 13: Emergency response

• Test-induced emergency
  • SREs break systems to see what happens
• Ex: want to flush out hidden dependencies on a distributed MySQL database
  • Plan: block access to 1/100 of DBs
  • Response: dependent services report that they’re unable to access key systems
  • SRE response: SRE aborts exercise, tries to roll back permissions change
  • Rollback attempt fails
  • Attempt to restore access to replicas works
  • Normal operation restored in 1 hour
  • What went well: dependent teams escalated issues immediately, were able to restore access
  • What we learned: had an insufficient understanding of the system and its interaction with other systems, failed to follow incident response that would have informed customers of outage, hadn’t tested rollback procedures in test env
• Change-induced emergency
  • Changes can cause failures!
• Ex: config change to abuse prevention infra pushed on Friday triggered crash-loop bug
  • Almost all externally facing systems depend on this, become unavailable
  • Many internal systems also have dependency and become unavailable
  • Alerts start firing with seconds
  • Within 5 minutes of config push, engineer who pushed change rolled back change and services started recovering
  • What went well: monitoring fired immediately, incident management worked well, out-of-band communications systems kept people up to date even though many systems were down, luck (engineer who pushed change was following real-time comms channels, which isn’t part of the release procedure)
  • What we learned: push to canary didn’t trigger same issue because it didn’t hit a specific config keyword combination; push was considered low-risk and went through less stringent canary process, alerting was too noisy during outage
• Process-induced emergency

No notes on process-induced example.

Chapter 14: Managing incidents

This is an area where we seem to actually be pretty good. No notes on this chapter.

Chapter 15: Postmortem culture: learning from failure

I'm in strong agreement with most of this chapter. No notes.

Chapter 16: Tracking outages

• Escalator: centralized system that tracks ACKs to alerts, notifies other people if necessary, etc.
• Outalator: gives time-interleaved view of notifications for multiple queues
  • Also saves related email and allows marking some messages as “important”, can collapse non-important messages, etc.

Our version of Escalator seems fine. We could really use something like Outalator, though.

Chapter 17: Testing for reliability

Preaching to the choir. No notes on this section. We could really do a lot better here, though.

Chapter 18: Software engineering in SRE

• Ex: Auxon, capacity planning automation tool
• Background: traditional capacity planning cycle
  • 1) collect demand forecasts (quarters to years in advance)
  • 2) Plan allocations
  • 3) Review plan
  • 4) Deploy and config resources
• Traditional approach cons
  • Many things can affect plan: increase in efficiency, increase in adoption rate, cluster delivery date slips, etc.
  • Even small changes require rechecking allocation plan
  • Large changes may require total rewrite of plan
  • Labor intensive and error prone
• Google solution: intent-based capacity planning
  • Specify requirements, not implementation
  • Encode requirements and autogenerate a capacity plan
  • In addition to saving labor, solvers can do better than human generated solutions => cost savings
• Ladder of examples of increasingly intent based planning
  • 1) Want 50 cores in clusters X, Y, and Z -- why those resources in those clusters?
  • 2) Want 50-core footprint in any 3 clusters in region -- why that many resources and why 3?
  • 3) Want to meet demand with N+2 redundancy -- why N+2?
  • 4) Want 5 9s of reliability. Could find, for example, that N+2 isn’t sufficient
• Found that greatest gains are from going to (3)
  • Some sophisticated services may go for (4)
• Putting constraints into tools allows tradeoffs to be consistent across fleet
  • As opposed to making individual ad hoc decisions
• Auxon inputs
  • Requirements (e.g., “service must be N+2 per continent”, “frontend servers no more than 50ms away from backend servers”
  • Dependencies
  • Budget priorities
  • Performance data (how a service scales)
  • Demand forecast data (note that services like Colossus have derived forecasts from dependent services)
  • Resource supply & pricing
• Inputs go into solver (mixed-integer or linear programming solver)

No notes on why SRE software, how to spin up a group, etc. TODO: re-read back half of this chapter and take notes if it’s ever directly relevant for me.

Chapter 19: Load balancing at the frontend

No notes on this section. Seems pretty similar to what we have in terms of high-level goals, and the chapter doesn’t go into low-level details. It’s notable that they do [redacted] differently from us, though. For more info on lower-level details, there’s the Maglev paper.

Chapter 20: Load balancing in the datacenter

• Flow control
• Need to avoid unhealthy tasks
• Naive flow control for unhealthy tasks
  • Track number of requests to a backend
  • Treat backend as unhealthy when threshold is reached
  • Cons: generally terrible
• Health-based flow control
  • Backend task can be in one of three states: {healthy, refusing connections, lame duck}
  • Lame duck state can still take connections, but sends backpressure request to all clients
  • Lame duck state simplifies clean shutdown
• Def: subsetting: limiting pool of backend tasks that a client task can interact with
  • Clients in RPC system maintain pool of connections to backends
  • Using pool reduces latency compared to doing setup/teardown when needed
  • Inactive connections are relatively cheap, but not free, even in “inactive” mode (reduced health checks, UDP instead of TCP, etc.)
• Choosing the correct subset
  • Typ: 20-100, choose base on workload
• Subset selection: random
  • Bad utilization
• Subset selection: round robin
  • Order is permuted; each round has its own permutation
• Load balancing
  • Subset selection is for connection balancing, but we still need to balance load
• Load balancing: round robin
  • In practice, observe 2x difference between most loaded and least load
  • In practice, most expensive request can be 1000x more expensive than cheapest request
  • In addition, there’s random unpredictable variation in requests
• Load balancing: least-loaded round robin
  • Exactly what it sounds like: round-robin among least loaded backends
  • Load appears to be measured in terms of connection count; may not always be the best metric
  • This is per client, not globally, so it’s possible to send requests to a backend with many requests from other clients
  • In practice, for larg services, find that most-loaded task uses twice as much CPU as least-loaded; similar to normal round robin
• Load balancing: weighted round robin
  • Same as above, but weight with other factors
  • In practice, much better load distribution than least-loaded round robin

I wonder what Heroku meant when they responded to Rap Genius by saying “after extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that beats the simplicity and robustness of random routing to web backends that can support multiple concurrent connections”.

Chapter 21: Handling overload

• Even with “good” load balancing, systems will become overloaded
• Typical strategy is to serve degraded responses, but under very high load that may not be possible
• Modeling capacity as QPS or as a function of requests (e.g., how many keys the requests read) is failure prone
  • These generally change slowly, but can change rapidly (e.g., because of a single checkin)
• Better solution: measure directly available resources
• CPU utilization is usually a good signal for provisioning
  • With GC, memory pressure turns into CPU utilization
  • With other systems, can provision other resources such that CPU is likely to be limiting factor
  • In cases where over-provisioning CPU is too expensive, take other resources into account

How much does it cost to generally over-provision CPU like that?

• Client-side throttling
  • Backends start rejecting requests when customer hits quota
  • Requests still use resources, even when rejected -- without throttling, backends can spend most of their resources on rejecting requests
• Criticality
  • Seems to be priority but with a different name?
  • First-class notion in RPC system
  • Client-side throttling keeps separate stats for each level of criticality
  • By default, criticality is propagated through subsequent RPCs
• Handling overloaded errors
  • Shed load to other DCs if DC is overloaded
  • Shed load to other backends if DC is ok but some backends are overloaded
• Clients retry when they get an overloaded response
  • Per-request retry budget (3)
  • Per-client retry budget (10%)
  • Failed retries from client cause “overloaded; don’t retry” response to be returned upstream

Having a “don’t retry” response is “obvious”, but relatively rare in practice. A lot of real systems have a problem with failed retries causing more retries up the stack. This is especially true when crossing a hardware/software boundary (e.g., filesystem read causes many retries on DVD/SSD/spinning disk, fails, and then gets retried at the filesystem level), but seems to be generally true in pure software too.

Chapter 22: Addressing cascading failures

• Typical failure scenarios?
• Server overload
• Ex: have two servers
  • One gets overloaded, failing
  • Other one now gets all traffic and also fails
• Resource exhaustion
  • CPU/memory/threads/file descriptors/etc.
• Ex: dependencies among resources
  • 1) Java frontend has poorly tuned GC params
  • 2) Frontend runs out of CPU due to GC
  • 3) CPU exhaustion slows down requests
  • 4) Increased queue depth uses more RAM
  • 5) Fixed memory allocation for entire frontend means that less memory is available for caching
  • 6) Lower hit rate
  • 7) More requests into backend
  • 8) Backend runs out of CPU or threads
  • 9) Health checks fail, starting cascading failure
  • Difficult to determine cause during outage
• Note: policies that avoid servers that serve errors can make things worse
  • fewer backends available, which get too many requests, which then become unavailable
• Preventing server overload
  • Load test! Must have realistic environment
  • Serve degraded results
  • Fail cheaply and early when overloaded
  • Have higher-level systems reject requests (at reverse proxy, load balancer, and on task level)
  • Perform capacity planning
• Queue management
  • Queues do nothing in steady state
  • Queued reqs consume memory and increase latency
  • If traffic is steady-ish, better to keep small queue size (say, 50% or less of thread pool size)
  • Ex: Gmail uses queueless servers with failover when threads are full
  • For bursty workloads, queue size should be function of #threads, time per req, size/freq of bursts
  • See also, adaptive LIFO and CoDel
• Graceful degradation
  • Note that it’s important to test graceful degradation path, maybe by running a small set of servers near overload regularly, since this path is rarely exercised under normal circumstances
  • Best to keep simple and easy to understand
• Retries
  • Always use randomized exponential backoff
  • See previous chapter on only retrying at a single level
  • Consider having a server-wide retry budget
• Deadlines
  • Don’t do work where deadline has been missed (common theme for cascading failure)
  • At each stage, check that deadline hasn’t been hit
  • Deadlines should be propagated (e.g., even through RPCs)
• Bimodal latency
  • Ex: problem with long deadline
  • Say frontend has 10 servers, 100 threads each (1k threads of total cap)
  • Normal operation: 1k QPS, reqs take 100ms => 100 worker threads occupied (1k QPS * .1s)
  • Say 5% of operations don’t complete and there’s a 100s deadline
  • That consumes 5k threads (50 QPS * 100s)
  • Frontend oversubscribed by 5x. Success rate = 1k / (5k + 95) = 19.6% => 80.4% error rate

Using deadlines instead of timeouts is great. We should really be more systematic about this.

Not allowing systems to fill up with pointless zombie requests by setting reasonable deadlines is “obvious”, but a lot of real systems seem to have arbitrary timeouts at nice round human numbers (30s, 60s, 100s, etc.) instead of deadlines that are assigned with load/cascading failures in mind.

• Try to avoid intra-layer communication
  • Simpler, avoids possible cascading failure paths
• Testing for cascading failures
  • Load test components!
  • Load testing both reveals breaking and point ferrets out components that will totally fall over under load
  • Make sure to test each component separately
  • Test non-critical backends (e.g., make sure that spelling suggestions for search don’t impede the critical path)
• Immediate steps to address cascading failures
  • Increase resources
  • Temporarily stop health check failures/deaths
  • Restart servers (only if that would help -- e.g., in GC death spiral or deadlock)
  • Drop traffic -- drastic, last resort
  • Enter degraded mode -- requires having built this into service previously
  • Eliminate batch load
  • Eliminate bad traffic

Chapter 23: Distributed consensus for reliability

• How do we agree on questions like…
  • Which process is the leader of a group of processes?
  • What is the set of processes in a group?
  • Has a message been successfully committed to a distributed queue?
  • Does a process hold a particular lease?
  • What’s the value in a datastore for a particular key?
• Ex1: split-brain
  • Service has replicated file servers in different racks
  • Must avoid writing simultaneously to both file servers in a set to avoid data corruption
  • Each pair of file servers has one leader & one follower
  • Servers monitor each other via heartbeats
  • If one server can’t contact the other, it sends a STONITH (shoot the other node in the head)
  • But what happens if the network is slow or packets get dropped?
  • What happens if both servers issue STONITH?

This reminds me of one of my favorite distributed database postmortems. The database is configured as a ring, where each node talks to and replicates data into a “neighborhood” of 5 servers. If some machines in the neighborhood go down, other servers join the neighborhood and data gets replicated appropriately.

Sounds good, but in the case where a server goes bad and decides that no data exists and all of its neighbors are bad, it can return results faster than any of its neighbors, as well as tell its neighbors that they’re all bad. Because the bad server has no data it’s very fast and can report that its neighbors are bad faster than its neighbors can report that it’s bad. Whoops!

• Ex2: failover requires human intervention
  • A highly sharded DB has a primary for each shard, which replicates to a secondary in another DC
  • External health checks decide if the primary should failover to its secondary
  • If the primary can’t see the secondary, it makes itself unavailable to avoid the problems from “Ex1”
  • This increases operational load
  • Problems are correlated and this is relatively likely to run into problems when people are busy with other issues
  • If there’s a network issues, there’s no reason to think that a human will have a better view into the state of the world than machines in the system
• Ex3: faulty group-membership algorithms
  • What it sounds like. No notes on this part
• Impossibility results
  • CAP: P is impossible in real networks, so choose C or A
  • FLP: async distributed consensus can’t gaurantee progress with unreliable network

Paxos

• Sequence of proposals, which may or may not be accepted by the majority of processes
  • Not accepted => fails
  • Sequence number per proposal, must be unique across system
• Proposal
  • Proposer sends seq number to acceptors
  • Acceptor agrees if it hasn’t seen a higher seq number
  • Proposers can try again with higher seq number
  • If proposer recvs agreement from majority, it commits by sending commit message with value
  • Acceptors must journal to persistent storage when they accept

Patterns

• Distributed consensus algorithms are a low-level primitive
• Reliable replicated state machines
  • Fundamental building block for data config/storage, locking, leader election, etc.
  • See these papers: Schnieder, Aguilera, Amir & Kirsch
• Reliable repliacted data and config stores
  • Non distributed-consensus-based systems often use timestamps: problematic because clock synchrony can't be gauranteed
  • See Spanner paper for an example of using distributed consensus
• Leader election
  • Equivalent to distributed consensus
  • Where work of the leader can performed performed by one process or sharded, leader election pattern allows writing distributed system as if it were a simple program
  • Used by, for example, GFS and Colussus
• Distributed coordination and locking services
  • Barrier used, for example, in MapReduce to make sure that Map is finished before Reduce proceeds
• Distributed queues and messaging
  • Queues: can tolerate failures from worker nodes, but system needs to ensure that claimed tasks are processed
  • Can use leases instead of removal from queue
  • Using RSM means that system can continue processing even when queue goes down
• Performance
  • Conventional wisdom that consensus algorithms can't be used for high-throughput low-latency systems is false
  • Distributed consensus at the core of many Google systems
  • Scale makes this worse for Google than most other companies, but it still works
• Multi-Paxos
  • Strong leader process: unless a leader has not yet been elected or a failure occurs, only one round trip required to reach consensus
  • Note that another process in the group can propose at any time
  • Can ping pong back and forth and pseudo-livelock
  • Not unqique to multi-paxos,
  • Standard solutions are to elect a proposer process or use rotating proposer
• Scaling read-heavy workloads
  • Ex: Photon allows reads from any replica
  • Read from stale replica requres extra work, but doesn't produce bad incorrect results
  • To gaurantee reads are up to date, do one of the following:
  • 1) Perform a read-only consensus operation
  • 2) Read data from replica that's guaranteed to be most-up-to-date (stable leader can provide this guarantee)
  • 3) Use quorum leases
• Quorum leases
  • Replicas can be granted lease over some (or all) data in the system
• Fast Paxos
  • Designed to be faster over WAN
  • Each client can send Propose to each member of a group of acceptors directly, instead of through a leader
  • Not necessarily faster than classic Paxos -- if RTT to acceptors is long, we've traded one message across slow link plus N in parallel across fast link for N across slow link
• Stable leaders
  • "Almost all distributed consensus systems that have been designed with performance in mind use either the single stable leader pattern or a system of rotating leadership"

TODO: finish this chapter?

Chapter 24: Distributed cron

TODO: go back and read in more detail, take notes.

Chapter 25: Data processing pipelines

• Examples of this are MapReduce or Flume
• Convenient and easy to reason about the happy case, but fragile
  • Initial install is usually ok because worker sizing, chunking, parameters are carefully tuned
  • Over time, load changes, causes problems

Chapter 26: Data integrity

• Definition not necessarily obvious
  • If an interface bug causes Gmail to fail to display messages, that’s the same as the data being gone from the user’s standpoint
  • 99.99% uptime means 1 hour of downtime per year. Probably ok for most apps
  • 99.99% good bytes in a 2GB file means 200K corrupt. Probably not ok for most apps
• Backup is non-trivial
  • May have mixture of transactional and non-transactional backup and restore
  • Different versions of business logic might be live at once
  • If services are independently versioned, maybe have many combinations of versions
  • Replicas aren’t sufficient -- replicas may sync corruption
• Study of 19 data recovery efforts at Google
  • Most common user-visible data loss caused by deletion or loss of referential integrity due to software bugs
  • Hardest cases were low-grade corruption discovered weeks to months later

Defense in depth

• First layer: soft deletion
  • Users should be able to delete their data
  • But that means that users will be able to accidentally delete their data
  • Also, account hijacking, etc.
  • Accidentally deletion can also happen due to bugs
  • Soft deletion delays actual deletion for some period of time
• Second layer: backups
  • Need to figure out how much data it’s ok to lose during recovery, how long recovery can take, and how far back backups need to go
  • Want backups to go back forever, since corruption can go unnoticed for months (or longer)
  • But changes to code and schema can make recovery of older backups expensive
  • Google usually has 30 to 90 day window, depending on the service
• Third layer: early detection
  • Out-of-band integrity checks
  • Hard to do this right!
  • Correct changes can cause checkers to fail
  • But loosening checks can cause failures to get missed

No notes on the two interesting case studies covered.

Chapter 27: Reliable product launches at scale

No notes on this chapter in particular. A lot of this material is covered by or at least implied by material in other chapters. Probably worth at least looking at example checklist items and action items before thinking about launch strategy, though. Also see appendix E, launch coordination checklist.

Chapters 28-32: Various chapters on management

No notes on these.

Notes on the notes

I like this book a lot. If you care about building reliable systems, reading through this book and seeing what the teams around you don’t do seems like a good exercise. That being said, the book isn't perfect. The two big downsides for me stem from the same issue: this is one of those books that's a collection of chapters by different people. Some of the editors are better than others, meaning that some of the chapters are clearer than others and that because the chapters seem designed to be readable as standalone chapters, there's a fair amount of redundancy in the book if you just read it straight through. Depending on how you plan to use the book, that can be a positive, but it's a negative to me. But even including he downsides, I'd say that this is the most valuable technical book I've read in the past year and I've covered probably 20% of the content in this set of notes. If you really like these notes, you'll probably want to read the full book.

If you found this set of notes way too dry, maybe try this much more entertaining set of notes on a totally different book. If you found this to only be slightly too dry, maybe try this set of notes on classes of errors commonly seen in postmortems. In any case, I’d appreciate feedback on these notes. Writing up notes is an experiment for me. If people find these useful, I'll try to write up notes on books I read more often. If not, I might try a different approach to writing up notes or some other kind of post entirely.

This is one of those “N technical things every programmer must read” lists, except that “programmer” is way too broad a term and the styles of writing people find helpful for them are too different for any such list to contain a non-zero number of items (if you want the entire list to be helpful to everyone). So here's a list of some things you might want to read, and why you might (or might not) want to read them.

Aleksey Shipilev

If you want to understand how the JVM really works, this is one of the best resources on the internet.

Bruce Dawson

Performance explorations of a Windows programmer. Often implicitly has nice demonstrations of tooling that has no publicly available peer on Linux.

Chip Huyen

A mix of summaries of ML conferences, data analyses (e.g., on interview data posted to glassdoor or compensation data posted to levels.fyi), and generaly commentary on the industry.

One of the rare blogs that has data-driven position pieces about the industry.

Chris Fenton

Computer related projects, by which I mean things like reconstructing the Cray-1A and building mechanical computers. Rarely updated, presumably due to the amount of work that goes into the creations, but almost always interesting.

The blog posts tend to be high-level, more like pitch decks than design docs, but there's often source code available if you want more detail.

Cindy Sridharan

More active on Twitter than on her blog, but has posts that review papers as well as some on "big" topics, like distributed tracing and testing in production.

Dan McKinley

A lot of great material on how engineering companies should be run. He has a lot of ideas that sound like common sense, e.g., choose boring technology, until you realize that it's actually uncommon to find opinions that are so sensible.

Mostly distilled wisdom (as opposed to, say, detailed explanations of code).

Eli Bendersky

I think of this as “the C++ blog”, but it's much wider ranging that that. It's too wide ranging for me to sum up, but if I had to commit to a description I might say that it's a collection of deep dives into various topics, often (but not always) relatively low-level, along with short blurbs about books, often (but not always) technical.

The book reviews tend to be easy reading, but the programming blog posts are often a mix of code and exposition that really demands your attention; usually not a light read.

Erik Sink

I think Erik has been the most consistently insightful writer about tech culture over the past 20 years. If you look at people who were blogging back when he started blogging, much of Steve Yegge's writing holds up as well as Erik's, but Steve hasn't continued writing consistently.

If you look at popular writers from that era, I think they generally tend to not really hold up very well.

Fabian Giesen

Covers a wide variety of technical topics. Emphasis on computer architecture, compression, graphics, and signal processing, but you'll find many other topics as well.

Posts tend towards being technically intense and not light reading and they usually explain concepts or ideas (as opposed to taking sides and writing opinion pieces).

Fabien Sanglard

In depth techincal dives on game related topics, such as this readthrough of the Doom source code, this history of Nvidia GPU architecture, or this read of a business card raytracer.

Fabrice Bellard

Not exactly a blog, but every time a new project appears on the front page, it's amazing. Some examples are QEMU, FFMPEG, a 4G LTE base station that runs on a PC, a JavaScript PC emulator that can boot Linux, etc.

Fred Akalin

Explanations of CS-related math topics (with a few that aren't directly CS related).

Gary Bernhardt

Another “not exactly a blog”, but it's more informative than most blogs, not to mention more entertaining. This is the best “blog” on the pervasive brokenness of modern software that I know of.

Jaana Dogan

rakyll.org has posts on Go, some of which are quite in depth, e.g., this set of notes on the Go generics proposal and Jaana's medium blog has some posts on Go as well as posts on various topics in distributed systems.

Also, Jaana's Twitter has what I think of as "intellectually honest critiques of the industry", which I think is unusual for critiques of the industry on Twitter. It's more typical to see people scoring points at the expense of nuance or even being vaugely in the vicinity of correctness, which is why I think it's worth calling out these honest critiques.

Jamie Brandon

I'm so happy that I managed to convince Jamie that, given his preferences, it would make sense to take a crack at blogging full-time to support himself. Since Jamie started taking donations until today, this blog has been an absolute power house with posts like this series on problems with SQL, this series on streaming systems, great work on technical projects like dida and imp, etc.

It remains to be seen whether or not Jamie will be able to convince me to try blogging as a full-time job.

Janet Davis

This is the story of how a professor moved from Grinnel to Whitman and started a CS program from scratch. The archives are great reading if you're interested in how organizations form or CS education.

Jeff Preshing

Mostly technical content relating to C++ and Python, but also includes topics that are generally useful for programmers, such as read-modify-write operations, fixed-point math, and memory models.

Jessica Kerr

Jessica is probably better known for her talks than her blog? Her talks are great! My favorite is probably this talk with explains different concurrency models in an easy to understand way, but the blog also has a lot of material I like.

As is the case with her talks, the diagrams often take a concept and clarify it, making something that wasn't obvious seem very obvious in retrospect.

John Regehr

I think of this as the “C is harder than you think, even if you think C is really hard” blog, although the blog actually covers a lot more than that. Some commonly covered topics are fuzzing, compiler optimization, and testing in general.

Posts tend to be conceptual. When there are code examples, they're often pretty easy to read, but there are also examples of bizzaro behavior that won't be easy to skim unless you're someone who knows the C standard by heart.

Juho Snellman

A lot of posts about networking, generally written so that they make sense even with minimal networking background. I wish more people with this kind of knowledge (in depth knowledge of systems, not just networking knowledge in particular) would write up explanations for a general audience. Also has interesting non-networking content, like this post on Finnish elections.

Julia Evans

AFAICT, the theme is “things Julia has learned recently”, which can be anything from Huffman coding to how to be happy when working in a remote job. When the posts are on a topic I don't already know, I learn something new. When they're on a topic I know, they remind me that the topic is exciting and contains a lot of wonder and mystery.

Many posts have more questions than answers, and are more of a live-blogged exploration of a topic than an explanation of the topic.

Karla Burnett

A mix of security-related topics and explanations of practical programming knowledge. This article on phishing, which includes a set of fun case studies on how effective phising can be, even after people take anti-phishing training, is an example of a security post. This post on printing out text via tracert. This post on writing an SSH client and this post on some coreutils puzzles are examples of practical programming explanations.

Although the blog is security oriented, posts are written for a general audience and don't assume specific expertise in security.

Kate Murphy

Mostly small, self-contained explorations like, what's up with this Python integer behavior, how do you make a git blow up with a simple repo, or how do you generate hash collisions in Lua?

Kavya Joshi

I generally prefer technical explanations in text over video, but her exposition is so clear that I'm putting these talks in this list of blogs. Some examples include an explanation of the go race detector, simple math that's handy for performance modeling, and time.

Kyle Kingsbury

90% of Kyle's posts are explanations of distributed systems testing, which expose bugs in real systems that most of us rely on. The other 10% are musings on programming that are as rigorous as Kyle's posts on distributed systems. Possibly the most educational programming blog of all time.

For those of us without a distributed systems background, understanding posts often requires a bit of Googling, despite the extensive explanations in the posts. Most new posts are now at jepsen.io

Laura Lindzey

Very infrequently updated (on the order of once a year) with explanations of things Laura has been working on, from Oragami PCB to Ice-Penetrating Radar.

Laurie Tratt

This blog has been going since 2004 and its changed over the years. Recently, it's had some of the best posts on benchmarking around:

• VM performance, part 1
  • Thoroughly refutes the idea that you can run a language VM for some warmup period and then take some numbers when they become stable
• VM performance, part 2
• Why not use minimum times when benchmarking
  • "Everyone" who's serious about performance knows this and it's generally considered too obvious to write up, but this is still a widely used technique in benchmarking even though it's only appropriate in limited circumstances

The blog isn't purely technical, this blog post on advice is also stellar. If those posts don't sound interesting to you, it's worth checking out the archives to see if some of the topics Lawrence used to write about more frequently are to your taste.

Marc Brooker

A mix of theory and wisdom from a distributed systems engineer on EBS at Amazon. The theory posts tend to be relatively short and easy to swallow; not at all intimidating, as theory sometimes is.

Marek Majkowski

This used to be a blog about random experiments Marek was doing, like this post on bitsliced SipHash. Since Marek joined Cloudflare, this has turned into a list of things Marek has learned while working in Cloudflare's networking stack, like this story about debugging slow downloads.

Posts tend to be relatively short, but with enough technical specifics that they're not light reads.

Nicole Express

Explorations on old systems, often gaming related. Some exmaples are this post on collision detection in Alf for the Sega Master System, this post on getting decent quality output from composite video, and this post on the Neo Geo CDZ.

Nikita Prokopov

Nikita has two blogs, both on related topics. The main blog has long-form articles, often how about modern software is terrible. THen there's grumpy.website, which gives examples of software being terrible.

Nitsan Wakart

More than you ever wanted to know about writing fast code for the JVM, from GV affects data structures to the subtleties of volatile reads.

Posts tend to involve lots of Java code, but the takeaways are often language agnostic.

Oona Raisanen

Adventures in signal processing. Everything from deblurring barcodes to figuring out what those signals from helicopters mean. If I'd known that signals and systems could be this interesting, I would have paid more attention in class.

Paul Khuong

Some content on Lisp, and some on low-level optimizations, with a trend towards low-level optimizations.

Posts are usually relatively long and self-contained explanations of technical ideas with very little fluff.

Rachel Kroll

Years of debugging stories from a long-time SRE, along with stories about big company nonsense. Many of the stories come from Lyft, Facebook, and Google. They're anonymized, but if you know about the companies, you can tell which ones are which.

The degree of anonymization often means that the stories won't really make sense unless you're familiar with the operation of systems similar to the ones in the stories.

Sophie Haskins

A blog about restoring old "pizza box" computers, with posts that generally describe the work that goes into getting these machines working again.

An example is the HP 712 ("low cost" PA-RISC workstations that went for roughly $5k to $15k in 1994 dollars, which ended up doomed due to the Intel workstation onslaught that started with the Pentium Pro in 1995), where the restoration process is described here in part 1 and then here in part 2.

Vyacheslav Egorov

In-depth explanations on how V8 works and how various constructs get optimized by a compiler dev on the V8 team. If I knew compilers were this interesting, I would have taken a compilers class back when I was in college.

Often takes topics that are considered hard and explains them in a way that makes them seem easy. Lots of diagrams, where appropriate, and detailed exposition on all the tricky bits.

whitequark

Her main site has to a variety of interesting tools she's made or worked on, many of which are FPGA or open hardware related, but some of which are completely different. Whitequark's lab notebook has a really wide variety of different results, from things like undocumented hardware quirks, to fairly serious home chemistry experiments, to various tidbits about programming and hardware development (usually low level, but not always).

She's also fairly active on twitter, with some commentary on hardware/firmware/low-level programming combined with a set of diverse topics that's too broad to easily summarize.

Yossi Kreinin

Mostly dormant since the author started doing art, but the archives have a lot of great content about hardware, low-level software, and general programming-related topics that aren't strictly programming.

90% of the time, when I get the desire to write a post about a common misconception software folks have about hardware, Yossi has already written the post and taken a lot of flak for it so I don't have to :-).

I also really like Yossi's career advice, like this response to Patrick McKenzie and this post on how managers get what they want and not what they ask for.

He's active on Twitter, where he posts extremely cynical and snarky takes on management and the industry.

This blog?

Common themes include:

• This thing that's often considered easy is harder than you might think
• This thing that's often considered hard is easier than you might think
• This obvious fact is not obvious at all
• Humans are human: humans make mistakes, and systems must be designed to account for that
• Computers will have faults, and systems must be designed to account for that
• Is it just me, or is stuff really broken?
• Hey! Look at this set of papers! Let's talk about their context and why they're interesting

The end

This list also doesn't include blogs that mostly aren't about programming, so it doesn't include, for example, Ben Kuhn's excellent blog.

Anyway, that's all for now, but this list is pretty much off the top of my head, so I'll add more as more blogs come to mind. I'll also keep this list updated with what I'm reading as I find new blogs. Please please please suggest other blogs I might like, and don't assume that I already know about a blog because it's popular. Just for example, I had no idea who either Jeff Atwood or Zed Shaw were until a few years ago, and they were probably two of the most well known programming bloggers in existence. Even with centralized link aggregators like HN and reddit, blog discovery has become haphazard and random with the decline of blogrolls and blogging as a dialogue, as opposed to the current practice of blogging as a monologue. Also, please don't assume that I don't want to read something just because it's different from the kind of blog I normally read. I'd love to read more from UX or front-end folks; I just don't know where to find that kind of thing!

Last update: 2021-07

Archive

Here are some blogs I've put into an archive section because they rarely or never update.

Alex Clemmer

This post on why making a competitor to Google search is a post in classic Alex Clemmer style. The post looks at a position that's commonly believed (web search isn't all that hard and someone should come up with a better Google) and explains why that's not an obviously correct position. That's also a common theme of his comments elsewhere, such as these comments on, stack ranking at MS, implementing POSIX on Windows, the size of the Windows codebase, Bond, and Bing.

He's sort of a modern mini-MSFT, in that it's incisive commentary on MS and MS related ventures.

Allison Kaptur

Explorations of various areas, often Python related, such as this this series on the Python interpreter and this series on the CPython peephole optimizer. Also, thoughts on broader topics like debugging and learning.

Often detailed, with inline code that's meant to be read and understood (with the help of exposition that's generally quite clear).

David Dalrymple

A mix of things from writing a 64-bit kernel from scratch shortly after learning assembly to a high-level overview of computer systems. Rarely updated, with few posts, but each post has a lot to think about.

EPITA Systems Lab

Low-level. A good example of a relatively high-level post from this blog is this post on the low fragmentation heap in Windows. Posts like how to hack a pinball machine and how to design a 386 compatible dev board are typical.

Posts are often quite detailed, with schematic/circuit diagrams. This is relatively heavy reading and I try to have pen and paper handy when I'm reading this blog.

Greg Wilson

Write-ups of papers that (should) have an impact on how people write software, like this paper on what causes failures in distributed systems or this paper on what makes people feel productive. Not updated much, but Greg still blogs on his personal site.

The posts tend to be extended abstracts that tease you into reading the paper, rather than detailed explanations of the methodology and results.

Gustavo Duarte

Explanations of how Linux works, as well as other low-level topics. This particular blog seems to be on hiatus, but "0xAX" seems to have picked up the slack with the linux-insides project.

If you've read Love's book on Linux, Duarte's explanations are similar, but tend to be more about the idea and less about the implementation. They're also heavier on providing diagrams and context. "0xAX" is a lot more focused on walking through the code than either Love or Duarte.

Huon Wilson

Explanations of various Rust-y things, from back when Huon was working on Rust. Not updated much anymore, but the content is still great for someone who's interested in technical tidbits related to Rust.

Kamal Marhubi

Technical explorations of various topics, with a systems-y bent. Kubernetes. Git push. Syscalls in Rust. Also, some musings on programming in general.

The technical explorations often get into enough nitty gritty detail that this is something you probably want to sit down to read, as opposed to skim on your phone.

Mary Rose Cook

Lengthy and very-detailed explanations of technical topics, mixed in with a wide variety of other posts.

The selection of topics is eclectic, and explained at a level of detail such that you'll come away with a solid understanding of the topic. The explanations are usually fine grained enough that it's hard to miss what's going on, even if you're a beginner programmer.

Rebecca Frankel

As far as I know, Rebecca doesn't have a programming blog, but if you look at her apparently off-the-cuff comments on other people's posts as a blog, it's one of the best written programming blogs out there. She used to be prolific on Piaw's Buzz (and probably elsewhere, although I don't know where), and you occasionally see comments elsewhere, like on this Steve Yegge blog post about brilliant engineers¹. I wish I could write like that.

Russell Smith

Homemade electronics projects from vim on a mechanical typewriter to building an electrobalance to proof spirits.

Posts tend to have a fair bit of detail, down to diagrams explaining parts of circuits, but the posts aren't as detailed as specs. But there are usually links to resources that will teach you enough to reproduce the project, if you want.

RWT

I find the archives to be fun reading for insight into what people were thinking about microprocessors and computer architecture over the past two decades. It can be a bit depressing to see that the same benchmarking controversies we had 15 years ago are being repeated today, sometimes with the same players. If anything, I'd say that the average benchmark you see passed around today is worse than what you would have seen 15 years ago, even though the industry as a whole has learned a lot about benchmarking since then.

walpurgusriot

The author of walpurgisriot seems to have abanoned the github account and moved on to another user name (and a squatter appears to have picked up her old account name), but this used to be a semi-frequently updated blog with a combination of short explorations on programming and thoughts on the industry. On pure quality of prose, this is one of the best tech blogs I've ever read; the technical content and thoughts on the industry are great as well.

This post was inspired by the two posts Julia Evans has on blogs she reads and by the Chicago undergraduate mathematics bibliography, which I've found to be the most useful set of book reviews I've ever encountered.

Thanks to Bartłomiej Filipek and Sean Barrett, Michel Schniz, Neil Henning, and Lindsey Kuper for comments/discussion/corrections.

Do concurrency bugs matter? From the literature, we know that most reported bugs in distributed systems have really simple causes and can be caught by trivial tests, even when we only look at bugs that cause really bad failures, like loss of a cluster or data corruption. The filesystem literature echos this result -- a simple checker that looks for totally unimplemented error handling can find hundreds of serious data corruption bugs. Most bugs are simple, at least if you measure by bug count. But if you measure by debugging time, the story is a bit different.

Just from personal experience, I've spent more time debugging complex non-deterministic failures than all other types of bugs combined. In fact, I've spent more time debugging some individual non-deterministic bugs (weeks or months) than on all other bug types combined. Non-deterministic bugs are rare, but they can be extremely hard to debug and they're a productivity killer. Bad non-deterministic bugs take so long to debug that relatively large investments in tools and prevention can be worth it¹.

Let's see what the academic literature has to say on non-deterministic bugs. There's a lot of literature out there, so let's narrow things down by looking at one relatively well studied area: concurrency bugs. We'll start with the literature on single-machine concurrency bugs and then look at distributed concurrency bugs.

Fonseca et al. DSN '10

They studied MySQL concurrency bugs from 2003 to 2009 and found the following:

More non-deadlock bugs (63%) than deadlock bugs (40%)

Note that these numbers sum to more than 100% because some bugs are tagged with multiple causes. This is roughly in line with the Lu et al. ASPLOS '08 paper (which we'll look at later), which found that 30% of the bugs they examined were deadlock bugs.

15% of examined failures were semantic

The paper defines a semantic failure as one "where the application provides the user with a result that violates the intended semantics of the application". The authors also find that "the vast majority of semantic bugs (92%) generated subtle violations of application semantics". By their nature, these failures are likely to be undercounted -- it's pretty hard to miss a deadlock, but it's easy to miss subtle data corruption.

15% of examined failures were latent

The paper defines latent as bugs that "do not become immediately visible to users.". Unsurprisingly, the paper finds that latent failures are closely related to semantic failures; 92% of latent failures are semantic and vice versa. The 92% number makes this finding sound more precise than it really is -- it's just that 11 out of the 12 semantic failures are latent and vice versa. That could have easily been 11 out of 11 (100%) or 10 out of 12 (83%).

That's interesting, but it's hard to tell from that if the results generalize to projects that aren't databases, or even projects that aren't MySQL.

Lu et al. ASPLOS '08

They looked at concurrency bugs in MySQL, Firefox, OpenOffice, and Apache. Some of their findings are:

97% of examined non-deadlock bugs were atomicity-violation or order-violation bugs

Of the 74 non-deadlock bugs studied, 51 were atomicity bugs, 24 were ordering bugs, and 2 were categorized as "other".

An example of an atomicity violation is this bug from MySQL:

Thread 1:

if (thd->proc_info)
  fputs(thd->proc_info, ...)

Thread 2:

thd->proc_info = NULL;

For anyone who isn't used to C or C++, thd is a pointer, and -> is the operator to access a field through a pointer. The first line in thread 1 checks if the field is null. The second line calls fputs, which writes the field. The intent is to only call fputs if and only if proc_info isn't NULL, but there's nothing preventing another thread from setting proc_info to NULL "between" the first and second lines of thread 1.

Like most bugs, this bug is obvious in retrospect, but if we look at the original bug report, we can see that it wasn't obvious at the time:

Description: I've just noticed with the latest bk tree than MySQL regularly crashes in InnoDB code ... How to repeat: I've still no clues on why this crash occurs.

As is common with large codebases, fixing the bug once it was diagnosed was more complicated than it first seemed. This bug was partially fixed in 2004, resurfaced again and was fixed in 2008. A fix for another bug caused a regression in 2009, which was also fixed in 2009. That fix introduced a deadlock that was found in 2011.

An example ordering bug is the following bug from Firefox:

Thread 1:

mThread=PR_CreateThread(mMain, ...);

Thread 2:

void mMain(...) {
  mState = mThread->State;
  }

Thread 1 launches Thread 2 with PR_CreateThread. Thread 2 assumes that, because the line that launched it assigned to mThread, mThread is valid. But Thread 2 can start executing before Thread 1 has assigned to mThread! The authors note that they call this an ordering bug and not an atomicity bug even though the bug could have been prevented if the line in thread 1 were atomic because their "bug pattern categorization is based on root cause, regardless of possible fix strategies".

An example of an "other" bug, one of only two studied, is this bug in MySQL:

Threads 1...n:

rw_lock(&lock);

Watchdog thread:

if (lock_wait_time[i] > fatal_timeout)
  assert(0);

This can cause a spurious crash when there's more than the expected amount of work. Note that the study doesn't look at performance bugs, so a bug where lock contention causes things to slow to a crawl but a watchdog doesn't kill the program wouldn't be considered.

An aside that's probably a topic for another post is that hardware often has deadlock or livelock detection built in, and that when a lock condition is detected, hardware will often try to push things into a state where normal execution can continue. After detecting and breaking deadlock/livelock, an error will typically be logged in a way that it will be noticed if it's caught in lab, but that external customers won't see. For some reason, that strategy seems rare in the software world, although it seems like it should be easier in software than in hardware.

Deadlock occurs if and only if the following four conditions are true:

1. Mutual exclusion: at least one resource must be held in a non-shareable mode. Only one process can use the resource at any given instant of time.
2. Hold and wait or resource holding: a process is currently holding at least one resource and requesting additional resources which are being held by other processes.
3. No preemption: a resource can be released only voluntarily by the process holding it.
4. Circular wait: a process must be waiting for a resource which is being held by another process, which in turn is waiting for the first process to release the resource.

There's nothing about these conditions that are unique to either hardware or software, and it's easier to build mechanisms that can back off and replay to relax (2) in software than in hardware. Anyway, back to the study findings.

96% of examined concurrency bugs could be reproduced by fixing the relative order of 2 specific threads

This sounds like great news for testing. Testing only orderings between thread pairs is much more tractable than testing all orderings between all threads. Similarly, 92% of examined bugs could be reproduced by fixing the order of four (or fewer) memory accesses. However, there's a kind of sampling bias here -- only bugs that could be reproduced could be analyzed for a root cause, and bugs that only require ordering between two threads or only a few memory accesses are easier to reproduce.

97% of examined deadlock bugs were caused by two threads waiting for at most two resources

Moreover, 22% of examined deadlock bugs were caused by a thread acquiring a resource held by the thread itself. The authors state that pairwise testing of acquisition and release sequences should be able to catch most deadlock bugs, and that pairwise testing of thread orderings should be able to catch most non-deadlock bugs. The claim seems plausibly true when read as written; the implication seems to be that virtually all bugs can be caught through some kind of pairwise testing, but I'm a bit skeptical of that due to the sample bias of the bugs studied.

I've seen bugs with many moving parts take months to track down. The worst bug I've seen consumed nearly a person-year's worth of time. Bugs like that mostly don't make it into studies like this because it's rare that a job allows someone the time to chase bugs that elusive. How many bugs like that are out there is still an open question.

Caveats

Note that all of the programs studied were written in C or C++, and that this study predates C++11. Moving to C++11 and using atomics and scoped locks would probably change the numbers substantially, not to mention moving to an entirely different concurrency model. There's some academic work on how different concurrency models affect bug rates, but it's not really clear how that work generalizes to codebases as large and mature as the ones studied, and by their nature, large and mature codebases are hard to do randomized trials on when the trial involves changing the fundamental primitives used. The authors note that 39% of examined bugs could have been prevented by using transactional memory, but it's not clear how many other bugs might have been introduced if transactional memory were used.

Tools

There are other papers on characterizing single-machine concurrency bugs, but in the interest of space, I'm going to skip those. There are also papers on distributed concurrency bugs, but before we get to that, let's look at some of the tooling for finding single-machine concurrency bugs that's in the literature. I find the papers to be pretty interesting, especially the model checking work, but realistically, I'm probably not going to build a tool from scratch if something is available, so let's look at what's out there.

HapSet

Uses run-time coverage to generate interleavings that haven't been covered yet. This is out of NEC labs; googling NEC labs HapSet returns the paper, some patent listings, but no obvious download for the tool.

CHESS

Generates unique interleavings of threads for each run. They claim that, by not tracking state, the checker is much simpler than it would otherwise be, and that they're able to avoid many of the disadvantages of tracking state via a detail that can't properly be described in this tiny little paragraph; read the paper if you're interested! Supports C# and C++. The page claims that it requires Visual Studio 2010 and that it's only been tested with 32-bit code. I haven't tried to run this on a modern *nix compiler, but IME requiring Visual Studio 2010 means that it would be a moderate effort to get it running on a modern version of Visual Studio, and a substantial effort to get it running on a modern version of gcc or clang. A quick Google search indicates that this might be patent encumbered².

Maple

Uses coverage to generate interleavings that haven't been covered yet. Instruments pthreads. The source is up on GitHub. It's possible this tool is still usable, and I'll probably give it a shot at some point, but it depends on at least one old, apparently unmaintained tool (PIN, a binary instrumentation tool from Intel). Googling (Binging?) for either Maple or PIN gives a number of results where people can't even get the tool to compile, let alone use the tool.

PACER

Samples using the FastTrack algorithm in order to keep overhead low enough "to consider in production software". Ironically, this was implemented on top of the Jikes RVM, which is unlikely to be used in actual production software. The only reference I could find for an actually downloadable tool is a completely different pacer.

ConLock / MagicLock / MagicFuzzer

There's a series of tools that are from one group which claims to get good results using various techniques, but AFAICT the source isn't available for any of the tools. There's a page that claims there's a version of MagicFuzzer available, but it's a link to a binary that doesn't specify what platform the binary is for and the link 404s.

OMEN / WOLF

I couldn't find a page for these tools (other than their papers), let alone a download link.

SherLock / AtomChase / Racageddon

Another series of tools that aren't obviously available.

Tools you can actually easily use

Valgrind / DRD / Helgrind

Instruments pthreads and easy to use -- just run valgrind with the appropriate options (-drd or -helgrind) on the binary. May require a couple tweaks if using C++11 threading.

clang thread sanitizer (TSan)

Can find data races. Flags when happens-before is violated. Works with pthreads and C++11 threads. Easy to use (just pass a -fsanitize=thread to clang).

A side effect of being so easy to use and actually available is that tsan has had a very large impact in the real world:

One interesting incident occurred in the open source Chrome browser. Up to 15% of known crashes were attributed to just one bug [5], which proved difficult to understand - the Chrome engineers spent over 6 months tracking this bug without success. On the other hand, the TSAN V1 team found the reason for this bug in a 30 minute run, without even knowing about these crashes. The crashes were caused by data races on a couple of reference counters. Once this reason was found, a relatively trivial fix was quickly made and patched in, and subsequently the bug was closed.

clang -Wthread-safety

Static analysis that uses annotations on shared state to determine if state wasn't correctly guarded.

FindBugs

General static analysis for Java with many features. Has @GuardedBy annotations, similar to -Wthread-safety.

CheckerFramework

Java framework for writing checkers. Has many different checkers. For concurrency in particular, uses @GuardedBy, like FindBugs.

rr

Deterministic replay for debugging. Easy to get and use, and appears to be actively maintained. Adds support for time-travel debugging in gdb.

DrDebug/PinPlay

General toolkit that can give you deterministic replay for debugging. Also gives you "dynamic slicing", which is watchpoint-like: it can tell you what statements affected a variable, as well as what statements are affected by a variable. Currently Linux only; claims Windows and Android support coming soon.

Other tools

This isn't an exhaustive list -- there's a ton of literature on this, and this is an area where, frankly, I'm pretty unlikely to have the time to implement a tool myself, so there's not much value for me in reading more papers to find out about techniques that I'd have to implement myself³. However, I'd be interested in hearing about other tools that are usable.

One thing I find interesting about this is that almost all of the papers for the academic tools claim to do something novel that lets them find bugs not found by other tools. They then run their tool on some codebase and show that the tool is capable of finding new bugs. But since almost no one goes and runs the older tools on any codebase, you'd never know if one of the newer tools only found a subset of the bugs that one of the older tools could catch.

Furthermore, you see cycles (livelock?) in how papers claim to be novel. Paper I will claim that it does X. Paper II will claim that it's novel because it doesn't need to do X, unlike Paper I. Then Paper III will claim that it's novel because, unlike Paper II, it does X.

Distributed systems

Now that we've looked at some of the literature on single-machine concurrency bugs, what about distributed concurrency bugs?

Leesatapornwongsa et al. ASPLOS 2016

They looked at 104 bugs in Cassandra, MapReduce, HBase, and Zookeeper. Let's look at some example bugs, which will clarify the terminology used in the study and make it easier to understanding the main findings.

Message-message race

This diagram is just for reference, so that we have a high-level idea of how different parts fit together in MapReduce:

In MapReduce bug #3274, a resource manager sends a task-init message to a node manager. Shortly afterwards, an application master sends a task-kill preemption to the same node manager. The intent is for the task-kill message to kill the task that was started with the task-init message, but the task-kill can win the race and arrive before the task-init. This example happens to be a case where two messages from different nodes are racing to get to a single node.

For example, in MapReduce bug #5358, an application master sends a kill message to node manager running a speculative task because another copy of the task finished. However, before the message is received by the node manager, the node manager's task completes, causing a complete message to be sent to the application master, causing an exception because a complete message was received after the task had completed.

Message-compute race

One example is MapReduce bug# 4157, where the application master unregisters with the resource manager. The application master then cleans up, but that clean-up races against the resource manager sending kill messages to the application's containers via node managers, causing the application master to get killed. Note that this is classified as a race and not an atomicity bug, which we'll get to shortly.

Compute-compute races can happen, but they're outside the scope of this study since this study only looks at distributed concurrency bugs.

Atomicity violation

For the purposes of this study, atomicity bugs are defined as "whenever a message comes in the middle of a set of events, which is a local computation or global communication, but not when the message comes either before or after the events". According to this definition, the message-compute race we looked at above isn't a atomicity bug because it would still be a bug if the message came in before the "computation" started. This definition also means that hardware failures that occur inside a block that must be atomic are not considered atomicity bugs.

I can see why you'd want to define those bugs as separate types of bugs, but I find this to be a bit counterintuitive, since I consider all of these to be different kinds of atomicity bugs because they're different bugs that are caused by breaking up something that needs to be atomic.

In any case, by the definition of this study, MapReduce bug #5009 is an atomicty bug. A node manager is in the process of committing data to HDFS. The resource manager kills the task, which doesn't cause the commit state to change. Any time the node tries to rerun the commit task, the task is killed by the application manager because a commit is believed to already be in process.

Fault timing

A fault is defined to be a "component failure", such as a crash, timeout, or unexpected latency. At one point, the paper refers to "hardware faults such as machine crashes", which seems to indicate that some faults that could be considered software faults are defined as hardware faults for the purposes of this study.

Anyway, for the purposes of this study, an example of a fault-timing issue is MapReduce bug #3858. A node manager crashes while committing results. When the task is re-run, later attempts to commit all fail.

Reboot timing

In this study, reboots are classified separately from other faults. MapReduce bug #3186 illustrates a reboot bug.

A resource manager sends a job to an application master. If the resource manager is rebooted before the application master sends a commit message back to the resource manager, the resource manager loses its state and throws an exception because it's getting an unexpected complete message.

Some of their main findings are:

47% of examined bugs led to latent failures

That's a pretty large difference when compared to the DSN' 10 paper that found that 15% of examined multithreading bugs were latent failures. It's plausible that this is a real difference and not just something due to a confounding variable, but it's hard to tell from the data.

63% of examined bugs were related to hardware faults

This is a large difference from what studies on "local" concurrency bugs found. I wonder how much of that is just because people mostly don't even bother filing and fixing bugs on hardware faults in non-distributed software.

64% of examined bugs were triggered by a single message's timing

44% were ordering violations, and 20% were atomicity violations. Furthermore, > 90% of bugs involved three messages (or fewer).

32% of examined bugs were due to fault or reboot timing. Note that, for the purposes of the study, a hardware fault or a reboot that breaks up a block that needed to be atomic isn't considered an atomicity bug -- here, atomicity bugs are bugs where a message arrives in the middle of a computation that needs to be atomic.

70% of bugs had simple fixes

30% were fixed by ignoring the badly timed message and 40% were fixed by delaying or ignoring the message.

Bug causes?

After reviewing the bugs, the authors propose common fallacies that lead to bugs:

1. One hop is faster than two hops
2. Zero hops are faster than one hop
3. Atomic blocks can't be broken

On (3), the authors note that it's not just hardware faults or reboots that break up atomic blocks -- systems can send kill or pre-emption messages that break up an atomic block. A fallacy which I've commonly seen in post-mortems that's not listed here, goes something like "bad nodes are obviously bad". A classic example of this is when a system starts "handling" queries by dropping them quickly, causing a load balancer to shift traffic the bad node because it's handling traffic so quickly.

One of my favorite bugs in this class from an actual system was in a ring-based storage system where nodes could do health checks on their neighbors and declare that their neighbors should be dropped if the health check fails. One node went bad, dropped all of its storage, and started reporting its neighbors as bad nodes. Its neighbors noticed that the bad node was bad, but because the bad node had dropped all of its storage, it was super fast and was able to report its good neighbors before the good neighbors could report the bad node. After ejecting its immediate neighbors, the bad node got new neighbors and raced the new neighbors, winning again for the same reason. This was repeated until the entire cluster died.

Tools

Mace

A set of language extensions (on C++) that helps you build distributed systems. Mace has a model checker that can check all possible event orderings of messages, interleaved with crashes, reboots, and timeouts. The Mace model checker is actually available, but AFAICT it requires using the Mace framework, and most distributed systems aren't written in Mace.

Modist

Another model checker that checks different orderings. Runs only one interleaving of independent actions (partial order reduction) to avoid checking redundant states. Also interleaves timeouts. Unlike Mace, doesn't inject reboots. Doesn't appear to be available.

Demeter

Like Modist, in that it's a model checker that injects the same types of faults. Uses a different technique to reduce the state space, which I don't know how to summarize succinctly. See paper for details. Doesn't appear to be available. Googling for Demeter returns some software used to model X-ray absorption?

SAMC

Another model checker. Can inject multiple crashes and reboots. Uses some understanding of the system to avoid redundant re-orderings (e.g., if a series of messages is invariant to when a reboot is injected, the system tries to avoid injecting the reboot between each message). Doesn't appear to be available.

Jepsen

As was the case for non-distributed concurrency bugs, there's a vast literature on academic tools, most of which appear to be grad-student code that hasn't been made available.

And of course there's Jepsen, which doesn't have any attached academic papers, but has probably had more real-world impact than any of the other tools because it's actually available and maintained. There's also chaos monkey, but if I'm understanding it correctly, unlike the other tools listed, it doesn't attempt to create reproducible failures.

Conclusion

Is this where you're supposed to have a conclusion? I don't have a conclusion. We've looked at some literature and found out some information about bugs that's interesting, but not necessarily actionable. We've read about tools that are interesting, but not actually available. And then there are some tools based on old techniques that are available and useful.

For example, the idea inside clang's TSan, using "happens-before" to find data races, goes back ages. There's a 2003 paper that discusses "combining two previously known race detection techniques -- lockset-based detection and happens-before-based detection -- to obtain fewer false positives than lockset-based detection alone". That's actually what TSan v1 did, but with TSan v2 they realized the tool would be more impactful if they only used happens-before because that avoids false positives, which means that people will actually use the tool. That's not something that's likely to turn into a paper that gets cited zillions of times, though. For anyone who's looked at how afl works, this story should sound familiar. AFL is emintently practical and has had a very large impact in the real world, mostly by eschewing fancy techniques from the recent literature.

If you must have a conclusion, maybe the conclusion is that individuals like Kyle Kingsbury or Michal Zalewski have had an outsized impact on industry, and that you too can probably pick an underserved area in testing and have a curiously large impact on an entire industry.

Unrelated miscellania

Rose Ames asked me to tell more "big company" stories, so here's a set of stories that explains why I haven't put a blog post up for a while. The proximal cause is that my VP has been getting negative comments about my writing. But the reasons for that are a bit of a long story. Part of it is the usual thing, where the comments I receive personally skew very heavily positive, but the comments my manager gets run the other way because it's weird to email someone's manager because you like their writing, but you might send an email if their writing really strikes a nerve.

That explains why someone in my management chain was getting emailed about my writing, but it doesn't explain why the emails went to my VP. That's because I switched teams a few months ago, and the org that I was going to switch into overhired and didn't have any headcount. I've heard conflicting numbers about how much they overhired, from 10 or 20 people to 10% or 20% (the org is quite large, and 10% would be much more than 20), as well as conflicting stories about why it happened (honest mistake vs. some group realizing that there was a hiring crunch coming and hiring as much as possible to take all of the reqs from the rest of the org). Anyway, for some reason, the org I would have worked in hired more than it was allowed to by at least one person and instituted a hiring freeze. Since my new manager couldn't hire me into that org, he transferred into an org that had spare headcount and hired me into the new org. The new org happens to be a sales org, which means that I technically work in sales now; this has some impact on my day-to-day life since there are some resources and tech talks that are only accessible by people in product groups, but that's another story. Anyway, for reasons that I don't fully understand, I got hired into the org before my new manager, and during the months it took for the org chart to get updated I was shown as being parked under my VP, which meant that anyone who wanted to fire off an email to my manager would look me up in the directory and accidentally email my VP instead.

It didn't seem like any individual email was a big deal, but since I don't have much interaction with my VP and I don't want to only be known as that guy who writes stuff which generates pushback from inside the company, I paused blogging for a while. I don't exactly want to be known that way to my manager either, but I interact with my manager frequently enough that at least I won't only be known for that.

I also wonder if these emails to my manager/VP are more likely at my current employer than at previous employers. I've never had this happen (that I know of) at another employer, but the total number of times it's happened here is low enough that it might just be coincidence.

Then again, I was just reading the archives of a really insightful internal blog and ran across a note that mentioned that the series of blog posts was being published internally because the author got static from Sinofsky about publishing posts that contradicted the party line, which eventually resulted in the author agreeing to email Sinofsky comments related to anything under Sinofsky's purview instead of publishing the comments publicly. But now that Sinofsky has moved on, the author wanted to share emails that would have otherwise been posts internally.

That kind of thing doesn't seem to be a freak occurance around here. At the same time I saw that thing about Sinofsky, I ran across a discussion on whether or not a PM was within their rights to tell someone to take down a negative review from the app store. Apparently, a PM found out that someone had written a negative rating on the PM's product in some app store and emailed the rater, telling them that they had to take the review down. It's not clear how the PM found out that the rater worked for us (do they search the internal directory for every negative rating they find?), but they somehow found out and then issued their demand. Most people thought that the PM was out of line, but there were a non-zero number of people (in addition to the PM) who thought that employees should not say anything that could be construed as negative about the company in public.

I feel like I see more of this kind of thing now than I have at other companies, but the company's really too big to tell if anyone's personal experience generalizes. Anyway, I'll probably start blogging again now that the org chart shows that I report to my actual manager, and maybe my manager will get some emails about that. Or maybe not.

Thanks to Leah Hanson, David Turner, Justin Mason, Joe Wilder, Matt Dziubinski, Alex Blewitt, Bruno Kim Medeiros Cesar, Luke Gilliam, Ben Karas, Julia Evans, Michael Ernst, and Stephen Tu for comments/corrections.

Tavish Armstrong has a great document where he describes how and when he learned the programming skills he has. I like this idea because I've found that the paths that people take to get into programming are much more varied than stereotypes give credit for, and I think it's useful to see that there are many possible paths into programming.

Personally, I spent a decade working as an electrical engineer before taking a programming job. When I talk to people about this, they often want to take away a smooth narrative of my history. Maybe it's that my math background gives me tools I can apply to a lot of problems, maybe it's that my hardware background gives me a good understanding of performance and testing, or maybe it's that the combination makes me a great fit for hardware/software co-design problems. People like a good narrative. One narrative people seem to like is that I'm a good problem solver, and that problem solving ability is generalizable. But reality is messy. Electrical engineering seemed like the most natural thing in the world, and I picked it up without trying very hard. Programming was unnatural for me, and didn't make any sense at all for years. If you believe in the common "you either have it or you don't" narrative about programmers, I definitely don't have it. And yet, I now make a living programming, and people seem to be pretty happy with the work I do.

How'd that happen? Well, if we go back to the beginning, before becoming a hardware engineer, I spent a fair amount of time doing failed kid-projects (e.g., writing a tic-tac-toe game and AI) and not really "getting" programming. I do sometimes get a lot of value out of my math or hardware skills, but I suspect I could teach someone the actually applicable math and hardware skills I have in less than a year. Spending five years in a school and a decade in industry to pick up those skills was a circuitous route to getting where I am. Amazingly, I've found that my path has been more direct than that of most of my co-workers, giving the lie to the narrative that most programmers are talented whiz kids who took to programming early.

And while I only use a small fraction of the technical skills I've learned on any given day, I find that I have a meta-skill set that I use all the time. There's nothing profound about the meta-skill set, but because I often work in new (to me) problem domains, I find my meta skillset to be more valuable than my actual skills. I don't think that you can communicate the importance of meta-skills (like communication) by writing a blog post any more than you can explain what a monad is by saying that it's like a burrito. That being said, I'm going to tell this story anyway.

Ineffective fumbling (1980s - 1996)

Many of my friends and I tried and failed multiple times to learn how to program. We tried BASIC, and could write some simple loops, use conditionals, and print to the screen, but never figured out how to do anything fun or useful.

We were exposed to some kind of lego-related programming, uhhh, thing in school, but none of us had any idea how to do anything beyond what was in the instructions. While it was fun, it was no more educational than a video game and had a similar impact.

One of us got a game programming book. We read it, tried to do a few things, and made no progress.

High school (1996 - 2000)

Our ineffective fumbling continued through high school. Due to an interest in gaming, I got interested in benchmarking, which eventually led to learning about CPUs and CPU microarchitecture. This was in the early days of Google, before Google Scholar, and before most CS/EE papers could be found online for free, so this was mostly material from enthusiast sites. Luckily, the internet was relatively young, as were the users on the sites I frequented. Much of the material on hardware was targeted at (and even written by) people like me, which made it accessible. Unfortunately, a lot of the material on programming was written by and targeted at professional programmers, things like Paul Hsieh's optimization guide. There were some beginner-friendly guides to programming out there, but my friends and I didn't stumble across them.

We had programming classes in high school: an introductory class that covered Visual Basic and an AP class that taught C++. Both classes were taught by someone who didn't really know how to program or how to teach programming. My class had a couple of kids who already knew how to program and would make good money doing programming competitions on topcoder when it opened, but they failed to test out of the intro class because that test included things like a screenshot of the VB6 IDE, where you got a point for correctly identifying what each button did. The class taught about as much as you'd expect from a class where the pre-test involved identifying UI elements from an IDE.

The AP class the year after was similarly effective. About halfway through the class, a couple of students organized an independent study group which worked through an alternate textbook because the class was clearly not preparing us for the AP exam. I passed the AP exam because it was one of those multiple choice tests that's possible to pass without knowing the material.

Although I didn't learn much, I wouldn't have graduated high school if not for AP classes. I failed enough individual classes that I almost didn't have enough credits to graduate. I got those necessary credits for two reasons: first, a lot of the teachers had a deal where, if you scored well on the AP exam, they would give you a passing grade in the class (usually an A, but sometimes a B). Even that wouldn't have been enough if my chemistry teacher hadn't also changed my grade to a passing grade when he found out I did well on the AP chemistry test¹.

Other than not failing out of high school, I'm not sure I got much out of my AP classes. My AP CS class actually had a net negative effect on my learning to program because the AP test let me opt out of the first two intro CS classes in college (an introduction to programming and a data structures course). In retrospect, I should have taken the intro classes, but I didn't, which left me with huge holes in my knowledge that I didn't really fill in for nearly a decade.

College (2000 - 2003)

Because I'd nearly failed out of high school, there was no reasonable way I could have gotten into a "good" college. Luckily, I grew up in Wisconsin, a state with a "good" school that used a formula to determine who would automatically get admitted: the GPA cutoff depended on standardized test scores, and anyone with standardized test scores above a certain mark was admitted regardless of GPA. During orientation, I talked to someone who did admissions and found out that my year was the last year they used the formula.

I majored in computer engineering and math for reasons that seem quite bad in retrospect. I had no idea what I really wanted to study. I settled on either computer engineering or engineering mechanics because both of those sounded "hard".

I made a number of attempts to come up with better criteria for choosing a major. The most serious was when I spent a week talking to professors in an attempt to find out what day-to-day life in different fields was like. That approach had two key flaws. First, most professors don't know what it's like to work in industry; now that I work in industry and talk to folks in academia, I see that most academics who haven't done stints in industry have a lot of misconceptions about what it's like. Second, even if I managed to get accurate descriptions of different fields, it turns out that there's a wide body of research that indicates that humans are basically hopeless at predicting which activities they'll enjoy. Ultimately, I decided by coin flip.

Math

I wasn't planning on majoring in math, but my freshman intro calculus course was so much fun that I ended up adding a math major. That only happened because a high-school friend of mine passed me the application form for the honors calculus sequence because he thought I might be interested in it (he'd already taken the entire calculus sequence as well as linear algebra). The professor for the class covered the material at an unusually fast pace: he finished what was supposed to be a year-long calculus textbook in part-way through the semester and then lectured on his research for the rest of the semester. The class was theorem-proof oriented and didn't involve any of that yucky memorization that I'd previously associated with math. That was the first time I'd found school engaging in my entire life and it made me really look forward to going to math classes. I later found out that non-honors calculus involved a lot of memorization when the engineering school required me to go back and take calculus II, which I'd skipped because I'd already covered the material in the intro calculus course.

If I hadn't had a friend drop the application for honors calculus in my lap, I probably wouldn't have majored in math and it's possible I never would have found any classes that seemed worth attending. Even as it was, all of the most engaging undergrad professors I had were math professors² and I mostly skipped my other classes. I don't know how much of that was because my math classes were much smaller, and therefore much more customized to the people in the class (computer engineering was very trendy at the time, and classes were overflowing), and how much was because these professors were really great teachers.

Although I occasionally get some use out of the math that I learned, most of the value was in becoming confident that I can learn and work through the math I need to solve any particular problem.

Engineering

In my engineering classes, I learned how to debug and how computers work down to the transistor level. I spent a fair amount of time skipping classes and reading about topics of interest in the library, which included things like computer arithmetic and circuit design. I still have fond memories of Koren's Computer Arithmetic Algorithms, Chandrakasan et al.'s Design of High-Performance Microprocessor Circuits. I also started reading papers; I spent a lot of time in libraries reading physics and engineering papers that mostly didn't make sense to me. The notable exception were systems papers, which I found to be easy reading. I distinctly remember reading the Dynamo paper (this was HP's paper on JITs, not the more recent Amazon work of the same name), but I can't recall any other papers I read back then.

Internships

I had two internships, one at Micron where I "worked on" flash memory, and another at IBM where I worked on the POWER6. The Micron internship was a textbook example of a bad internship. When I showed up, my manager was surprised that he was getting an intern and had nothing for me to do. After a while (perhaps a day), he found an assignment for me: press buttons on a phone. He'd managed to find a phone that used Micron flash chips; he handed it to me, told me to test it, and walked off.

After poking at the phone for an hour or two and not being able to find any obvious bugs, I walked around and found people who had tasks I could do. Most of them were only slightly less manual than "testing" a phone by mashing buttons, but I did one not-totally-uninteresting task, which was to verify that a flash chip's controller behaved correctly. Unlike my other tasks, this was amenable to automation and I was able to write a Perl script to do the testing for me.

I chose Perl because someone had a Perl book on their desk that I could borrow, which seemed like as good a reason as any at the time. I called up a friend of mine to tell him about this great "new" language and we implemented Age of Renaissance, a board game we'd played in high school. We didn't finish, but Perl was easy enough to use that we felt like we could write a program that actually did something interesting.

Besides learning Perl, I learned that I could ask people for books and read them, and I spent most of the rest of my internship half keeping an eye on a manual task while reading the books people had lying around. Most of the books had to do with either analog circuit design or flash memory, so that's what I learned. None of the specifics have really been useful to me in my career, but I learned two meta-items that were useful.

First, no one's going to stop you from spending time reading at work or spending time learning (on most teams). Micron did its best to keep interns from learning by having a default policy of blocking interns from having internet access (managers could override the policy, but mine didn't), but no one will go out of their way to prevent an intern from reading books when their other task is to randomly push buttons on a phone.

Second, I learned that there are a lot of engineering problems we can solve without anyone knowing why. One of the books I read was a survey of then-current research on flash memory. At the time, flash memory relied on some behaviors that were well characterized but not really understood. There were theories about how the underlying physical mechanisms might work, but determining which theory was correct was still an open question.

The next year, I had a much more educational internship at IBM. I was attached to a logic design team on the POWER6, and since they didn't really know what to do with me, they had me do verification on the logic they were writing. They had a relatively new tool called SixthSense, which you can think of as a souped-up quickcheck. The obvious skill I learned was how to write tests using a fancy testing framework, but the meta-thing I learned which has been even more useful is the fact that writing a test-case generator and a checker is often much more productive than the manual test-case writing that passes for automated testing in most places.

The other thing I encountered for the first time at IBM was version control (CVS, unfortunately). Looking back, I find it a bit surprising that not only did I never use version control in any of my classes, but I'd never met any other students who were using version control. My IBM internship was between undergrad and grad school, so I managed to get a B.S. degree without ever using or seeing anyone use version control.

Computer Science

I took a couple of CS classes. The first was algorithms, which was poorly taught and so heavily curved as a result that I got an A despite not learning anything at all. The course involved no programming and while I could have done some implementation in my free time, I was much more interested in engineering and didn't try to apply any of the material.

The second course was databases. There were a couple of programming projects, but they were all projects where you got some scaffolding and only had to implement a few key methods to make things work, so it was possible to do ok without having any idea how to program. I got involved in a competition to see who could attend fewest possible classes, didn't learn anything, and scraped by with a B.

Grad school (2003 - 2005)

After undergrad, I decided to go to grad school for a couple of silly reasons. One was a combination of "why not?" and the argument that most of professors gave, which was that you'll never go if you don't go immediately after undergrad because it's really hard to go back to school later. But the reason that people don't go back later is because they have more information (they know what both school and work are like), and they almost always choose work! The other major reason was that I thought I'd get a more interesting job with a master's degree. That's not obviously wrong, but it appears to be untrue in general for people going into electrical engineering and programming.

I don't know that I learned anything that I use today, either in the direct sense or in a meta sense. I had some great professors³ and I made some good friends, but I think that this wasn't a good use of time because of two bad decisions I made at the age of 19 or 20. Rather than attended a school that had a lot of people working in an area I was interested in, I went with a school that gave me a fellowship that only had one person working in an area I was really interested. That person left just before I started.

I ended up studying optics, and while learning a new field was a lot of fun, the experience was of no particular value to me, and I could have had fun studying something I had more of an interest in.

While I was officially studying optics, I still spent a lot of time learning unrelated things. At one point, I decided I should learn Lisp or Haskell, probably because of something Paul Graham wrote. I couldn't find a Lisp textbook in the library, but I found a Haskell textbook. After I worked through the exercises, I had no idea how to accomplish anything practical. But I did learn about list comprehensions and got in the habit of using higher-order functions.

Based on internet comments and advice, I had the idea learning more languages would teach me how to be a good programmer so I worked through introductory books on Python and Ruby. As far as I can tell, this taught me basically nothing useful and I would have been much better off learning about a specific area (like algorithms or networking) than learning lots of languages.

First real job (2005 - 2013)

Towards the end of grad school, I mostly looked for, and found, electrical/computer engineering jobs. The one notable exception was Google, which called me up in order to fly me out to Mountain View for an interview. I told them that they probably had the wrong person because they hadn't even done a phone screen, so they offered to do a phone interview instead. I took the phone interview expecting to fail because I didn't have any CS background, and I failed as expected. In retrospect, I should have asked to interview for a hardware position, but at the time I didn't know they had hardware positions, even though they'd been putting together their own servers and designing some of their own hardware for years.

Anyway, I ended up at a little chip company called Centaur. I was hesitant about taking the job because the interview was the easiest interview I had at any company⁴, which made me wonder if they had a low hiring bar, and therefore relatively weak engineers. It turns out that, on average, that's the best group of people I've ever worked with. I didn't realize it at the time, but this would later teach me that companies that claim to have brilliant engineers because they have super hard interviews are full of it, and that the interview difficulty one-upmanship a lot of companies promote is more of a prestige play than anything else.

But I'm getting ahead of myself — my first role was something they call "regression debug", which included debugging test failures for both newly generated tests as well as regression tests. The main goal of this job was to teach new employees the ins-and-outs of the x86 architecture. At the time, Centaur's testing was very heavily based on chip-level testing done by injecting real instructions, interrupts, etc., onto the bus, so debugging test failures taught new employees everything there is to know about x86.

The Intel x86 manual is thousands of pages long and it isn't sufficient to implement a compatible x86 chip. When Centaur made its first x86 chip, they followed the Intel manual in perfect detail, and left all instances of undefined behavior up to individual implementers. When they got their first chip back and tried it, they found that some compilers produced code that relied on the behavior that's technically undefined on x86, but happened to always be the same on Intel chips. While that's technically a compiler bug, you can't ship a chip that isn't compatible with actually existing software, and ever since then, Centaur has implemented x86 chips by making sure that the chips match the exact behavior of Intel chips, down to matching officially undefined behavior⁵.

For years afterwards, I had encyclopedic knowledge of x86 and could set bits in control registers and MSRs from memory. I didn't have a use for any of that knowledge at any future job, but the meta-skill of not being afraid of low-level hardware comes in handy pretty often, especially when I run into compiler or chip bugs. People look at you like you're a crackpot if you say you've found a hardware bug, but because we were so careful about characterizing the exact behavior of Intel chips, we would regularly find bugs and then have discussions about whether we should match the bug or match the spec (the Intel manual).

The other thing I took away from the regression debug experience was a lifelong love of automation. Debugging often involves a large number of mechanical steps. After I learned enough about x86 that debugging became boring, I started automating debugging. At that point, I knew how to write simple scripts but didn't really know how to program, so I wasn't able to totally automate the process. However, I was able to automate enough that, for 99% of failures, I just had to glance at a quick summary to figure out what the bug was, rather than spend what might be hours debugging. That turned what was previously a full-time job into something that took maybe 30-60 minutes a day (excluding days when I'd hit a bug that involved some obscure corner of x86 I wasn't already familiar with, or some bug that my script couldn't give a useful summary of).

At that point, I did two things that I'd previously learned in internships. First, I started reading at work. I began with online commentary about programming, but there wasn't much of that, so I asked if I could expense books and read them at work. This seemed perfectly normal because a lot of other people did the same thing, and there were at least two people who averaged more than one technical book per week, including one person who averaged a technical book every 2 or 3 days.

I settled in at a pace of somewhere between a book a week and a book a month. I read a lot of engineering books that imparted some knowledge that I no longer use, now that I spend most of my time writing software; some "big idea" software engineering books like Design Patterns and Refactoring, which I didn't really appreciate because I was just writing scripts; and a ton of books on different programming languages, which doesn't seem to have had any impact on me.

The only book I read back then that changed how I write software in a way that's obvious to me was The Design of Everyday Things. The core idea of the book is that while people beat themselves up for failing to use hard-to-understand interfaces, we should blame designers for designing poor interfaces, not users for failing to use them.

If you ever run into a door that you incorrectly try to pull instead of push (or vice versa) and have some spare time, try watching how other people use the door. Whenever I do this, I'll see something like half the people who try the door use it incorrectly. That's a design flaw!

The Design of Everyday Things has made me a lot more receptive to API and UX feedback, and a lot less tolerant of programmers who say things like "it's fine — everyone knows that the arguments to foo and bar just have to be given in the opposite order" or "Duh! Everyone knows that you just need to click on the menu X, select Y, navigate to tab Z, open AA, go to tab AB, and then slide the setting to AC."

I don't think all of that reading was a waste of time, exactly, but I would have been better off picking a few sub-fields in CS or EE and learning about them, rather than reading the sorts of books O'Reilly and Manning produce.

It's not that these books aren't useful, it's that almost all of them are written to make sense without any particular background beyond what any random programmer might have, and you can only get so much out of reading your 50th book targeted at random programmers. IMO, most non-academic conferences have the same problem. As a speaker, you want to give a talk that works for everyone in the audience, but a side effect of that is that many talks have relatively little educational value to experienced programmers who have been to a few conferences.

I think I got positive things out of all that reading as well, but I don't know yet how to figure out what those things are.

As a result of my reading, I also did two things that were, in retrospect, quite harmful.

One was that I really got into functional programming and used a functional style everywhere I could. Immutability, higher-order X for any possible value of X, etc. The result was code that I could write and modify quickly that was incomprehensible to anyone but a couple of coworkers who were also into functional programming.

The second big negative was that I became convinced that Perl was causing us a lot of problems. We had Perl scripts that were hard to understand and modify. They'd often be thousands of lines of code with only one or two functions and no tests which used every obscure Perl feature you could think of. Static! Magic sigils! Implicit everything! You name it, we used it. For me, the last straw was when I inserted a new function between two functions which didn't explicitly pass any arguments and return values — and broke the script because one of the functions was returning a value into an implicit variable which was getting read by the next function. By putting another function in between the two closely coupled functions, I broke the script.

After that, I convinced a bunch of people to use Ruby and started using it myself. The problem was that I only managed to convince half of my team to do this The other half kept using Perl, which resulted in language fragmentation. Worse yet, in another group, they also got fed up with Perl, but started using Python, resulting in the company having code in Perl, Python, and Ruby.

Centaur has an explicit policy of not telling people how to do anything, which precludes having team-wide or company-wide standards. Given the environment, using a "better" language seemed like a natural thing to do, but I didn't recognize the cost of fragmentation until, later in my career, I saw a company that uses standardization to good effect.

Anyway, while I was causing horrific fragmentation, I also automated away most of my regression debug job. I got bored of spending 80% of my time at work reading and I started poking around for other things to do, which is something I continued for my entire time at Centaur. I like learning new things, so I did almost everything you can do related to chip design. The only things I didn't do were circuit design (the TL of circuit design didn't want a non-specialist interfering in his area) and a few roles where I was told "Dan, you can do that if you really want to, but we pay you too much to have you do it full-time."

If I hadn't interviewed regularly (about once a year, even though I was happy with my job), I probably would've wondered if I was stunting my career by doing so many different things, because the big chip companies produce specialists pretty much exclusively. But in interviews I found that my experience was valued because it was something they couldn't get in-house. The irony is that every single role I was offered would have turned me into a specialist. Big chip companies talk about wanting their employees to move around and try different things, but when you dig into what that means, it's that they like to have people work one very narrow role for two or three years before moving on to their next very narrow role.

For a while, I wondered if I was doomed to either eventually move to a big company and pick up a hyper-specialized role, or stay at Centaur for my entire career (not a bad fate — Centaur has, by far, the lowest attrition rate of any place I've worked because people like it so much). But I later found that software companies building hardware accelerators actually have generalist roles for hardware engineers, and that software companies have generalist roles for programmers, although that might be a moot point since most software folks would probably consider me an extremely niche specialist.

Regardless of whether spending a lot of time in different hardware-related roles makes you think of me as a generalist or a specialist, I picked up a lot of skills which came in handy when I worked on hardware accelerators, but that don't really generalize to the pure software project I'm working on today. A lot of the meta-skills I learned transfer over pretty well, though.

If I had to pick the three most useful meta-skills I learned back then, I'd say they were debugging, bug tracking, and figuring out how to approach hard problems.

Debugging is a funny skill to claim to have because everyone thinks they know how to debug. For me, I wouldn't even say that I learned how to debug at Centaur, but that I learned how to be persistent. Non-deterministic hardware bugs are so much worse than non-deterministic software bugs that I always believe I can track down software bugs. In the absolute worst case, when there's a bug that isn't caught in logs and can't be caught in a debugger, I can always add tracing information until the bug becomes obvious. The same thing's true in hardware, but "recompiling" to add tracing information takes 3 months per "recompile"; compared to that experience, tracking down a software bug that takes three months to figure out feels downright pleasant.

Bug tracking is another meta-skill that everyone thinks they have, but when when I look at most projects I find that they literally don't know what bugs they have and they lose bugs all the time due to a failure to triage bugs effectively. I didn't even know that I'd developed this skill until after I left Centaur and saw teams that don't know how to track bugs. At Centaur, depending on the phase of the project, we'd have between zero and a thousand open bugs. The people I worked with most closely kept a mental model of what bugs were open; this seemed totally normal at the time, and the fact that a bunch of people did this made it easy for people to be on the same page about the state of the project and which areas were ahead of schedule and which were behind.

Outside of Centaur, I find that I'm lucky to even find one person who's tracking what the major outstanding bugs are. Until I've been on the team for a while, people are often uncomfortable with the idea of taking a major problem and putting it into a bug instead of fixing it immediately because they're so used to bugs getting forgotten that they don't trust bugs. But that's what bug tracking is for! I view this as analogous to teams whose test coverage is so low and staging system is so flaky that they don't trust themselves to make changes because they don't have confidence that issues will be caught before hitting production. It's a huge drag on productivity, but people don't really see it until they've seen the alternative.

Perhaps the most important meta-skill I picked up was learning how to solve large problems. When I joined Centaur, I saw people solving problems I didn't even know how to approach. There were folks like Glenn Henry, a fellow from IBM back when IBM was at the forefront of computing, and Terry Parks, who Glenn called the best engineer he knew at IBM. It wasn't that they were 10x engineers; they didn't just work faster. In fact, I can probably type 10x as quickly as Glenn (a hunt and peck typist) and could solve trivial problems that are limited by typing speed more quickly than him. But Glenn, Terry, and some of the other wizards knew how to approach problems that I couldn't even get started on.

I can't cite any particular a-ha moment. It was just eight years of work. When I went looking for problems to solve, Glenn would often hand me a problem that was slightly harder than I thought possible for me. I'd tell him that I didn't think I could solve the problem, he'd tell me to try anyway, and maybe 80% of the time I'd solve the problem. We repeated that for maybe five or six years before I stopped telling Glenn that I didn't think I could solve the problem. Even though I don't know when it happened, I know that I eventually started thinking of myself as someone who could solve any open problem that we had.

Grad school, again (2008 - 2010)

At some point during my tenure at Centaur, I switched to being part-time and did a stint taking classes and doing a bit of research at the local university. For reasons which I can't recall, I split my time between software engineering and CS theory.

I read a lot of software engineering papers and came to the conclusion that we know very little about what makes teams (or even individuals) productive, and that the field is unlikely to have actionable answers in the near future. I also got my name on a couple of papers that I don't think made meaningful contributions to the state of human knowledge.

On the CS theory side of things, I took some graduate level theory classes. That was genuinely educational and I really "got" algorithms for the first time in my life, as well as complexity theory, etc. I could have gotten my name on a paper that I didn't think made a meaningful contribution to the state of human knowledge, but my would-be co-author felt the same way and we didn't write it up.

I originally tried grad school again because I was considering getting a PhD, but I didn't find the work I was doing to be any more "interesting" than the work I had at Centaur, and after seeing the job outcomes of people in the program, I decided there was less than 1% chance that a PhD would provide any real value to me and went back to Centaur full time.

RC (Spring 2013)

After eight years at Centaur, I wanted to do something besides microprocessors. I had enough friends at other hardware companies to know that I'd be downgrading in basically every dimension except name recognition if I switched to another hardware company, so I started applying to software jobs.

While I was applying to jobs, I heard about RC. It sounded great, maybe even too great: when I showed my friends what people were saying about it, they thought the comments were fake. It was a great experience, and I can see why so many people raved about it, to the point where real comments sound impossibly positive. It was transformative for a lot of people; I heard a lot of exclamations like "I learned more here in 3 months here than in N years of school" or "I was totally burnt out and this was the first time I've been productive in a year". It wasn't transformative for me, but it was as fun a 3 month period as I've ever had, and I even learned a thing or two.

From a learning standpoint, the one major thing I got out of RC was feedback from Marek, whom I worked with for about two months. While the freedom and lack of oversight at Centaur was great for letting me develop my ability to work independently, I basically didn't get any feedback on my work⁶ since they didn't do code review while I was there, and I never really got any actionable feedback in performance reviews.

Marek is really great at giving feedback while pair programming, and working with him broke me of a number of bad habits as well as teaching me some new approaches for solving problems. At a meta level, RC is relatively more focused on pair programming than most places and it got me to pair program for the first time. I hadn't realized how effective pair programming with someone is in terms of learning how they operate and what makes them effective. Since then, I've asked a number of super productive programmers to pair program and I've gotten something out of it every time.

Second real job (2013 - 2014)

I was in the right place at the right time to land on a project that was just transitioning from Andy Phelps' pet 20% time project into what would later be called the Google TPU.

As far as I can tell, it was pure luck that I was the second engineer on the project as opposed to the fifth or the tenth. I got to see what it looks like to take a project from its conception and turn it into something real. There was a sense in which I got that at Centaur, but every project I worked on was either part of a CPU, or a tool whose goal was to make CPU development better. This was the first time I worked on a non-trivial project from its inception, where I wasn't just working on part of the project but the whole thing.

That would have been educational regardless of the methodology used, but it was a particularly great learning experience because of how the design was done. We started with a lengthy discussion on what core algorithm we were going to use. After we figured out an algorithm that would give us acceptable performance, we coded up design docs for every major module before getting serious about implementation.

Many people consider writing design docs to be a waste of time nowadays, but going through this process, which took months, had a couple big advantages. The first is that working through a design collaboratively teaches everyone on the team everyone else's tricks. It's a lot like the kind of skill transfer you get with pair programming, but applied to design. This was great for me, because as someone with only a decade of experience, I was one of the least experienced people in the room.

The second is that the iteration speed is much faster in the design phase, where throwing away a design just means erasing a whiteboard. Once you start coding, iterating on the design can mean throwing away code; for infrastructure projects, that can easily be person-years or even tens of persons-years of work. Since working on the TPU project, I've seen a couple of teams on projects of similar scope insist on getting "working" code as soon as possible. In every single case, that resulted in massive delays as huge chunks of code had to be re-written, and in a few cases the project was fundamentally flawed in a way that required the team had to start over from scratch.

I get that on product-y projects, where you can't tell how much traction you're going to get from something, you might want to get an MVP out the door and iterate, but for pure infrastructure, it's often possible to predict how useful something will be in the design phase.

The other big thing I got out of the job was a better understanding of what's possible when a company makes a real effort to make engineers productive. Something I'd seen repeatedly at Centaur was that someone would come in, take a look around, find the tooling to be a huge productivity sink, and then make a bunch of improvements. They'd then feel satisfied that they'd improved things a lot and then move on to other problems. Then the next new hire would come in, have the same reaction, and do the same thing. The result was tools that improved a lot while I was there, but not to the point where someone coming in would be satisfied with them. Google was the only place I'd worked where a lot of the tools seem like magic compared to what exists in the outside world⁷. Sure, people complain that a lot of the tooling is falling over, that there isn't enough documentation, and that a lot of it is out of date. All true. But the situation is much better than it's been at any other company I've worked at. That doesn't seem to actually be a competitive advantage for Google's business, but it makes the development experience really pleasant.

Third real job (2015 - 2017)

This was a surprising experience. I think I'm too close to it to really know what I got out of the experience, so fully filling in this section is a TODO.

One thing that was really interesting is there are a lot of things I used to think of as "table stakes" for getting things done that it appears that one can do without. An example is version control. I was and still am strongly in favor of using version control, but the project I worked on with a TL that was strongly against version control was still basically successful. There was a lot of overheard until we started using version control, but dealing with the fallout of not having version control and having people not really sync changes only cost me a day or two a week of manually merging in changes in my private repo to get the build to consistently work. That's obviously far from ideal, but, across the entire team, not enough of a cost to make the difference between success and failure.

RC (2017 - present)

I wanted a fun break after my last job, so I went back to RC to do fun programming-related stuff and recharge. I haven't written up most of what I've worked on (e.g., an analysis of 80k games on Terra Mystica, MTA (NYC) subway data analysis, etc.). I've written up a few things, like latency analysis of computers, terminals, keyboards, and websites, though.

One thing my time at RC has got me thinking about is why it's so hard to get paid well to write. There appears to be a lot of demand for "good" writing, but companies don't seem very willing to create roles for people who could program but want to write. Steve Klabnik has had a tremendous impact on Rust through his writing, probably more impact than the median programmer on most projects, but my impression is that he's taking a significant pay cut over what he could make as a programmer in order to do this really useful and important thing.

I've tried pitching this kind of role at a few places and the response so far has mostly been a combination of:

• We value writing! I don't think it makes sense to write full-time or even half-time, but you join my team where we support writing and you can write as a 20%-time project or in your spare time!
• Uhhh, we could work something out, but why would anyone who can program want to write?

Neither of these responses makes me think that writing would actually be as valued as programming on those teams even if writing is more valued on those teams relative to most. There are some "developer evangelist" roles that involve writing, but when I read engineering blogs written by people with that title, most of the writing appears to be thinly disguised press releases (there are obviously exceptions to this, but even in the cases where blogs have interesting engineering output, the interesting output is often interleaved with pseudo press releases). In addition to be boring, that kind of thing seems pretty ineffective. At one company I worked for, I ran the traffic numbers for their developer evangelist blogs vs. my own blog, and there were a lot of months where my blog got more traffic than all of their hosted evangelist blogs combined. I don't think it's surprising to find that programmers would rather read explanations/analysis/history than PR, but it seems difficult to convince the right people of this, so I'll probably go back to a programming job after this. We'll see.

BTW, this isn't to say that I don't enjoy programming or don't think that it's important. It's just that writing seems undervalued in a way that makes it relatively easy to have outsized impact through writing. But the same the same forces that make it easy to have outsized impact also make it difficult to get paid well!

What about the bad stuff?

When I think about my career, it seems to me that it's been one lucky event after the next. I've been unlucky a few times, but I don't really know what to take away from the times I've been unlucky.

For example, I'd consider my upbringing to be mildly abusive. I remember having nights where I couldn't sleep because I'd have nightmares about my father every time I fell asleep. Being awake during the day wasn't a great experience, either. That's obviously not good and in retrospect it seems pretty directly related to the academic problems I had until I moved out, but I don't know that I could give useful advice to a younger version of myself. Don't be born into an abusive family? That's something people would already do if they had any control over the matter.

Or to pick a more recent example, I once joined a team that scored a 1 on the Joel Test. The Joel Test is now considered to be obsolete because it awards points for things like "Do you have testers?" and "Do you fix bugs before writing new code?", which aren't considered best practices by most devs today. Of the items that aren't controversial, many seem so obvious that they're not worth asking about, things like:

• Do you use source control?
• Can you make a build in one step?
• Do you make (at least) daily builds?
• Do you have a bug database?

For anyone who cares about this kind of thing, it's clearly not a great idea to join a team that does, at most, 1 item off of Joel's checklist (and the 1 wasn't any of the above). Getting first-hand experience on a team that scored a 1 didn't give me any new information that would make me reconsider my opinion.

You might say that I should have asked about those things. It's true! I should have, and I probably will in the future. However, when I was hired, the TL who was against version control and other forms of automation hadn't been hired yet, so I wouldn't have found out about this if I'd asked. Furthermore, even if he'd already been hired, I'm still not sure I would have found out about it — this is the only time I've joined a team and then found that most of the factual statements made during the recruiting process were untrue. I made sure to ask specific, concrete, questions about the state of the project, processes, experiments that had been run, etc., but it turned out the answers were outright falsehoods. When I was on that team, every day featured a running joke between team members about how false the recruiting pitch was!

I could try to prevent similar problems in the future by asking for concrete evidence of factual claims (e.g., if someone claims the attrition rate is X, I could ask for access to the HR database to verify), but considering that I have a finite amount of time and the relatively low probability of being told outright falsehoods, I think I'm going to continue to prioritize finding out other information when I'm considering a job and just accept that there's a tiny probability I'll end up in a similar situation in the future.

When I look at the bad career-related stuff I've experienced, almost all of it falls into one of two categories: something obviously bad that was basically unavoidable, or something obviously bad that I don't know how to reasonably avoid, given limited resources. I don't see much to learn from that. That's not to say that I haven't made and learned from mistakes. I've made a lot of mistakes and do a lot of things differently as a result of mistakes! But my worst experiences have come out of things that I don't know how to prevent in any reasonable way.

This also seems to be true for most people I know. For example, something I've seen a lot is that a friend of mine will end up with a manager whose view is that managers are people who dole out rewards and punishments (as opposed to someone who believes that managers should make the team as effective as possible, or someone who believes that managers should help people grow). When you have a manager like that, a common failure mode is that you're given work that's a bad fit, and then maybe you don't do a great job because the work is a bad fit. If you ask for something that's a better fit, that's refused (why should you be rewarded with doing something you want when you're not doing good work, instead you should be punished by having to do more of this thing you don't like), which causes a spiral that ends in the person leaving or getting fired. In the most recent case I saw, the firing was a surprise to both the person getting fired and their closest co-workers: my friend had managed to find a role that was a good fit despite the best efforts of management; when management decided to fire my friend, they didn't bother to consult the co-workers on the new project, who thought that my friend was doing great and had been doing great for months!

I hear a lot of stories like that, and I'm happy to listen because I like stories, but I don't know that there's anything actionable here. Avoid managers who prefer doling out punishments to helping their employees? Obvious but not actionable.

Conclusion

The most common sort of career advice I see is "you should do what I did because I'm successful". It's usually phrased differently, but that's the gist of it. That basically never works. When I compare notes with friends and acquaintances, it's pretty clear that my career has been unusual in a number of ways, but it's not really clear why.

Just for example, I've almost always had a supportive manager who's willing to not only let me learn whatever I want on my own, but who's willing to expend substantial time and effort to help me improve as an engineer. Most folks I've talked to have never had that. Why the difference? I have no idea.

One story might be: the two times I had unsupportive managers, I quickly found other positions, whereas a lot of friends of mine will stay in roles that are a bad fit for years. Maybe I could spin it to make it sound like the moral of the story is that you should leave roles sooner than you think, but both of the bad situations I ended up in, I only ended up in because I left a role sooner than I should have, so the advice can't be "prefer to leave roles sooner than you think". Maybe the moral of the story should be "leave bad roles more quickly and stay in good roles longer", but that's so obvious that it's not even worth stating. This is arguably non-obvious because people do, in fact, stay in roles where they're miserable, but when I think of people who do so, they fall into one of two categories. Either they're stuck for extrinsic reasons (e.g., need to wait out the visa clock) or they know that they should leave but can't bring themselves to do so. There's not much to do about the former case, and in the latter case, knowing that they should leave isn't the problem. Every strategy that I can think of is either incorrect in the general case, or so obvious there's no reason to talk about it.

Another story might be: I've learned a lot of meta-skills that are valuable, so you should learn these skills. But you probably shouldn't. The particular set of meta-skills I've picked have been great for me because they're skills I could easily pick up in places I worked (often because I had a great mentor) and because they're things I really strongly believe in doing. Your circumstances and core beliefs are probably different from mine and you have to figure out for yourself what it makes sense to learn.

Yet another story might be: while a lot of opportunities come from serendipity, I've had a lot of opportunities because I spend a lot of time generating possible opportunities. When I passed around the draft of this post to some friends, basically everyone told me that I emphasized luck too much in my narrative and that all of my lucky breaks came from a combination of hard work and trying to create opportunities. While there's a sense in which that's true, many of my opportunities also came out of making outright bad decisions.

For example, I ended up at Centaur because I turned down the chance to work at IBM for a terrible reason! At the end of my internship, my manager made an attempt to convince me to stay on as a full-time employee, but I declined because I was going to grad school. But I was only going to grad school because I wanted to get a microprocessor logic design position, something I thought I couldn't get with just a bachelor's degree. But I could have gotten that position if I hadn't turned my manager down! I'd just forgotten the reason that I'd decided to go to grad school and incorrectly used the cached decision as a reason to turn down the job. By sheer luck, that happened to work out well and I got better opportunities than anyone I know from my intern cohort who decided to take a job at IBM. Have I "mostly" been lucky or prepared? Hard to say; maybe even impossible.

Careers don't have the logging infrastructure you'd need to determine the impact of individual decisions. Careers in programming, anyway. Many sports now track play-by-play data in a way that makes it possible to try to determine how much of success in any particular game or any particular season was luck and how much was skill.

Take baseball, which is one of the better understood sports. If we look at the statistical understanding we have of performance today, it's clear that almost no one had a good idea about what factors made players successful 20 years ago. One thing I find particularly interesting is that we now have much better understanding of which factors are fundamental and which factors come down to luck, and it's not at all what almost anyone would have thought 20 years ago. We can now look at a pitcher and say something like "they've gotten unlucky this season, but their foo, bar, and baz rates are all great so it appears to be bad luck on balls in play as opposed any sort of decline in skill", and we can also make statements like "they've done well this season but their fundamental stats haven't moved so it's likely that their future performance will be no better than their past performance before this season". We couldn't have made a statement like that 20 years ago. And this is a sport that's had play-by-play video available going back what seems like forever, where play-by-play stats have been kept for a century, etc.

In this sport where everything is measured, it wasn't until relatively recently that we could disambiguate between fluctuations in performance due to luck and fluctuations due to changes in skill. And then there's programming, where it's generally believed to be impossible to measure people's performance and the state of the art in grading people's performance is that you ask five people for their comments on someone and then aggregate the comments. If we're only just now able to make comments on what's attributable to luck and what's attributable to skill in a sport where every last detail of someone's work is available, how could we possibly be anywhere close to making claims about what comes down to luck vs. other factors in something as nebulous as a programming career?

In conclusion, life is messy and I don't have any advice.

Appendix A: meta-skills I'd like to learn

Documentation

I once worked with Jared Davis, a documentation wizard whose documentation was so good that I'd go to him to understand how a module worked before I talked to the owner the module. As far as I could tell, he wrote documentation on things he was trying to understand to make life easier for himself, but his documentation was so good that it was a force multiplier for the entire company.

Later, at Google, I noticed a curiously strong correlation between the quality of initial design docs and the success of projects. Since then, I've tried to write solid design docs and documentation for my projects, but I still have a ways to go.

Fixing totally broken situations

So far, I've only landed on teams where things are much better than average and on teams where things are much worse than average. You might think that, because there's so much low hanging fruit on teams that are much worse than average, it should be easier to improve things on teams that are terrible, but it's just the opposite. The places that have a lot of problems have problems because something makes it hard to fix the problems.

When I joined the team that scored a 1 on the Joel Test, it took months of campaigning just to get everyone to use version control.

I've never seen an environment go from "bad" to "good" and I'd be curious to know what that looks like and how it happens. Yossi Kreinen's thesis is that only management can fix broken situations. That might be true, but I'm not quite ready to believe it just yet, even though I don't have any evidence to the contrary.

Appendix B: other "how I became a programmer" stories

Kragen. Describes 27 years of learning to program. Heavy emphasis on conceptual phases of development (e.g., understanding how to use provided functions vs. understanding that you can write arbitrary functions)

Julia Evans. Started programming on a TI-83 in 2004. Dabbled in programming until college (2006-2011) and has been working as a professional programmer ever since. Some emphasis on the "journey" and how long it takes to improve.

Philip Guo. A non-traditional story of learning to program, which might be surprising if you know that Philip's career path was MIT -> Stanford -> Google.

Tavish Armstrong. 4th grade through college. Emphasis on particular technologies (e.g., LaTeX or Python).

Caitie McCaffrey. Started programming in AP computer science. Emphasis on how interests led to a career in programming.

Matt DeBoard. Spent 12 weeks learning Django with the help of a mentor. Emphasis on the fact that it's possible to become a programmer without programming background.

Kristina Chodorow. Started in college. Emphasis on alternatives (math, grad school).

Michael Bernstein. Story of learning Haskell over the course of years. Emphasis on how long it took to become even minimally proficient.

Thanks to Leah Hanson, Lindsey Kuper, Kelley Eskridge, Jeshua Smith, Tejas Sapre, Joe Wilder, Adrien Lamarque, Maggie Zhou, Lisa Neigut, Steve McCarthy, Darius Bacon, Kaylyn Gibilterra, Sarah Ransohoff, @HamsterRaging, Alex Allain, and "biktian" for comments/criticism/discussion.

Developer compensation has skyrocketed since the demise of the Google et al. wage-suppressing no-hire agreement, to the point where compensation rivals and maybe even exceeds compensation in traditionally remunerative fields like law, consulting, etc. In software, "senior" dev salary at a high-paying tech company is $350k/yr, where "senior" can mean "someone three years of out school" and it's not uncommon for someone who's considered a high performing engineer to make seven figures.

The fields have sharply bimodal income distributions. Are programmers in for the same fate? Let's see what data we can find. First, let's look at data from the National Association for Law Placement, which shows when legal salaries become bimodal.

Lawyers in 1991

Median salary is $40k, with the numbers slowly trickling off until about $90k. According to the BLS $90k in 1991 is worth $160k in 2016 dollars. That's a pretty generous starting salary.

Lawyers in 2000

By 2000, the distribution had become bimodal. The lower peak is about the same in nominal (non-inflation-adjusted) terms, putting it substantially lower in real (inflation-adjusted) terms, and there's an upper peak at around $125k, with almost everyone coming in under $130k. $130k in 2000 is $180k in 2016 dollars. The peak on the left has moved from roughly $30k in 1991 dollars to roughly $40k in 2000 dollars; both of those translate to roughly $55k in 2016 dollars. People in the right mode are doing better, while people in the left mode are doing about the same.

I won't belabor the point with more graphs, but if you look at more recent data, the middle area between the two modes has hollowed out, increasing the level of inequality within the field. As a profession, lawyers have gotten hit hard by automation, and in real terms, 95%-ile offers today aren't really better than they were in 2000. But 50%-ile and even 75%-ile offers are worse off due to the bimodal distribution.

Programmers in 2015

Enough about lawyers! What about programmers? Unfortunately, it's hard to get good data on this. Anecdotally, it sure seems to me like we're going down the same road. Unfortunately, almost all of the public data sources that are available, like H1B data, have salary numbers and not total compensation numbers. Since compensation at the the upper end is disproportionately bonus and stock, most data sets I can find don't capture what's going on.

One notable exception is the new grad compensation data recorded by Dan Zhang and Jesse Collins:

There's certainly a wide range here, and while it's technically bimodal, there isn't a huge gulf in the middle like you see in law and business. Note that this data is mostly bachelors grads with a few master's grads. PhD numbers, which sometimes go much higher, aren't included.

Do you know of a better (larger) source of data? This is from about 100 data points, members of the "Hackathon Hackers" Facebook group, in 2015. Dan and Jesse also have data from 2014, but it would be nice to get data over a wider timeframe and just plain more data. Also, this data is pretty clearly biased towards the high end — if you look at national averages for programmers at all levels of experience, the average comes in much lower than the average for new grads in this data set. The data here match the numbers I hear when we compete for people, but the population of "people negotiating offers at Microsoft" also isn't representative.

If we had more representative data it's possible that we'd see a lot more data points in the $40k to $60k range along with the data we have here, which would make the data look bimodal. It's also possible that we'd see a lot more points in the $40k to $60k range, many more in the $70k to $80k range, some more in the $90k+ range, etc., and we'd see a smooth drop-off instead of two distinct modes.

Stepping back from the meager data we have and looking at the circumstances, "should" programmer compensation be bimodal? Most other fields that have bimodal compensation have a very different compensation structure than we see in programming. For example, top law and consulting firms have an up-or-out structure, which is effectively a tournament, which distorts compensation and certainly makes it seem more likely that compensation is likely to end up being bimodal. Additionally, competitive firms pay the same rate to all 1st year employees, which they determine by matching whoever appears to be paying the most. For example, this year, Cravath announced that it would pay first-year associates $180k, and many other firms followed suit. Like most high-end firms, Cravath has a salary schedule that's entirely based on experience:

• 0 years: $180k
• 1 year: $190k
• 2 years: $210k
• 3 years: $235k
• 4 years: $260k
• 5 years: $280k
• 6 years: $300k
• 7 years: $315k

In software, compensation tends to be on a case-by-case basis, which makes it much less likely that we'll see a sharp peak the way we do in law. If I had to guess, I'd say that while the dispersion in programmer compensation is increasing, it's not bimodal, but I don't really have the right data set to conclusively say anything. Please point me to any data you have that's better.

Appendix A: please don't send me these

• H-1B: mostly salary only.
• Stack Overflow survey: salary only. Also, data is skewed by the heavy web focus of the survey —— I stopped doing the survey when none of their job descriptions matched anyone in my entire building, and I know other people who stopped for the same reason.
• Glassdoor: weirdly inconsistent about whether or not it includes stock compensation. Numbers for some companies seem to, but numbers for other companies don't.
• O'Reilly survey: salary focused.
• BLS: doesn't make fine-grained distribution available.
• IRS: they must have the data, but they're not sharing.
• IDG: only has averages.
• internal company data: too narrow
• compensation survey companies like PayScale: when I've talked to people from these companies, they acknowledge that they have very poor visibility into large company compensation, but that's what drives the upper end of the market (outside of finance).
• #talkpay on twitter: numbers skew low¹.

Appendix B: why are programmers well paid?

Since we have both programmer and lawyer compensation handy, let's examine that. Programming pays so well that it seems a bit absurd. If you look at other careers with similar compensation, there are multiple factors that act as barriers or disincentives to entry.

If you look at law, you have to win the prestige lottery and get into a top school, which will cost hundreds of thousands of dollars (while it's possible to get a full scholarship, a relatively small fraction of students at top schools are on full scholarships). Then you have to win the grades lottery and get good enough grades to get into a top firm. And then you have to continue winning tournaments to avoid getting kicked out, which requires sacrificing any semblance of a personal life. Consulting, investment banking, etc., are similar. Compensation appears to be proportional to the level of sacrifice (e.g., investment bankers are paid better, but work even longer hours than lawyers, private equity is somewhere between investment and banking and law in hours and compensation, etc.).

Medicine seems to be a bit better from the sacrifice standpoint because there's a cartel which limits entry into the field, but the combination of medical school and residency is still incredibly brutal compared to most jobs at places like Facebook and Google.

Programming also doesn't have a licensing body limiting the number of programmers, nor is there the same prestige filter where you have to go to a top school to get a well paying job. Sure, there are a lot of startups who basically only hire from MIT, Stanford, CMU, and a few other prestigious schools, and I see job ads like the following whenever I look at startups (the following is from a company that was advertising on Slate Star Codex for quite a long time):

Our team of 14 includes 6 MIT alumni, 3 ex-Googlers, 1 Wharton MBA, 1 MIT Master in CS, 1 CMU CS alum, and 1 "20 under 20" Thiel fellow. Candidates often remark we're the strongest team they've ever seen.

We’re not for everyone. We’re an enterprise SaaS company your mom will probably never hear of. We work really hard 6 days a week because we believe in the future of mobile and we want to win.

Prestige obsessed places exist. But, in programming, measuring people by markers of prestige seems to be a Silicon Valley startup thing and not a top-paying companies thing. Big companies, which pay a lot better than startups, don't filter people out by prestige nearly as often. Not only do you not need the right degree from the right school, you also don't need to have the right kind of degree, or any degree at all. Although it's getting rarer to not have a degree, I still meet new hires with no experience and either no degree or a degree in an unrelated field (like sociology or philosophy).

How is it possible that programmers are paid so well without these other barriers to entry that similarly remunerative fields have? One possibility is that we have a shortage of programmers. If that's the case, you'd expect more programmers to enter the field, bringing down compensation. CS enrollments have been at record levels recently, so this may already be happening. Another possibility is that programming is uniquely hard in some way, but that seems implausible to me. Programming doesn't seem inherently harder than electrical engineering or chemical engineering and it certainly hasn't gotten much harder over the past decade, but during that timeframe, programming has gone from having similar compensation to most engineering fields to paying much better. The last time I was negotiating with a EE company about offers, they remarked to me that their VPs don't make as much as I do, and I work at a software company that pays relatively poorly compared to its peers. There's no reason to believe that we won't see a flow of people from engineering fields into programming until compensation is balanced.

Another possibility is that U.S. immigration laws act as a protectionist barrier to prop up programmer compensation. It seems impossible for this to last (why shouldn't there by really valuable non-U.S. companies), but it does appear to be somewhat true for now. When I was at Google, one thing that was remarkable to me was that they'd pay you approximately the same thing in Washington or Colorado as they do Silicon Valley, but they'd pay you much less in London. Whenever one of these discussions comes up, people always bring up the "fact" that SV salaries aren't really as good as they sound because the cost of living is so high, but companies will not only match SV offers in Seattle, they'll match them in places like Pittsburgh. My best guess for why this happens is that someone in the Midwest can credibly threaten to move to SV and take a job at any company there, whereas someone in London can't². While we seem unlikely to loosen current immigration restrictions, our immigration restrictions have caused and continue to cause people who would otherwise have founded companies in the U.S. to found companies elsewhere. Given that the U.S. doesn't have a monopoly on people who found startups and that we do our best to keep people who want to found startups here out, it seems inevitable that there will eventually be Facebooks and Googles founded outside of the U.S. who compete for programmers the same way companies compete inside the U.S.

Another theory that I've heard a lot lately is that programmers at large companies get paid a lot because of the phenomenon described in Kremer's O-ring model. This model assumes that productivity is multiplicative. If your co-workers are better, you're more productive and produce more value. If that's the case, you expect a kind of assortive matching where you end up with high-skill firms that pay better, and low-skill firms that pay worse. This model has a kind of intuitive appeal to it, but it can't explain why programming compensation has higher dispersion than (for example) electrical engineering compensation. With the prevalence of open source, it's much easier to utilize the work of productive people outside your firm than in most fields. This model should be less true of programming than in most engineering fields, but the dispersion in compensation is higher.

A related theory that can't be correct for similar reasons is that high-paid software engineers are extra elite, the best of the best, and are simply paid more because they're productive. If you look at how many programmers the BLS says exist in the US (on the order of a few million) and how many engineers high-paying tech companies employ in the U.S. (on the order of a couple or a few hundred thousand), high-paying software compnaies literally can't consist of the top 1%. Even if their filters were perfect (as opposed to the complete joke that they're widely regarded to be), they couldn't be better than 90%-ile. Realistically, it's more likely that the median programmer at a high-paying tech company is a bit above 50%-ile.

The most common theory I've heard is that "software is eating the world". The theory goes: of course programmers get paid a lot and will continue to get paid a lot because software is important and only becoming more important. Despite being the most commonly stated theory I've heard, this seems nonsensical if you compare other fields. You could've said this about microprocessor design in the late 90s as well as fiber optics. Those fields are both more important today than they were in the 90s, not only is there more demand for processing power and bandwidth than ever before, demand for software is actually dependent on those. And yet, the optics engineering job market still hasn't recovered from the dot com crash and the microprocessor design engineer market, after recovering, still pays experienced PhDs less than a CS new grad at Facebook.

Furthermore, any argument for high programmer pay that relies on some inherent property of market conditions, the economy at large, the impact of programming, etc., seems like it cannot be correct if you look at what's actually driven up programmer pay. FB declined to participate in Google/Apple wage fixing agreement that became basically industry wide, which mean that FB was outpaying other major tech companies. When the wage-fixing agreement was lifted, other companies "had to" come close to matching FB compensation to avoid losing people both to FB and to each other. When they did that, FB kept raising the bar on compensation and compensation kept getting better. [2022 update] This can most clearly be seen with changes to benefits and pay structure, where FB would make a change, Google would follow suit immediately, and other companies would pick up the change later, as when FB removed vesting cliffs and Google did the same within weeks and the change trickled out across the industry. There are companies that were paying programmers as well or better than FB, like Netflix and a variety of finance companies, but major tech companies tended to not match offers from those places because they were too small to hire away enough programmers to be concerning, but FB is large and hires enough to be a concern to Google, which matches FB and combined, are large enough to be a concern to other major tech companies.

Because the mechanism for compensation increases has been arbitrary (FB could not exist, or the person who's in total control of FB, Zuckerberg, could decide on different compensation policy), it's quite arbitrary that programmer pay is as good as it is.

In conclusion, high programmer pay seems like a mystery to me and would love to hear a compelling theory for why programming "should" pay more than other similar fields, or why it should pay as much as fields that have much higher barriers to entry.

Update

Eric Roberts has observed that it takes a long time for CS enrollments to recover after a downturn, leading to a large deficit in the number of people with CS degrees vs. demand.

The 2001 bubble bursting caused a severe drop in CS enrollment. CS enrollment didn't hit its previous peak again until 2014, and if you fit the graph and extrapolate against the peaks, it took another year or two for enrollments to hit the historical trend. If we didn't have any data, it wouldn't be surprising to find that there's a five year delay. Of the people who graduate in four years (as opposed to five or more), most aren't going to change their major after mid or late sophmore year, so that's already two to three years of delay right there. And after a downturn, it takes some time to recover, so we'd expect at least another two to three years. Roberts makes a case that the additional latency came from a number of other factors including the fear that even though things looked ok, jobs would be outsourced soon and a slow response by colleges.

Dan Wang has noted that, according to the SO survey, 3/4 of developers have a BS degree (or higher). If it's statistically "hard" to get a high-paying job without a CS degree and there's over a decade hangover from the 2001 downturn, that could explain why programmer compensation is so high. Of course, most of us know people in the industry without a degree, but it seems to be harder to find an entry-level position without a credential.

It's not clear what this means for the future. Even if the lack of candidates with the appropriate credential is a major driver in programmer compensation, it's unclear what the record CS enrollments over the past few years means for future compensation. It's possible that record enrollments mean that we should expect compensation to come back down to the levels we see in other fields that require similar skills, like electrical engineering. It's also possible that enrollment continues to lag behind demand by a decade and that record enrollments are just keeping pace with demand from a decade ago, in which case we might expect elevated compensation to persist (as long as other factors, like hiring outside of the U.S., don't influence things too much). Since there's so much latency, another possibility is that enrollment has or will overshoot demand and we should expect compensaton programmer compensation to decline. And it's not even clear that the Roberts paper makes sense as an explanation for high current comp because Roberts also found a huge capacity crunch in the 80s and, while some programmers were paid very well, the fraction of programmers who were paid "very well" seems to have been much smaller than it is today. Google alone employs 30k engineers. If 20k are programmers in the U.S. and the estimate that there are 3 million programmers in the U.S., Google alone employs 0.6% of programmers in the U.S. If you add in the other large companies that are known to pay competitively (Amazon, Facebook, etc.), that's a significant fraction of all programmers in the U.S., which I believe is quite different from the situation in the 80s.

The most common response I've gotten to this post is that we should expect programmers to be well-paid because software is everywhere and there will be at least as much software in the future. This exact same line of reasoning could apply to electrical engineering, which is more fundamental than software, in that software requires hardware, and yet electrical engineering comp isn't in the same league as programmer comp. Highly paid programmers couldn't get their work done without microprocessors, and there are more processors sold than ever before, but the comp packages for a "senior" person at places like Intel and Qualcomm aren't even within a factor of two as at Google or Facebook. You could also make a similar argument for people who work on water and sewage systems, but those folks also don't seem compensation that's in the same range as programmers either. Any argument of the form, "the price for X is high because X is important" implicitly assumes that there's some force constraining the supply of X. The claim that "X is important" or "we need a lot of X" is missing half the story. Another problem with claims like "X is important" or "X is hard" is that these statements don't seem any less true of industries that pay much less. If your explanation of why programmers are well paid is just as true of any "classical" engineering discipline, you need some explanation of why those other fields shouldn't be as well paid.

The second most common comment that I hear is that, of course programmers are well paid, software companies are worth so much, which makes it inevitable. But there's nothing inevitable about workers actually being well compensated because a company is profitable. Someone who made this argument sent me a link to this list of the most profitable companies per employee. The list has some software companies that pay quite well, like Alphabet (Google) and Facebook, but we also see hardware companies like Qualcomm, Cisco, TSMC (and arguably SoftBank now that they've acquired ARM) that don't even pay as well software companies that don't turn a profit or that barely make money and have no path to being wildly profitable in the future. Moreover, the compensation at the software companies that are listed isn't very strongly related to their profit per employee.

To take a specific example that I'm familiar with because I grew up in Madison, the execs at Epic Systems have built a company that's generated so much wealth that its founder has an estimated net worth of $3.6 billion, which is much more than all but the most successful founders in tech. But line engineers at Epic are paid significantly less than engineers at tech companies that compete with SV for talent, even tech companies that have never made any money. What is it about some software companies that make a similar amount of money that prevents them from funneling virtually all of the wealth they generate up to the top? The typical answer to this cost of living, but as we've seen, that makes even less sense than usual in this case since Google has an office in the same city as Epic, and Google pays well over double what Epic does for typical dev. If there were some kind of simple cost of living adjustment, you'd expect Google to pay less in Madison than in Toronto or London, but it seems to be the other way around. This isn't unique to Madison — just for example, you can find a number of successful software companies in Austin that pay roughly half what Amazon and Facebook pay in the same city, where upper management does very well for themselves and line engineers make a fine living, but nowhere near as much as they'd make if they moved to a company like Amazon or Facebook.

The thing all of these theories have in common is that they apply to other fields as well, so they cannot be, as stated, the reason programmers are better paid than people in these other fields. Someone could argue that programming has a unique combination of many of these or that one these reasons should be expected to apply much more strongly than to any other field, but I haven't seen anyone make that case. Instead, people just make obviously bogus statements like "programming is really hard" (which is only valid as a reason, in this discussion, if literally the hardest field in existence and much harder than other engineering fields).

I can't think of a single large software company that doesn't regularly draw internet comments of the form “What do all the employees do? I could build their product myself.” Benjamin Pollack and Jeff Atwood called out people who do that with Stack Overflow. But Stack Overflow is relatively obviously lean, so the general response is something like “oh, sure maybe Stack Overflow is lean, but FooCorp must really be bloated”. And since most people have relatively little visibility into FooCorp, for any given value of FooCorp, that sounds like a plausible statement. After all, what product could possible require hundreds, or even thousands of engineers?

A few years ago, in the wake of the rapgenius SEO controversy, a number of folks called for someone to write a better Google. Alex Clemmer responded that maybe building a better Google is a non-trivial problem. Considering how much of Google's $500B market cap comes from search, and how much money has been spent by tens (hundreds?) of competitors in an attempt to capture some of that value, it seems plausible to me that search isn't a trivial problem. But in the comments on Alex's posts, multiple people respond and say that Lucene basically does the same thing Google does and that Lucene is poised to surpass Google's capabilities in the next few years. It's been long enough since then that we can look back and say that Lucene hasn't improved so much that Google is in danger from a startup that puts together a Lucene cluster. If anything, the cost of creating a viable competitor to Google search has gone up.

For making a viable Google competitor, I believe that ranking is a harder problem than indexing, but even if we just look at indexing, there are individual domains that contain on the order of one trillion pages we might want to index (like Twitter) and I'd guess that we can find on the order a trillion domains. If you try to configure any off-the-shelf search index to hold an index of some number of trillions of items to handle a load of, say, 1/100th Google's load, with a latency budget of, say, 100ms (most of the latency should be for ranking, not indexing), I think you'll find that this isn't trivial. And if you use Google to search Twitter, you can observe that, at least for select users or tweets, Google indexes Twitter quickly enough that it's basically real-time from the standpoint of users. Anyone who's tried to do real-time indexing with Lucene on a large corpus under high load will also find this to be non-trivial. You might say that this isn't totally fair since it's possible to find tweets that aren't indexed by major search engines, but if you want to make a call on what to index or not, well, that's also a problem that's non trivial in the general case. And we're only talking about indexing here, indexing is one of the easier parts of building a search engine.

Businesses that actually care about turning a profit will spend a lot of time (hence, a lot of engineers) working on optimizing systems, even if an MVP for the system could have been built in a weekend. There's also a wide body of research that's found that decreasing latency has a significant effect on revenue over a pretty wide range of latencies for some businesses. Increasing performance also has the benefit of reducing costs. Businesses should keep adding engineers to work on optimization until the cost of adding an engineer equals the revenue gain plus the cost savings at the margin. This is often many more engineers than people realize.

And that's just performance. Features also matter: when I talk to engineers working on basically any product at any company, they'll often find that there are seemingly trivial individual features that can add integer percentage points to revenue. Just as with performance, people underestimate how many engineers you can add to a product before engineers stop paying for themselves.

Additionally, features are often much more complex than outsiders realize. If we look at search, how do we make sure that different forms of dates and phone numbers give the same results? How about internationalization? Each language has unique quirks that have to be accounted for. In french, “l'foo” should often match “un foo” and vice versa, but American search engines from the 90s didn't actually handle that correctly. How about tokenizing Chinese queries, where words don't have spaces between them, and sentences don't have unique tokenizations? How about Japanese, where queries can easily contain four different alphabets? How about handling Arabic, which is mostly read right-to-left, except for the bits that are read left-to-right? And that's not even the most complicated part of handling Arabic! It's fine to ignore this stuff for a weekend-project MVP, but ignoring it in a real business means ignoring the majority of the market! Some of these are handled ok by open source projects, but many of the problems involve open research problems.

There's also security! If you don't “bloat” your company by hiring security people, you'll end up like hotmail or yahoo, where your product is better known for how often it's hacked than for any of its other features.

Everything we've looked at so far is a technical problem. Compared to organizational problems, technical problems are straightforward. Distributed systems are considered hard because real systems might drop something like 0.1% of messages, corrupt an even smaller percentage of messages, and see latencies in the microsecond to millisecond range. When I talk to higher-ups and compare what they think they're saying to what my coworkers think they're saying, I find that the rate of lost messages is well over 50%, every message gets corrupted, and latency can be months or years¹. When people imagine how long it should take to build something, they're often imagining a team that works perfectly and spends 100% of its time coding. But that's impossible to scale up. The question isn't whether or not there will inefficiencies, but how much inefficiency. A company that could eliminate organizational inefficiency would be a larger innovation than any tech startup, ever. But when doing the math on how many employees a company “should” have, people usually assume that the company is an efficient organization.

This post happens to use search as an example because I ran across some people who claimed that Lucene was going to surpass Google's capabilities any day now, but there's nothing about this post that's unique to search. If you talk to people in almost any field, you'll hear stories about how people wildly underestimate the complexity of the problems in the field. The point here isn't that it would be impossible for a small team to build something better than Google search. It's entirely plausible that someone will have an innovation as great as PageRank, and that a small team could turn that into a viable company. But once that company is past the VC-funded hyper growth phase and wants to maximize its profits, it will end up with a multi-thousand person platforms org, just like Google's, unless the company wants to leave hundreds of millions or billions of dollars a year on the table due to hardware and software inefficiency. And the company will want to handle languages like Thai, Arabic, Chinese, and Japanese, each of which is non-trivial. And the company will want to have relatively good security. And there are the hundreds of little features that users don't even realize that are there, each of which provides a noticeable increase in revenue. It's "obvious" that companies should outsource their billing, except that when you talk to companies that handle their own billing, they can point to individual features that increase conversion by single or double digit percentages that they can't get from Stripe or Braintree. That fifty person billing team is totally worth it, beyond a certain size. And then there's sales, which most engineers don't even think of²; the exact same line of reasoning that applies to optimization also applies to sales -- as long as marginal benefit of adding another salesperson exceeds the cost, you should expect the company to keep adding salespeople, which can often result in a sales force that's larger than the engineering team. There's also research which, almost by definition, involves a lot of bets that don't pan out!

It's not that all of those things are necessary to run a service at all; it's that almost every large service is leaving money on the table if they don't seriously address those things. This reminds me of a common fallacy we see in unreliable systems, where people build the happy path with the idea that the happy path is the “real” work, and that error handling can be tacked on later. For reliable systems, error handling is more work than the happy path. The same thing is true for large services -- all of this stuff that people don't think of as “real” work is more work than the core service³.

Correction

I often make minor tweaks and add new information without comment, but the original version of this post had an error and removing the error was a large enough change that I believe it's worth pointing out the change. I had a back of the envelope calculation on the cost of indexing the web with Lucene, but the numbers were based on benchmarks results from some papers and comments from people who work on a commercial search engine. When I tried to reproduce the results from the papers, I found that it was trivial to get orders of magnitude better performance than reported in one paper and when I tried to track down the underlying source for the comments by people who work on a commercial search engine, I found that there was no experimental evidence underlying the comments, so I removed the example.

I'm experimenting with writing blog posts stream-of-consciousness, without much editing. Both this post and my last post were written that way. Let me know what you think of these posts relative to my “normal” posts!

Thanks to Leah Hanson, Joel Wilder, Kay Rhodes, Heath Borders, Kris Shamloo, Justin Blank, and Ivar Refsdal for corrections.

Joel Spolsky has a classic blog post on "Finding Great Developers" where he popularized the meme that great developers are impossible to find, a corollary of which is that if you can find someone, they're not great. Joel writes,

The great software developers, indeed, the best people in every field, are quite simply never on the market.

The average great software developer will apply for, total, maybe, four jobs in their entire career.

...

If you're lucky, if you're really lucky, they show up on the open job market once, when, say, their spouse decides to accept a medical internship in Anchorage and they actually send their resume out to what they think are the few places they'd like to work at in Anchorage.

But for the most part, great developers (and this is almost a tautology) are, uh, great, (ok, it is a tautology), and, usually, prospective employers recognize their greatness quickly, which means, basically, they get to work wherever they want, so they honestly don't send out a lot of resumes or apply for a lot of jobs.

Does this sound like the kind of person you want to hire? It should.The corollary of that rule--the rule that the great people are never on the market--is that the bad people--the seriously unqualified--are on the market quite a lot. They get fired all the time, because they can't do their job. Their companies fail--sometimes because any company that would hire them would probably also hire a lot of unqualified programmers, so it all adds up to failure--but sometimes because they actually are so unqualified that they ruined the company. Yep, it happens.

These morbidly unqualified people rarely get jobs, thankfully, but they do keep applying, and when they apply, they go to Monster.com and check off 300 or 1000 jobs at once trying to win the lottery.

Astute readers, I expect, will point out that I'm leaving out the largest group yet, the solid, competent people. They're on the market more than the great people, but less than the incompetent, and all in all they will show up in small numbers in your 1000 resume pile, but for the most part, almost every hiring manager in Palo Alto right now with 1000 resumes on their desk has the same exact set of 970 resumes from the same minority of 970 incompetent people that are applying for every job in Palo Alto, and probably will be for life, and only 30 resumes even worth considering, of which maybe, rarely, one is a great programmer. OK, maybe not even one.

Joel's claim is basically that "great" developers won't have that many jobs compared to "bad" developers because companies will try to keep "great" developers. Joel also posits that companies can recognize prospective "great" developers easily. But these two statements are hard to reconcile. If it's so easy to identify prospective "great" developers, why not try to recruit them? You could just as easily make the case that "great" developers are overrepresented in the market because they have better opportunities and it's the "bad" developers who will cling to their jobs. This kind of adverse selection is common in companies that are declining; I saw that in my intern cohort at IBM¹, among other places.

Should "good" developers be overrepresented in the market or underrepresented? If we listen to the anecdotal griping about hiring, we might ask if the market for developers is a market for lemons. This idea goes back to Akerlof's Nobel prize winning 1970 paper, "The Market for 'Lemons': Quality Uncertainty and the Market Mechanism". Akerlof takes used car sales as an example, splitting the market into good used cars and bad used cars (bad cars are called "lemons"). If there's no way to distinguish between good cars and lemons, good cars and lemons will sell for the same price. Since buyers can't distinguish between good cars and bad cars, the price they're willing to pay is based on the quality of the average in the market. Since owners know if their car is a lemon or not, owners of non-lemons won't sell because the average price is driven down by the existence of lemons. This results in a feedback loop which causes lemons to be the only thing available.

This model is certainly different from Joel's model. Joel's model assumes that "great" developers are sticky -- that they stay at each job for a long time. This comes from two assumptions; first, that it's easy for prospective employers to identify who's "great", and second, that once someone is identified as "great", their current employer will do anything to keep them (as in the market for lemons). But the first assumption alone is enough to prevent the developer job market from being a market for lemons. If you can tell that a potential employee is great, you can simply go and offer them twice as much as they're currently making (something that I've seen actually happen). You need an information asymmetry to create a market for lemons, and Joel posits that there's no information asymmetry.

If we put aside Joel's argument and look at the job market, there's incomplete information, but both current and prospective employers have incomplete information, and whose information is better varies widely. It's actually quite common for prospective employers to have better information than current employers!

Just for example, there's someone I've worked with, let's call him Bob, who's saved two different projects by doing the grunt work necessary to keep the project from totally imploding. The projects were both declared successes, promotions went out, they did a big PR blitz which involves seeding articles in all the usual suspects; Wired, Fortune, and so on and so forth. That's worked out great for the people who are good at taking credit for things, but it hasn't worked out so well for Bob. In fact, someone else I've worked with recently mentioned to me that management keeps asking him why Bob takes so long to do simple tasks. The answer is that Bob's busy making sure the services he works on don't have global outages when they launch, but that's not the kind of thing you get credit for in Bob's org. The result of that is that Bob has a network who knows that he's great, which makes it easy for him to get a job anywhere else at market rate. But his management chain has no idea, and based on what I've seen of offers today, they're paying him about half what he could make elsewhere. There's no shortage of cases where information transfer inside a company is so poor that external management has a better view of someone's productivity than internal management. I have one particular example in mind, but if I just think of the Bob archetype, off the top of my head, I know of four people who are currently in similar situations. It helps that I currently work at a company that's notorious for being dysfunctional in this exact way, but this happens everywhere. When I worked at a small company, we regularly hired great engineers from big companies that were too clueless to know what kind of talent they had.

Another problem with the idea that "great" developers are sticky is that this assumes that companies are capable of creating groups that developers want to work for on demand. This is usually not the case. Just for example, I once joined a team where the TL was pretty strongly against using version control or having tests. As a result of those (and other) practices, it took five devs one year to produce 10k lines of kinda-sorta working code for a straightforward problem. Additionally, it was a pressure cooker where people were expected to put in 80+ hour weeks, where the PM would shame people into putting in longer hours. Within a year, three of the seven people who were on the team when I joined had left; two of them went to different companies. The company didn't want to lose those two people, but it wasn't capable of creating an environment that would keep them.

Around when I joined that team, a friend of mine joined a really great team. They do work that materially impacts the world, they have room for freedom and creativity, a large component of their jobs involves learning new and interesting things, and so on and so forth. Whenever I heard about someone who was looking for work, I'd forward them that team. That team is now full for the foreseeable future because everyone whose network included that team forwarded people into that team. But if you look at the team that lost three out of seven people in a year, that team is hiring. A lot. The result of this dynamic is that, as a dev, if you join a random team, you're overwhelmingly likely to join a team that has a lot of churn. Additionally, if you know of a good team, it's likely to be full.

Joel's model implicitly assumes that, proportionally, there are many more dysfunctional developers than dysfunctional work environments.

At the last conference I attended, I asked most people I met two questions:

1. Do you know of any companies that aren't highly dysfunctional?
2. Do you know of any particular teams that are great and are hiring?

Not one single person told me that their company meets the criteria in (1). A few people suggested that, maybe, Dropbox is ok, or that, maybe, Jane Street is ok, but the answers were of the form "I know a few people there and I haven't heard any terrible horror stories yet, plus I sometimes hear good stories", not "that company is great and you should definitely work there". Most people said that they didn't know of any companies that weren't a total mess.

A few people had suggestions for (2), but the most common answer was something like "LOL no, if I knew that I'd go work there". The second most common answer was of the form "I know some people on the Google Brain team and it sounds great". There are a few teams that are well known for being great places to work, but the fact that they're so few and far between that it's basically impossible to get a job on one of those teams. A few people knew of actual teams that they'd strongly recommend who were hiring, but that was rare. Much rarer than finding a developer who I'd want to work with who would consider moving. If I flipped the question around and asked if they knew of any good developers who were looking for work, the answer was usually "yes"².

Another problem with the idea that "great" developers are impossible to find because they join companies and then stick is that developers (and companies) aren't immutable. Because I've been lucky enough to work in environments that allow people to really flourish, I've seen a lot of people go from unremarkable to amazing. Because most companies invest pretty much nothing in helping people, you can do really well here without investing much effort.

On the flip side, I've seen entire teams of devs go on the market because their environment changed. Just for example, I used to know a lot of people who worked at company X under Marc Yun. It was the kind of place that has low attrition because people really enjoy working there. And then Marc left. Over the next two years, literally everyone I knew who worked there left. This one change both created a lemon in the searching-for-a-team job market and put a bunch of good developers on the market. This kind of thing happens all the time, even more now than in the past because of today's acquisition-heavy environment.

Is developer hiring a market for lemons? Well, it depends on what you mean by that. Both developers and hiring managers have incomplete information. It's not obvious if having a market for lemons in one direction makes the other direction better or worse. The fact that joining a new team is uncertain makes developers less likely to leave existing teams, which makes it harder to hire developers. But the fact that developers often join teams which they dislike makes it easier to hire developers. What's the net effect of that? I have no idea.

From where I'm standing, it seems really hard to find a good manager/team, and I don't know of any replicable strategy for doing so; I have a lot of sympathy for people who can't find a good fit because I get how hard that is. But I have seen replicable strategies for hiring, so I don't have nearly as much sympathy for hiring managers who complain that hiring "great" developers is impossible.

When a hiring manager complains about hiring, in every single case I've seen so far, the hiring manager has one of the following problems:

1. They pay too little. The last time I went looking for work, I found a 6x difference in compensation between companies who might hire me in the same geographic region. Basically all of the companies thought that they were competitive, even when they were at the bottom end of the range. I don't know what it is, but companies always seem to think that they pay well, even when they're not even close to being in the right range. Almost everyone I talk to tells me that they pay as much as any reasonable company. Sure, there are some companies out there that pay a bit more, but they're overpaying! You can actually see this if you read Joel's writing -- back when he wrote the post I'm quoting above, he talked about how well Fog Creek paid. A couple years later, he complained that Google was overpaying for college kids with no experience, and more recently he's pretty much said that you don't want to work at companies that pay well.

2. They pass on good or even "great" developers³. Earlier, I claimed that I knew lots of good developers who are looking for work. You might ask, if there are so many good developers looking for work, why's it so hard to find them? Joel claims that out of a 1000 resumes, maybe 30 people will be "solid" and 970 will be "incompetent". It seems to me it's more like 400 will be solid and 20 will be really good. It's just that almost everyone uses the same filters, so everyone ends up fighting over the 30 people who they think are solid. When people do randomized trials on what actually causes resumes to get filtered out, it often turns out that traits that are tangentially related or unrelated to job performance make huge differences. For example, in this study of law firm recruiting, the authors found that a combination of being male and having "high-class" signifiers on the resume (sailing, polo, and classical music instead of track and field, pick-up soccer, and country music) with no other changes caused a 4x increase in interview invites.

   The first company I worked at, Centaur, had an onsite interview process that was less stringent than the phone screen at places like Google and Facebook. If you listen to people like Joel, you'd think that Centaur was full of bozos, but after over a decade in industry (including time at Google), Centaur had the best mean and median level of developer productivity of any place I've worked.

   Matasano famously solved their hiring problem by using a different set of filters and getting a different set of people. Despite the resounding success of their strategy, pretty much everyone insists on sticking with the standard strategy of picking people with brand name pedigrees and running basically the same interview process as everyone else, bidding up the price of folks who are trendy and ignoring everyone else.

   If I look at developers I know who are in high-demand today, a large fraction of them went through a multi-year period where they were underemployed and practically begging for interesting work. These people are very easy to hire if you can find them.

3. They're trying to hire for some combination of rare skills. Right now, if you're trying to hire for someone with experience in deep learning and, well, anything else, you're going to have a bad time.

4. They're much more dysfunctional than they realize. I know one hiring manager who complains about how hard it is to hire. What he doesn't realize is that literally everyone on his team is bitterly unhappy and a significant fraction of his team gives anti-referrals to friends and tells them to stay away.

   That's an extreme case, but it's quite common to see a VP or founder baffled by why hiring is so hard when employees consider the place to be mediocre or even bad.

Of these problems, (1), low pay, is both the most common and the simplest to fix.

In the past few years, Oracle and Alibaba have spun up new cloud computing groups in Seattle. This is a relatively competitive area, and both companies have reputations that work against them when hiring⁴. If you believe the complaints about how hard it is to hire, you wouldn't think one company, let alone two, could spin up entire cloud teams in Seattle. Both companies solved the problem by paying substantially more than their competitors were offering for people with similar experience. Alibaba became known for such generous offers that when I was negotiating my offer from Microsoft, MS told me that they'd match an offer from any company except Alibaba. I believe Oracle and Alibaba have hired hundreds of engineers over the past few years.

Most companies don't need to hire anywhere near a hundreds of people; they can pay competitively without hiring so many developers that the entire market moves upwards, but they still refuse to do so, while complaining about how hard it is to hire.

(2), filtering out good potential employees, seems like the modern version of "no one ever got fired for hiring IBM". If you hire someone with a trendy background who's good at traditional coding interviews and they don't work out, who could blame you? And no one's going to notice all the people you missed out on. Like (1), this is something that almost everyone thinks they do well and they'll say things like "we'd have to lower our bar to hire more people, and no one wants that". But I've never worked at a place that doesn't filter out a lot of people who end up doing great work elsewhere. I've tried to get underrated programmers⁵ hired at places I've worked, and I've literally never succeeded in getting one hired. Once, someone I failed to get hired managed to get a job at Google after something like four years being underemployed (and is a star there). That guy then got me hired at Google. Not hiring that guy didn't only cost them my brilliant friend, it eventually cost them me!

BTW, this illustrates a problem with Joel's idea that "great" devs never apply for jobs. There's often a long time period where a "great" dev has an extremely hard time getting hired, even through their network who knows that they're great, because they don't look like what people think "great" developers look like. Additionally, Google, which has heavily studied which hiring channels give good results, has found that referrals and internal recommendations don't actually generate much signal. While people will refer "great" devs, they'll also refer terrible ones. The referral bonus scheme that most companies set up skews incentives in a way that makes referrals worse than you might expect. Because of this and other problems, many companies don't weight referrals particularly heavily, and "great" developers still go through the normal hiring process, just like everyone else.

(3), needing a weird combination of skills, can be solved by hiring people with half or a third of the expertise you need and training people. People don't seem to need much convincing on this one, and I see this happen all the time.

(4), dysfunction seems hard to fix. If I knew how to do that, I'd be manager.

As a dev, it seems to me that teams I know of that are actually good environments that pay well have no problems hiring, and that teams that have trouble hiring can pretty easily solve that problem. But I'm biased. I'm not a hiring manager. There's probably some hiring manager out there thinking: "every developer I know who complains that it's hard to find a good team has one of these four obvious problems; if only my problems were that easy to solve!"

Thanks to Leah Hanson, David Turner, Tim Abbott, Vaibhav Sagar, Victor Felder, Ezekiel Smithburg, Juliano Bortolozzo Solanho, Stephen Tu, Pierre-Yves Baccou, Jorge Montero, Alkin Kaz, Ben Kuhn, and Lindsey Kuper for comments and corrections.

If you liked this post, you'd probably enjoy this other post on the bogosity of claims that there can't possibly be discrimination in tech hiring.

There are a lot of “12 CS books every programmer must read” lists floating around out there. That's nonsense. The field is too broad for almost any topic to be required reading for all programmers, and even if a topic is that important, people's learning preferences differ too much for any book on that topic to be the best book on the topic for all people.

This is a list of topics and books where I've read the book, am familiar enough with the topic to say what you might get out of learning more about the topic, and have read other books and can say why you'd want to read one book over another.

Algorithms / Data Structures / Complexity

Why should you care? Well, there's the pragmatic argument: even if you never use this stuff in your job, most of the best paying companies will quiz you on this stuff in interviews. On the non-bullshit side of things, I find algorithms to be useful in the same way I find math to be useful. The probability of any particular algorithm being useful for any particular problem is low, but having a general picture of what kinds of problems are solved problems, what kinds of problems are intractable, and when approximations will be effective, is often useful.

McDowell; Cracking the Coding Interview

Some problems and solutions, with explanations, matching the level of questions you see in entry-level interviews at Google, Facebook, Microsoft, etc. I usually recommend this book to people who want to pass interviews but not really learn about algorithms. It has just enough to get by, but doesn't really teach you the why behind anything. If you want to actually learn about algorithms and data structures, see below.

Dasgupta, Papadimitriou, and Vazirani; Algorithms

Everything about this book seems perfect to me. It breaks up algorithms into classes (e.g., divide and conquer or greedy), and teaches you how to recognize what kind of algorithm should be used to solve a particular problem. It has a good selection of topics for an intro book, it's the right length to read over a few weekends, and it has exercises that are appropriate for an intro book. Additionally, it has sub-questions in the middle of chapters to make you reflect on non-obvious ideas to make sure you don't miss anything.

I know some folks don't like it because it's relatively math-y/proof focused. If that's you, you'll probably prefer Skiena.

Skiena; The Algorithm Design Manual

The longer, more comprehensive, more practical, less math-y version of Dasgupta. It's similar in that it attempts to teach you how to identify problems, use the correct algorithm, and give a clear explanation of the algorithm. Book is well motivated with “war stories” that show the impact of algorithms in real world programming.

CLRS; Introduction to Algorithms

This book somehow manages to make it into half of these “N books all programmers must read” lists despite being so comprehensive and rigorous that almost no practitioners actually read the entire thing. It's great as a textbook for an algorithms class, where you get a selection of topics. As a class textbook, it's nice bonus that it has exercises that are hard enough that they can be used for graduate level classes (about half the exercises from my grad level algorithms class were pulled from CLRS, and the other half were from Kleinberg & Tardos), but this is wildly impractical as a standalone introduction for most people.

Just for example, there's an entire chapter on Van Emde Boas trees. They're really neat -- it's a little surprising that a balanced-tree-like structure with O(lg lg n) insert, delete, as well as find, successor, and predecessor is possible, but a first introduction to algorithms shouldn't include Van Emde Boas trees.

Kleinberg & Tardos; Algorithm Design

Same comments as for CLRS -- it's widely recommended as an introductory book even though it doesn't make sense as an introductory book. Personally, I found the exposition in Kleinberg to be much easier to follow than in CLRS, but plenty of people find the opposite.

Demaine; Advanced Data Structures

This is a set of lectures and notes and not a book, but if you want a coherent (but not intractably comprehensive) set of material on data structures that you're unlikely to see in most undergraduate courses, this is great. The notes aren't designed to be standalone, so you'll want to watch the videos if you haven't already seen this material.

Okasaki; Purely Functional Data Structures

Fun to work through, but, unlike the other algorithms and data structures books, I've yet to be able to apply anything from this book to a problem domain where performance really matters.

For a couple years after I read this, when someone would tell me that it's not that hard to reason about the performance of purely functional lazy data structures, I'd ask them about part of a proof that stumped me in this book. I'm not talking about some obscure super hard exercise, either. I'm talking about something that's in the main body of the text that was considered too obvious to the author to explain. No one could explain it. Reasoning about this kind of thing is harder than people often claim.

Dominus; Higher Order Perl

A gentle introduction to functional programming that happens to use Perl. You could probably work through this book just as easily in Python or Ruby.

If you keep up with what's trendy, this book might seem a bit dated today, but only because so many of the ideas have become mainstream. If you're wondering why you should care about this "functional programming" thing people keep talking about, and some of the slogans you hear don't speak to you or are even off-putting (types are propositions, it's great because it's math, etc.), give this book a chance.

Levitin; Algorithms

I ordered this off amazon after seeing these two blurbs: “Other learning-enhancement features include chapter summaries, hints to the exercises, and a detailed solution manual.” and “Student learning is further supported by exercise hints and chapter summaries.” One of these blurbs is even printed on the book itself, but after getting the book, the only self-study resources I could find were some yahoo answers posts asking where you could find hints or solutions.

I ended up picking up Dasgupta instead, which was available off an author's website for free.

Mitzenmacher & Upfal; Probability and Computing: Randomized Algorithms and Probabilistic Analysis

I've probably gotten more mileage out of this than out of any other algorithms book. A lot of randomized algorithms are trivial to port to other applications and can simplify things a lot.

The text has enough of an intro to probability that you don't need to have any probability background. Also, the material on tails bounds (e.g., Chernoff bounds) is useful for a lot of CS theory proofs and isn't covered in the intro probability texts I've seen.

Sipser; Introduction to the Theory of Computation

Classic intro to theory of computation. Turing machines, etc. Proofs are often given at an intuitive, “proof sketch”, level of detail. A lot of important results (e.g, Rice's Theorem) are pushed into the exercises, so you really have to do the key exercises. Unfortunately, most of the key exercises don't have solutions, so you can't check your work.

For something with a more modern topic selection, maybe see Aurora & Barak.

Bernhardt; Computation

Covers a few theory of computation highlights. The explanations are delightful and I've watched some of the videos more than once just to watch Bernhardt explain things. Targeted at a general programmer audience with no background in CS.

Kearns & Vazirani; An Introduction to Computational Learning Theory

Classic, but dated and riddled with errors, with no errata available. When I wanted to learn this material, I ended up cobbling together notes from a couple of courses, one by Klivans and one by Blum.

Operating Systems

Why should you care? Having a bit of knowledge about operating systems can save days or week of debugging time. This is a regular theme on Julia Evans's blog, and I've found the same thing to be true of my experience. I'm hard pressed to think of anyone who builds practical systems and knows a bit about operating systems who hasn't found their operating systems knowledge to be a time saver. However, there's a bias in who reads operating systems books -- it tends to be people who do related work! It's possible you won't get the same thing out of reading these if you do really high-level stuff.

Silberchatz, Galvin, and Gagne; Operating System Concepts

This was what we used at Wisconsin before the comet book became standard. I guess it's ok. It covers concepts at a high level and hits the major points, but it's lacking in technical depth, details on how things work, advanced topics, and clear exposition.

Cox, Kasshoek, and Morris; xv6

This book is great! It explains how you can actually implement things in a real system, and it comes with its own implementation of an OS that you can play with. By design, the authors favor simple implementations over optimized ones, so the algorithms and data structures used are often quite different than what you see in production systems.

This book goes well when paired with a book that talks about how more modern operating systems work, like Love's Linux Kernel Development or Russinovich's Windows Internals.

Arpaci-Dusseau and Arpaci-Dusseau; Operating Systems: Three Easy Pieces

Nice explanation of a variety of OS topics. Goes into much more detail than any other intro OS book I know of. For example, the chapters on file systems describe the details of multiple, real, filessytems, and discusses the major implementation features of ext4. If I have one criticism about the book, it's that it's very *nix focused. Many things that are described are simply how things are done in *nix and not inherent, but the text mostly doesn't say when something is inherent vs. when it's a *nix implementation detail.

Love; Linux Kernel Development

The title can be a bit misleading -- this is basically a book about how the Linux kernel works: how things fit together, what algorithms and data structures are used, etc. I read the 2nd edition, which is now quite dated. The 3rd edition has some updates, but introduced some errors and inconsistencies, and is still dated (it was published in 2010, and covers 2.6.34). Even so, it's a nice introduction into how a relatively modern operating system works.

The other downside of this book is that the author loses all objectivity any time Linux and Windows are compared. Basically every time they're compared, the author says that Linux has clearly and incontrovertibly made the right choice and that Windows is doing something stupid. On balance, I prefer Linux to Windows, but there are a number of areas where Windows is superior, as well as areas where there's parity but Windows was ahead for years. You'll never find out what they are from this book, though.

Russinovich, Solomon, and Ionescu; Windows Internals

The most comprehensive book about how a modern operating system works. It just happens to be about Windows. Coming from a *nix background, I found this interesting to read just to see the differences.

This is definitely not an intro book, and you should have some knowledge of operating systems before reading this. If you're going to buy a physical copy of this book, you might want to wait until the 7th edition is released (early in 2017).

Downey; The Little Book of Semaphores

Takes a topic that's normally one or two sections in an operating systems textbook and turns it into its own 300 page book. The book is a series of exercises, a bit like The Little Schemer, but with more exposition. It starts by explaining what semaphore is, and then has a series of exercises that builds up higher level concurrency primitives.

This book was very helpful when I first started to write threading/concurrency code. I subscribe to the Butler Lampson school of concurrency, which is to say that I prefer to have all the concurrency-related code stuffed into a black box that someone else writes. But sometimes you're stuck writing the black box, and if so, this book has a nice introduction to the style of thinking required to write maybe possibly not totally wrong concurrent code.

I wish someone would write a book in this style, but both lower level and higher level. I'd love to see exercises like this, but starting with instruction-level primitives for a couple different architectures with different memory models (say, x86 and Alpha) instead of semaphores. If I'm writing grungy low-level threading code today, I'm overwhelmingly likely to be using c++11 threading primitives, so I'd like something that uses those instead of semaphores, which I might have used if I was writing threading code against the Win32 API. But since that book doesn't exist, this seems like the next best thing.

I've heard that Doug Lea's Concurrent Programming in Java is also quite good, but I've only taken a quick look at it.

Computer architecture

Why should you care? The specific facts and trivia you'll learn will be useful when you're doing low-level performance optimizations, but the real value is learning how to reason about tradeoffs between performance and other factors, whether that's power, cost, size, weight, or something else.

In theory, that kind of reasoning should be taught regardless of specialization, but my experience is that comp arch folks are much more likely to “get” that kind of reasoning and do back of the envelope calculations that will save them from throwing away a 2x or 10x (or 100x) factor in performance for no reason. This sounds obvious, but I can think of multiple production systems at large companies that are giving up 10x to 100x in performance which are operating at a scale where even a 2x difference in performance could pay a VP's salary -- all because people didn't think through the performance implications of their design.

Hennessy & Patterson; Computer Architecture: A Quantitative Approach

This book teaches you how to do systems design with multiple constraints (e.g., performance, TCO, and power) and how to reason about tradeoffs. It happens to mostly do so using microprocessors and supercomputers as examples.

New editions of this book have substantive additions and you really want the latest version. For example, the latest version added, among other things, a chapter on data center design, and it answers questions like, how much opex/capex is spent on power, power distribution, and cooling, and how much is spent on support staff and machines, what's the effect of using lower power machines on tail latency and result quality (bing search results are used as an example), and what other factors should you consider when designing a data center.

Assumes some background, but that background is presented in the appendices (which are available online for free).

Shen & Lipasti: Modern Processor Design

Presents most of what you need to know to architect a high performance Pentium Pro (1995) era microprocessor. That's no mean feat, considering the complexity involved in such a processor. Additionally, presents some more advanced ideas and bounds on how much parallelism can be extracted from various workloads (and how you might go about doing such a calculation). Has an unusually large section on value prediction, because the authors invented the concept and it was still hot when the first edition was published.

For pure CPU architecture, this is probably the best book available.

Hill, Jouppi, and Sohi, Readings in Computer Architecture

Read for historical reasons and to see how much better we've gotten at explaining things. For example, compare Amdahl's paper on Amdahl's law (two pages, with a single non-obvious graph presented, and no formulas), vs. the presentation in a modern textbook (one paragraph, one formula, and maybe one graph to clarify, although it's usually clear enough that no extra graph is needed).

This seems to be worse the further back you go; since comp arch is a relatively young field, nothing here is really hard to understand. If you want to see a dramatic example of how we've gotten better at explaining things, compare Maxwell's original paper on Maxwell's equations to a modern treatment of the same material. Fun if you like history, but a bit of slog if you're just trying to learn something.

Algorithmic game theory / auction theory / mechanism design

Why should you care? Some of the world's biggest tech companies run on ad revenue, and those ads are sold through auctions. This field explains how and why they work. Additionally, this material is useful any time you're trying to figure out how to design systems that allocate resources effectively.¹

In particular, incentive compatible mechanism design (roughly, how to create systems that provide globally optimal outcomes when people behave in their own selfish best interest) should be required reading for anyone who designs internal incentive systems at companies. If you've ever worked at a large company that "gets" this and one that doesn't, you'll see that the company that doesn't get it has giant piles of money that are basically being lit on fire because the people who set up incentives created systems that are hugely wasteful. This field gives you the background to understand what sorts of mechanisms give you what sorts of outcomes; reading case studies gives you a very long (and entertaining) list of mistakes that can cost millions or even billions of dollars.

Krishna; Auction Theory

The last time I looked, this was the only game in town for a comprehensive, modern, introduction to auction theory. Covers the classic second price auction result in the first chapter, and then moves on to cover risk aversion, bidding rings, interdependent values, multiple auctions, asymmetrical information, and other real-world issues.

Relatively dry. Unlikely to be motivating unless you're already interested in the topic. Requires an understanding of basic probability and calculus.

Steighlitz; Snipers, Shills, and Sharks: eBay and Human Behavior

Seems designed as an entertaining introduction to auction theory for the layperson. Requires no mathematical background and relegates math to the small print. Covers maybe, 1/10th of the material of Krishna, if that. Fun read.

Crampton, Shoham, and Steinberg; Combinatorial Auctions

Discusses things like how FCC spectrum auctions got to be the way they are and how “bugs” in mechanism design can leave hundreds of millions or billions of dollars on the table. This is one of those books where each chapter is by a different author. Despite that, it still manages to be coherent and I didn't mind reading it straight through. It's self-contained enough that you could probably read this without reading Krishna first, but I wouldn't recommend it.

Shoham and Leyton-Brown; Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations

The title is the worst thing about this book. Otherwise, it's a nice introduction to algorithmic game theory. The book covers basic game theory, auction theory, and other classic topics that CS folks might not already know, and then covers the intersection of CS with these topics. Assumes no particular background in the topic.

Nisan, Roughgarden, Tardos, and Vazirani; Algorithmic Game Theory

A survey of various results in algorithmic game theory. Requires a fair amount of background (consider reading Shoham and Leyton-Brown first). For example, chapter five is basically Devanur, Papadimitriou, Saberi, and Vazirani's JACM paper, Market Equilibrium via a Primal-Dual Algorithm for a Convex Program, with a bit more motivation and some related problems thrown in. The exposition is good and the result is interesting (if you're into that kind of thing), but it's not necessarily what you want if you want to read a book straight through and get an introduction to the field.

Misc

Beyer, Jones, Petoff, and Murphy; Site Reliability Engineering

A description of how Google handles operations. Has the typical Google tone, which is off-putting to a lot of folks with a “traditional” ops background, and assumes that many things can only be done with the SRE model when they can, in fact, be done without going full SRE.

For a much longer description, see this 22 page set of notes on Google's SRE book.

Fowler, Beck, Brant, Opdyke, and Roberts; Refactoring

At the time I read it, it was worth the price of admission for the section on code smells alone. But this book has been so successful that the ideas of refactoring and code smells have become mainstream.

Steve Yegge has a great pitch for this book:

When I read this book for the first time, in October 2003, I felt this horrid cold feeling, the way you might feel if you just realized you've been coming to work for 5 years with your pants down around your ankles. I asked around casually the next day: "Yeah, uh, you've read that, um, Refactoring book, of course, right? Ha, ha, I only ask because I read it a very long time ago, not just now, of course." Only 1 person of 20 I surveyed had read it. Thank goodness all of us had our pants down, not just me.

...

If you're a relatively experienced engineer, you'll recognize 80% or more of the techniques in the book as things you've already figured out and started doing out of habit. But it gives them all names and discusses their pros and cons objectively, which I found very useful. And it debunked two or three practices that I had cherished since my earliest days as a programmer. Don't comment your code? Local variables are the root of all evil? Is this guy a madman? Read it and decide for yourself!

Demarco & Lister, Peopleware

This book seemed convincing when I read it in college. It even had all sorts of studies backing up what they said. No deadlines is better than having deadlines. Offices are better than cubicles. Basically all devs I talk to agree with this stuff.

But virtually every successful company is run the opposite way. Even Microsoft is remodeling buildings from individual offices to open plan layouts. Could it be that all of this stuff just doesn't matter that much? If it really is that important, how come companies that are true believers, like Fog Creek, aren't running roughshod over their competitors?

This book agrees with my biases and I'd love for this book to be right, but the meta evidence makes me want to re-read this with a critical eye and look up primary sources.

Drummond; Renegades of the Empire

This book explains how Microsoft's aggressive culture got to be the way it is today. The intro reads:

Microsoft didn't necessarily hire clones of Gates (although there were plenty on the corporate campus) so much as recruiter those who shared some of Gates's more notable traits -- arrogance, aggressiveness, and high intelligence.

…

Gates is infamous for ridiculing someone's idea as “stupid”, or worse, “random”, just to see how he or she defends a position. This hostile managerial technique invariably spread through the chain of command and created a culture of conflict.

…

Microsoft nurtures a Darwinian order where resources are often plundered and hoarded for power, wealth, and prestige. A manager who leaves on vacation might return to find his turf raided by a rival and his project put under a different command or canceled altogether

On interviewing at Microsoft:

“What do you like about Microsoft?” “Bill kicks ass”, St. John said. “I like kicking ass. I enjoy the feeling of killing competitors and dominating markets”.

…

He was unsure how he was doing and thought he stumbled then asked if he was a "people person". "No, I think most people are idiots", St. John replied.

These answers were exactly what Microsoft was looking for. They resulted in a strong offer and an aggresive courtship.

On developer evangalism at Microsoft:

At one time, Microsoft evangelists were also usually chartered with disrupting competitors by showing up at their conferences, securing positions on and then tangling standards commitees, and trying to influence the media.

…

"We're the group at Microsoft whose job is to fuck Microsoft's competitors"

Read this book if you're considering a job at Microsoft. Although it's been a long time since the events described in this book, you can still see strains of this culture in Microsoft today.

Bilton; Hatching Twitter

An entertaining book about the backstabbing, mismangement, and random firings that happened in Twitter's early days. When I say random, I mean that there were instances where critical engineers were allegedly fired so that the "decider" could show other important people that current management was still in charge.

I don't know folks who were at Twitter back then, but I know plenty of folks who were at the next generation of startups in their early days and there are a couple of companies where people had eerily similar experiences. Read this book if you're considering a job at a trendy startup.

Galenson; Old Masters and Young Geniuses

This book is about art and how productivity changes with age, but if its thesis is valid, it probably also applies to programming. Galenson applies statistics to determine the "greatness" of art and then uses that to draw conclusions about how the productivty of artists change as they age. I don't have time to go over the data in detail, so I'll have to remain skeptical of this until I have more free time, but I think it's interesting reading even for a skeptic.

Math

Why should you care? From a pure ROI perspective, I doubt learning math is “worth it” for 99% of jobs out there. AFAICT, I use math more often than most programmers, and I don't use it all that often. But having the right math background sometimes comes in handy and I really enjoy learning math. YMMV.

Bertsekas; Introduction to Probability

Introductory undergrad text that tends towards intuitive explanations over epsilon-delta rigor. For anyone who cares to do more rigorous derivations, there are some exercises at the back of the book that go into more detail.

Has many exercises with available solutions, making this a good text for self-study.

Ross; A First Course in Probability

This is one of those books where they regularly crank out new editions to make students pay for new copies of the book (this is presently priced at a whopping $174 on Amazon)². This was the standard text when I took probability at Wisconsin, and I literally cannot think of a single person who found it helpful. Avoid.

Brualdi; Introductory Combinatorics

Brualdi is a great lecturer, one of the best I had in undergrad, but this book was full of errors and not particularly clear. There have been two new editions since I used this book, but according to the Amazon reviews the book still has a lot of errors.

For an alternate introductory text, I've heard good things about Camina & Lewis's book, but I haven't read it myself. Also, Lovasz is a great book on combinatorics, but it's not exactly introductory.

Apostol; Calculus

Volume 1 covers what you'd expect in a calculus I + calculus II book. Volume 2 covers linear algebra and multivariable calculus. It covers linear algebra before multivariable calculus, which makes multi-variable calculus a lot easier to understand.

It also makes a lot of sense from a programming standpoint, since a lot of the value I get out of calculus is its applications to approximations, etc., and that's a lot clearer when taught in this sequence.

This book is probably a rough intro if you don't have a professor or TA to help you along. The Spring SUMS series tends to be pretty good for self-study introductions to various areas, but I haven't actually read their intro calculus book so I can't actually recommend it.

Stewart; Calculus

Another one of those books where they crank out new editions with trivial changes to make money. This was the standard text for non-honors calculus at Wisconsin, and the result of that was I taught a lot of people to do complex integrals with the methods covered in Apostol, which are much more intuitive to many folks.

This book takes the approach that, for a type of problem, you should pattern match to one of many possible formulas and then apply the formula. Apostol is more about teaching you a few tricks and some intuition that you can apply to a wide variety of problems. I'm not sure why you'd buy this unless you were required to for some class.

Hardware basics

Why should you care? People often claim that, to be a good programmer, you have to understand every abstraction you use. That's nonsense. Modern computing is too complicated for any human to have a real full-stack understanding of what's going on. In fact, one reason modern computing can accomplish what it does is that it's possible to be productive without having a deep understanding of much of the stack that sits below the level you're operating at.

That being said, if you're curious about what sits below software, here are a few books that will get you started.

Nisan & Shocken; nand2tetris

If you only want to read one single thing, this should probably be it. It's a “101” level intro that goes down to gates and Boolean logic. As implied by the name, it takes you from NAND gates to a working tetris program.

Roth; Fundamentals of Logic Design

Much more detail on gates and logic design than you'll see in nand2tetris. The book is full of exercises and appears to be designed to work for self-study. Note that the link above is to the 5th edition. There are newer, more expensive, editions, but they don't seem to be much improved, have a lot of errors in the new material, and are much more expensive.

Weste; Harris, and Bannerjee; CMOS VLSI Design

One level below Boolean gates, you get to VLSI, a historical acronym (very large scale integration) that doesn't really have any meaning today.

Broader and deeper than the alternatives, with clear exposition. Explores the design space (e.g., the section on adders doesn't just mention a few different types in an ad hoc way, it explores all the tradeoffs you can make. Also, has both problems and solutions, which makes it great for self study.

Kang & Leblebici; CMOS Digital Integrated Circuits

This was the standard text at Wisconsin way back in the day. It was hard enough to follow that the TA basically re-explained pretty much everything necessary for the projects and the exams. I find that it's ok as a reference, but it wasn't a great book to learn from.

Compared to West et al., Weste spends a lot more effort talking about tradeoffs in design (e.g., when creating a parallel prefix tree adder, what does it really mean to be at some particular point in the design space?).

Pierret; Semiconductor Device Fundamentals

One level below VLSI, you have how transistors actually work.

Really beautiful explanation of solid state devices. The text nails the fundamentals of what you need to know to really understand this stuff (e.g., band diagrams), and then uses those fundamentals along with clear explanations to give you a good mental model of how different types of junctions and devices work.

Streetman & Bannerjee; Solid State Electronic Devices

Covers the same material as Pierret, but seems to substitute mathematical formulas for the intuitive understanding that Pierret goes for.

Ida; Engineering Electromagnetics

One level below transistors, you have electromagnetics.

Two to three times thicker than other intro texts because it has more worked examples and diagrams. Breaks things down into types of problems and subproblems, making things easy to follow. For self-study, A much gentler introduction than Griffiths or Purcell.

Shanley; Pentium Pro and Pentium II System Architecture

Unlike the other books in this section, this book is about practice instead of theory. It's a bit like Windows Internals, in that it goes into the details of a real, working, system. Topics include hardware bus protocols, how I/O actually works (e.g., APIC), etc.

The problem with a practical introduction is that there's been an exponential increase in complexity ever since the 8080. The further back you go, the easier it is to understand the most important moving parts in the system, and the more irrelevant the knowledge. This book seems like an ok compromise in that the bus and I/O protocols had to handle multiprocessors, and many of the elements that are in modern systems were in these systems, just in a simpler form.

Not covered

Of the books that I've liked, I'd say this captures at most 25% of the software books and 5% of the hardware books. On average, the books that have been left off the list are more specialized. This list is also missing many entire topic areas, like PL, practical books on how to learn languages, networking, etc.

The reasons for leaving off topic areas vary; I don't have any PL books listed because I don't read PL books. I don't have any networking books because, although I've read a couple, I don't know enough about the area to really say how useful the books are. The vast majority of hardware books aren't included because they cover material that you wouldn't care about unless you were a specialist (e.g., Skew-Tolerant Circuit Design or Ultrafast Optics). The same goes for areas like math and CS theory, where I left off a number of books that I think are great but have basically zero probability of being useful in my day-to-day programming life, e.g., Extremal Combinatorics. I also didn't include books I didn't read all or most of, unless I stopped because the book was atrocious. This means that I don't list classics I haven't finished like SICP and The Little Schemer, since those book seem fine and I just didn't finish them for one reason or another.

This list also doesn't include many books on history and culture, like Inside Intel or Masters of Doom. I'll probably add more at some point, but I've been trying an experiment where I try to write more like Julia Evans (stream of consciousness, fewer or no drafts). I'd have to go back and re-read the books I read 10+ years ago to write meaningful comments, which doesn't exactly fit with the experiment. On that note, since this list is from memory and I got rid of almost all of my books a couple years ago, I'm probably forgetting a lot of books that I meant to add.

_If you liked this, you might also like Thomas Ptacek's Application Security Reading List or this list of programming blogs, which is written in a similar style_

_Thanks to @tytrdev for comments/corrections/discussion.

HN comments are terrible. On any topic I’m informed about, the vast majority of comments are pretty clearly wrong. Most of the time, there are zero comments from people who know anything about the topic and the top comment is reasonable sounding but totally incorrect. Additionally, many comments are gratuitously mean. You'll often hear mean comments backed up with something like "this is better than the other possibility, where everyone just pats each other on the back with comments like 'this is great'", as if being an asshole is some sort of talisman against empty platitudes. I've seen people push back against that; when pressed, people often say that it’s either impossible or inefficient to teach someone without being mean, as if telling someone that they're stupid somehow helps them learn. It's as if people learned how to explain things by watching Simon Cowell and can't comprehend the concept of an explanation that isn't littered with personal insults. Paul Graham has said, "Oh, you should never read Hacker News comments about anything you write”. Most of the negative things you hear about HN comments are true.

And yet, I haven’t found a public internet forum with better technical commentary. On topics I'm familiar with, while it's rare that a thread will have even a single comment that's well-informed, when those comments appear, they usually float to the top. On other forums, well-informed comments are either non-existent or get buried by reasonable sounding but totally wrong comments when they appear, and they appear even more rarely than on HN.

By volume, there are probably more interesting technical “posts” in comments than in links. Well, that depends on what you find interesting, but that’s true for my interests. If I see a low-level optimization comment from nkurz, a comment on business from patio11, a comment on how companies operate by nostrademons, I almost certainly know that I’m going to read an interesting comment. There are maybe 20 to 30 people I can think of who don’t blog much, but write great comments on HN and I doubt I even know of half the people who are writing great comments on HN¹.

I compiled a very abbreviated list of comments I like because comments seem to get lost. If you write a blog post, people will refer it years later, but comments mostly disappear. I think that’s sad -- there’s a lot of great material on HN (and yes, even more not-so-great material).

What’s the deal with MS Word’s file format?

Basically, the Word file format is a binary dump of memory. I kid you not. They just took whatever was in memory and wrote it out to disk. We can try to reason why (maybe it was faster, maybe it made the code smaller), but I think the overriding reason is that the original developers didn't know any better.

Later as they tried to add features they had to try to make it backward compatible. This is where a lot of the complexity lies. There are lots of crazy workarounds for things that would be simple if you allowed yourself to redesign the file format. It's pretty clear that this was mandated by management, because no software developer would put themselves through that hell for no reason.

Later they added a fast-save feature (I forget what it is actually called). This appends changes to the file without changing the original file. The way they implemented this was really ingenious, but complicates the file structure a lot.

One thing I feel I must point out (I remember posting a huge thing on slashdot when this article was originally posted) is that 2 way file conversion is next to impossible for word processors. That's because the file formats do not contain enough information to format the document. The most obvious place to see this is pagination. The file format does not say where to paginate a text flow (unless it is explicitly entered by the user). It relies of the formatter to do it. Each word processor formats text completely differently. Word, for example famously paginates footnotes incorrectly. They can't change it, though, because it will break backwards compatibility. This is one of the only reasons that Word Perfect survives today -- it is the only word processor that paginates legal documents the way the US Department of Justice requires.

Just considering the pagination issue, you can see what the problem is. When reading a Word document, you have to paginate it like Word -- only the file format doesn't tell you what that is. Then if someone modifies the document and you need to resave it, you need to somehow mark that it should be paginated like Word (even though it might now have features that are not in Word). If it was only pagination, you might be able to do it, but practically everything is like that.

I recommend reading (a bit of) the XML Word file format for those who are interested. You will see large numbers of flags for things like "Format like Word 95". The format doesn't say what that is -- because it's pretty obvious that the authors of the file format don't know. It's lost in a hopeless mess of legacy code and nobody can figure out what it does now.

Fun with NULL

Here's another example of this fine feature:

  #include <stdio.h>
  #include <string.h>
  #include <stdlib.h>
  #define LENGTH 128

  int main(int argc, char **argv) {
      char *string = NULL;
      int length = 0;
      if (argc > 1) {
          string = argv[1];
          length = strlen(string);
          if (length >= LENGTH) exit(1);
      }

      char buffer[LENGTH];
      memcpy(buffer, string, length);
      buffer[length] = 0;

      if (string == NULL) {
          printf("String is null, so cancel the launch.\n");
      } else {
          printf("String is not null, so launch the missiles!\n");
      }

      printf("string: %s\n", string);  // undefined for null but works in practice

      #if SEGFAULT_ON_NULL
      printf("%s\n", string);          // segfaults on null when bare "%s\n"
      #endif

      return 0;
  }

  nate@skylake:~/src$ clang-3.8 -Wall -O3 null_check.c -o null_check
  nate@skylake:~/src$ null_check
  String is null, so cancel the launch.
  string: (null)

  nate@skylake:~/src$ icc-17 -Wall -O3 null_check.c -o null_check
  nate@skylake:~/src$ null_check
  String is null, so cancel the launch.
  string: (null)

  nate@skylake:~/src$ gcc-5 -Wall -O3 null_check.c -o null_check
  nate@skylake:~/src$ null_check
  String is not null, so launch the missiles!
  string: (null)

It appear that Intel's ICC and Clang still haven't caught up with GCC's optimizations. Ouch if you were depending on that optimization to get the performance you need! But before picking on GCC too much, consider that all three of those compilers segfault on printf("string: "); printf("%s\n", string) when string is NULL, despite having no problem with printf("string: %s\n", string) as a single statement. Can you see why using two separate statements would cause a segfault? If not, see here for a hint: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25609

How do you make sure the autopilot backup is paying attention?

Good engineering eliminates users being able to do the wrong thing as much as possible. . . . You don't design a feature that invites misuse and then use instructions to try to prevent that misuse.

There was a derailment in Australia called the Waterfall derailment [1]. It occurred because the driver had a heart attack and was responsible for 7 deaths (a miracle it was so low, honestly). The root cause was the failure of the dead-man's switch.

In the case of Waterfall, the driver had 2 dead-man switches he could use - 1) the throttle handle had to be held against a spring at a small rotation, or 2) a bar on the floor could be depressed. You had to do 1 of these things, the idea being that you prevent wrist or foot cramping by allowing the driver to alternate between the two. Failure to do either triggers an emergency brake.

It turns out that this driver was fat enough that when he had a heart attack, his leg was able to depress the pedal enough to hold the emergency system off. Thus, the dead-man's system never triggered with a whole lot of dead man in the driver's seat.

I can't quite remember the specifics of the system at Waterfall, but one method to combat this is to require the pedal to be held halfway between released and fully depressed. The idea being that a dead leg would fully depress the pedal so that would trigger a brake, and a fully released pedal would also trigger a brake. I don't know if they had that system but certainly that's one approach used in rail.

Either way, the problem is equally possible in cars. If you lose consciousness and your foot goes limp, a heavy enough leg will be able to hold the pedal down a bit depending on where it's positioned relative to the pedal and the leverage it has on the floor.

The other major system I'm familiar with for ensuring drivers are alive at the helm is called 'vigilance'. The way it works is that periodically, a light starts flashing on the dash and the driver has to acknowledge that. If they do not, a buzzer alarm starts sounding. If they still don't acknowledge it, the train brakes apply and the driver is assumed incapacitated. Let me tell you some stories of my involvement in it.

When we first started, we had a simple vigi system. Every 30 seconds or so (for example), the driver would press a button. Ok cool. Except that then drivers became so hard-wired to pressing the button every 30 seconds that we were having instances of drivers falling asleep/dozing off and still pressing the button right on every 30 seconds because it was so ingrained into them that it was literally a subconscious action.

So we introduced random-timing vigilance, where the time varies 30-60 seconds (for example) and you could only acknowledge it within a small period of time once the light started flashing. Again, drivers started falling asleep/semi asleep and would hit it as soon as the alarm buzzed, each and every time.

So we introduced random-timing, task-linked vigilance and that finally broke the back of the problem. Now, the driver has to press a button, or turn a knob, or do a number of different activities and they must do that randomly-chosen activity, at a randomly-chosen time, for them to acknowledge their consciousness. It was only at that point that we finally nailed out driver alertness.

See also.

Prestige

Curious why he would need to move to a more prestigious position? Most people realize by their 30s that prestige is a sucker's game; it's a way of inducing people to do things that aren't much fun and they wouldn't really want to do on their own, by lauding them with accolades from people they don't really care about.

Why is FedEx based in Mephis?

. . . we noticed that we also needed:
(1) A suitable, existing airport at the hub location.
(2) Good weather at the hub location, e.g., relatively little snow, fog, or rain.
(3) Access to good ramp space, that is, where to park and service the airplanes and sort the packages.
(4) Good labor supply, e.g., for the sort center.
(5) Relatively low cost of living to keep down prices.
(6) Friendly regulatory environment.
(7) Candidate airport not too busy, e.g., don't want arriving planes to have to circle a long time before being able to land.
(8) Airport with relatively little in cross winds and with more than one runway to pick from in case of winds.
(9) Runway altitude not too high, e.g., not high enough to restrict maximum total gross take off weight, e.g., rule out Denver.
(10) No tall obstacles, e.g., mountains, near the ends of the runways.
(11) Good supplies of jet fuel.
(12) Good access to roads for 18 wheel trucks for exchange of packages between trucks and planes, e.g., so that some parts could be trucked to the hub and stored there and shipped directly via the planes to customers that place orders, say, as late as 11 PM for delivery before 10 AM.
So, there were about three candidate locations, Memphis and, as I recall, Cincinnati and Kansas City.
The Memphis airport had some old WWII hangers next to the runway that FedEx could use for the sort center, aircraft maintenance, and HQ office space. Deal done -- it was Memphis.

Why etherpad joined Wave, and why it didn’t work out as expected

The decision to sell to Google was one of the toughest decisions I and my cofounders ever had to wrestle with in our lives. We were excited by the Wave vision though we saw the flaws in the product. The Wave team told us about how they wanted our help making wave simpler and more like etherpad, and we thought we could help with that, though in the end we were unsuccessful at making wave simpler. We were scared of Google as a competitor: they had more engineers and more money behind this project, yet they were running it much more like an independent startup than a normal big-company department. The Wave office was in Australia and had almost total autonomy. And finally, after 1.5 years of being on the brink of failure with AppJet, it was tempting to be able to declare our endeavor a success and provide a decent return to all our investors who had risked their money on us.

In the end, our decision to join Wave did not work out as we had hoped. The biggest lessons learned were that having more engineers and money behind a project can actually be more harmful than helpful, so we were wrong to be scared of Wave as a competitor for this reason. It seems obvious in hindsight, but at the time it wasn't. Second, I totally underestimated how hard it would be to iterate on the Wave codebase. I was used to rewriting major portions of software in a single all-nighter. Because of the software development process Wave was using, it was practically impossible to iterate on the product. I should have done more diligence on their specific software engineering processes, but instead I assumed because they seemed to be operating like a startup, that they would be able to iterate like a startup. A lot of the product problems were known to the whole Wave team, but we were crippled by a large complex codebase built on poor technical choices and a cumbersome engineering process that prevented fast iteration.

The accuracy of tech news

When I've had inside information about a story that later breaks in the tech press, I'm always shocked at how differently it's perceived by readers of the article vs. how I experienced it. Among startups & major feature launches I've been party to, I've seen: executives that flat-out say that they're not working on a product category when there's been a whole department devoted to it for a year; startups that were founded 1.5 years before the dates listed in Crunchbase/Wikipedia; reporters that count the number of people they meet in a visit and report that as a the "team size", because the company refuses to release that info; funding rounds that never make it to the press; acquisitions that are reported as "for an undisclosed sum" but actually are less than the founders would've made if they'd taken a salaried job at the company; project start dates that are actually when the project was staffed up to its current size and ignore the year or so that a small team spent working on the problem (or the 3-4 years that other small teams spent working on the problem); and algorithms or other technologies that are widely reported as being the core of the company's success, but actually aren't even used by the company.

Self-destructing speakers from Dell

As the main developer of VLC, we know about this story since a long time, and this is just Dell putting crap components on their machine and blaming others. Any discussion was impossible with them. So let me explain a bit...

In this case, VLC just uses the Windows APIs (DirectSound), and sends signed integers of 16bits (s16) to the Windows Kernel.

VLC allows amplification of the INPUT above the sound that was decoded. This is just like replay gain, broken codecs, badly recorded files or post-amplification and can lead to saturation.

But this is exactly the same if you put your mp3 file through Audacity and increase it and play with WMP, or if you put a DirectShow filter that amplifies the volume after your codec output. For example, for a long time, VLC ac3 and mp3 codecs were too low (-6dB) compared to the reference output.

At worse, this will reduce the dynamics and saturate a lot, but this is not going to break your hardware.

VLC does not (and cannot) modify the OUTPUT volume to destroy the speakers. VLC is a Software using the OFFICIAL platforms APIs.

The issue here is that Dell sound cards output power (that can be approached by a factor of the quadratic of the amplitude) that Dell speakers cannot handle. Simply said, the sound card outputs at max 10W, and the speakers only can take 6W in, and neither their BIOS or drivers block this.

And as VLC is present on a lot of machines, it's simple to blame VLC. "Correlation does not mean causation" is something that seems too complex for cheap Dell support…

Learning on the job, startups vs. big companies

Working for someone else's startup, I learned how to quickly cobble solutions together. I learned about uncertainty and picking a direction regardless of whether you're sure it'll work. I learned that most startups fail, and that when they fail, the people who end up doing well are the ones who were looking out for their own interests all along. I learned a lot of basic technical skills, how to write code quickly and learn new APIs quickly and deploy software to multiple machines. I learned how quickly problems of scaling a development team crop up, and how early you should start investing in automation.

Working for Google, I learned how to fix problems once and for all and build that culture into the organization. I learned that even in successful companies, everything is temporary, and that great products are usually built through a lot of hard work by many people rather than great ah-ha insights. I learned how to architect systems for scale, and a lot of practices used for robust, high-availability, frequently-deployed systems. I learned the value of research and of spending a lot of time on a single important problem: many startups take a scattershot approach, trying one weekend hackathon after another and finding nobody wants any of them, while oftentimes there are opportunities that nobody has solved because nobody wants to put in the work. I learned how to work in teams and try to understand what other people want. I learned what problems are really painful for big organizations. I learned how to rigorously research the market and use data to make product decisions, rather than making decisions based on what seems best to one person.

We failed this person, what are we going to do differently?

Having been in on the company's leadership meetings where departures were noted with a simple 'regret yes/no' flag it was my experience that no single departure had any effect. Mass departures did, trends did, but one person never did, even when that person was a founder.

The rationalizations always put the issue back on the departing employee, "They were burned out", "They had lost their ability to be effective", "They have moved on", "They just haven't grown with the company" never was it "We failed this person, what are we going to do differently?"

AWS’s origin story

Anyway, the SOA effort was in full swing when I was there. It was a pain, and it was a mess because every team did things differently and every API was different and based on different assumptions and written in a different language.

But I want to correct the misperception that this lead to AWS. It didn't. S3 was written by its own team, from scratch. At the time I was at Amazon, working on the retail site, none of Amazon.com was running on AWS. I know, when AWS was announced, with great fanfare, they said "the services that power Amazon.com can now power your business!" or words to that effect. This was a flat out lie. The only thing they shared was data centers and a standard hardware configuration. Even by the time I left, when AWS was running full steam ahead (and probably running Reddit already), none of Amazon.com was running on AWS, except for a few, small, experimental and relatively new projects. I'm sure more of it has been adopted now, but AWS was always a separate team (and a better managed one, from what I could see.)

Why is Windows so slow?

I (and others) have put a lot of effort into making the Linux Chrome build fast. Some examples are multiple new implementations of the build system (http://neugierig.org/software/chromium/notes/2011/02/ninja.h... ), experimentation with the gold linker (e.g. measuring and adjusting the still off-by-default thread flags https://groups.google.com/a/chromium.org/group/chromium-dev/... ) as well as digging into bugs in it, and other underdocumented things like 'thin' ar archives.

But it's also true that people who are more of Windows wizards than I am a Linux apprentice have worked on Chrome's Windows build. If you asked me the original question, I'd say the underlying problem is that on Windows all you have is what Microsoft gives you and you can't typically do better than that. For example, migrating the Chrome build off of Visual Studio would be a large undertaking, large enough that it's rarely considered. (Another way of phrasing this is it's the IDE problem: you get all of the IDE or you get nothing.)

When addressing the poor Windows performance people first bought SSDs, something that never even occurred to me ("your system has enough RAM that the kernel cache of the file system should be in memory anyway!"). But for whatever reason on the Linux side some Googlers saw it fit to rewrite the Linux linker to make it twice as fast (this effort predated Chrome), and all Linux developers now get to benefit from that. Perhaps the difference is that when people write awesome tools for Windows or Mac they try to sell them rather than give them away.

Why is Windows so slow, an insider view

I'm a developer in Windows and contribute to the NT kernel. (Proof: the SHA1 hash of revision #102 of [Edit: filename redacted] is [Edit: hash redacted].) I'm posting through Tor for obvious reasons.

Windows is indeed slower than other operating systems in many scenarios, and the gap is worsening. The cause of the problem is social. There's almost none of the improvement for its own sake, for the sake of glory, that you see in the Linux world.

Granted, occasionally one sees naive people try to make things better. These people almost always fail. We can and do improve performance for specific scenarios that people with the ability to allocate resources believe impact business goals, but this work is Sisyphean. There's no formal or informal program of systemic performance improvement. We started caring about security because pre-SP3 Windows XP was an existential threat to the business. Our low performance is not an existential threat to the business.

See, component owners are generally openly hostile to outside patches: if you're a dev, accepting an outside patch makes your lead angry (due to the need to maintain this patch and to justify in in shiproom the unplanned design change), makes test angry (because test is on the hook for making sure the change doesn't break anything, and you just made work for them), and PM is angry (due to the schedule implications of code churn). There's just no incentive to accept changes from outside your own team. You can always find a reason to say "no", and you have very little incentive to say "yes".

What’s the probability of a successful exit by city?

See link for giant table :-).

The hiring crunch

Broken record: startups are also probably rejecting a lot of engineering candidates that would perform as well or better than anyone on their existing team, because tech industry hiring processes are folkloric and irrational.

I co-manage a consultancy. We operate in the valley. We're in a very specialized niche that is especially demanding of software development skills. Our skills needs also track the market, because we have to play on our clients turf. Consultancies running in steady state have an especially direct relationship between recruiting and revenue.

A few years ago, we found ourselves crunched. We turned a lot of different knobs to try to solve the problem. For a while, Hacker News was our #1 recruiting vehicle. We ran ads. We went to events at schools. We shook down our networks and those of our team (by offering larger and larger recruiting bonuses, among other things).

We have since resolved this problem. My current perspective is that we have little trouble filling slots as we add them, in any market --- we operate in Chicago (where it is trivially easy to recruit), SFBA (harder), and NYC (hardest). We've been in a comfortable place with recruiting for almost a year now (ie, about half the lifetime of a typical startup).

I attribute our success to just a few things:

• We created long-running outreach events (the Watsi-pledging crypto challenges, the joint Square MSP CTF) that are graded so that large numbers of people can engage and get value from them, but people who are especially interested in them can self-select their way to talking to us about a job. Worth mentioning: the crypto challenges, which are currently by far our most successful recruiting vehicle (followed by Stripe's CTF #2) are just a series of emails we send; they're essentially a blog post that we weaponized instead of wasting on a blog.

• We totally overhauled our interview process, with three main goals: (1) we over-communicate and sell our roles before we ever get selective with candidates, (2) we use quantifiable work-sample tests as the most important weighted component in selecting candidates, and (3) we standardize interviews so we can track what is and isn't predictive of success.

Both of these approaches have paid off, but improving interviews has been the more important of the two. Compare the first 2/3rds of Matasano's lifetime to the last 1/3rd. The typical candidate we've hired lately would never have gotten hired at early Matasano, because (a) they wouldn't have had the resume for it, and (b) we over-weighted intangibles like how convincing candidates were in face-to-face interviews. But the candidates we've hired lately compare extremely well to our earlier teams! It's actually kind of magical: we interview people whose only prior work experience is "Line of Business .NET Developer", and they end up showing us how to write exploits for elliptic curve partial nonce bias attacks that involve Fourier transforms and BKZ lattice reduction steps that take 6 hours to run.

How? By running an outreach program that attracts people who are interested in crypto, and building an interview process that doesn't care what your resume says or how slick you are in an interview.

Call it the "Moneyball" strategy.

Should you leave a bad job?

I am 42-year-old very successful programmer who has been through a lot of situations in my career so far, many of them highly demotivating. And the best advice I have for you is to get out of what you are doing. Really. Even though you state that you are not in a position to do that, you really are. It is okay. You are free. Okay, you are helping your boyfriend's startup but what is the appropriate cost for this? Would he have you do it if he knew it was crushing your soul?

I don't use the phrase "crushing your soul" lightly. When it happens slowly, as it does in these cases, it is hard to see the scale of what is happening. But this is a very serious situation and if left unchecked it may damage the potential for you to do good work for the rest of your life.

The commenters who are warning about burnout are right. Burnout is a very serious situation. If you burn yourself out hard, it will be difficult to be effective at any future job you go to, even if it is ostensibly a wonderful job. Treat burnout like a physical injury. I burned myself out once and it took at least 12 years to regain full productivity. Don't do it.

• More broadly, the best and most creative work comes from a root of joy and excitement. If you lose your ability to feel joy and excitement about programming-related things, you'll be unable to do the best work. That this issue is separate from and parallel to burnout! If you are burned out, you might still be able to feel the joy and excitement briefly at the start of a project/idea, but they will fade quickly as the reality of day-to-day work sets in. Alternatively, if you are not burned out but also do not have a sense of wonder, it is likely you will never get yourself started on the good work.

• The earlier in your career it is now, the more important this time is for your development. Programmers learn by doing. If you put yourself into an environment where you are constantly challenged and are working at the top threshold of your ability, then after a few years have gone by, your skills will have increased tremendously. It is like going to intensively learn kung fu for a few years, or going into Navy SEAL training or something. But this isn't just a one-time constant increase. The faster you get things done, and the more thorough and error-free they are, the more ideas you can execute on, which means you will learn faster in the future too. Over the long term, programming skill is like compound interest. More now means a LOT more later. Less now means a LOT less later.

So if you are putting yourself into a position that is not really challenging, that is a bummer day in and day out, and you get things done slowly, you aren't just having a slow time now. You are bringing down that compound interest curve for the rest of your career. It is a serious problem. If I could go back to my early career I would mercilessly cut out all the shitty jobs I did (and there were many of them).

Creating change when politically unpopular

A small anecdote. An acquaintance related a story of fixing the 'drainage' in their back yard. They were trying to grow some plants that were sensitive to excessive moisture, and the plants were dying. Not watering them, watering them a little, didn't seem to change. They died. A professional gardner suggested that their problem was drainage. So they dug down about 3' (where the soil was very very wet) and tried to build in better drainage. As they were on the side of a hill, water table issues were not considered. It turned out their "problem" was that the water main that fed their house and the houses up the hill, was so pressurized at their property (because it had maintain pressure at the top of the hill too) that the pipe seams were leaking and it was pumping gallons of water into the ground underneath their property. The problem wasn't their garden, the problem was that the city water supply was poorly designed.

While I have never been asked if I was an engineer on the phone, I have experienced similar things to Rachel in meetings and with regard to suggestions. Co-workers will create an internal assessment of your value and then respond based on that assessment. If they have written you off they will ignore you, if you prove their assessment wrong in a public forum they will attack you. These are management issues, and something which was sorely lacking in the stories.

If you are the "owner" of a meeting, and someone is trying to be heard and isn't. It is incumbent on you to let them be heard. By your position power as "the boss" you can naturally interrupt a discussion to collect more data from other members. Its also important to ask questions like "does anyone have any concerns?" to draw out people who have valid input but are too timid to share it.

In a highly political environment there are two ways to create change, one is through overt manipulation, which is to collect political power to yourself and then exert it to enact change, and the other is covert manipulation, which is to enact change subtly enough that the political organism doesn't react. (sometimes called "triggering the antibodies").

The problem with the latter is that if you help make positive change while keeping everyone not pissed off, no one attributes it to you (which is good for the change agent because if they knew the anti-bodies would react, but bad if your manager doesn't recognize it). I asked my manager what change he wanted to be 'true' yet he (or others) had been unsuccessful making true, he gave me one, and 18 months later that change was in place. He didn't believe that I was the one who had made the change. I suggested he pick a change he wanted to happen and not tell me, then in 18 months we could see if that one happened :-). But he also didn't understand enough about organizational dynamics to know that making change without having the source of that change point back at you was even possible.

How to get tech support from Google

Heavily relying on Google product? ✓
Hitting a dead-end with Google's customer service? ✓
Have an existing audience you can leverage to get some random Google employee's attention? ✓
Reach front page of Hacker News? ✓
Good news! You should have your problem fixed in 2-5 business days. The rest of us suckers relying on google services get to stare at our inboxes helplessly, waiting for a response to our support ticket (which will never come). I feel like it's almost a [rite] of passage these days to rely heavily on a Google service, only to have something go wrong and be left out in the cold.

Taking funding

IIRC PayPal was very similar - it was sold for $1.5B, but Max Levchin's share was only about $30M, and Elon Musk's was only about $100M. By comparison, many early Web 2.0 darlings (Del.icio.us, Blogger, Flickr) sold for only $20-40M, but their founders had only taken small seed rounds, and so the vast majority of the purchase price went to the founders. 75% of a $40M acquisition = 3% of a $1B acquisition.

Something for founders to think about when they're taking funding. If you look at the gigantic tech fortunes - Gates, Page/Brin, Omidyar, Bezos, Zuckerburg, Hewlett/Packard - they usually came from having a company that was already profitable or was already well down the hockey-stick user growth curve and had a clear path to monetization by the time they sought investment. Companies that fight tooth & nail for customers and need lots of outside capital to do it usually have much worse financial outcomes.

StackOverflow vs. Experts-Exchange

A lot of the people who were involved in some way in Experts-Exchange don't understand Stack Overflow.

The basic value flow of EE is that "experts" provide valuable "answers" for novices with questions. In that equation there's one person asking a question and one person writing an answer.

Stack Overflow recognizes that for every person who asks a question, 100 - 10,000 people will type that same question into Google and find an answer that has already been written. In our equation, we are a community of people writing answers that will be read by hundreds or thousands of people. Ours is a project more like wikipedia -- collaboratively creating a resource for the Internet at large.

Because that resource is provided by the community, it belongs to the community. That's why our data is freely available and licensed under creative commons. We did this specifically because of the negative experience we had with EE taking a community-generated resource and deciding to slap a paywall around it.

The attitude of many EE contributors, like Greg Young who calculates that he "worked" for half a year for free, is not shared by the 60,000 people who write answers on SO every month. When you talk to them you realize that on Stack Overflow, answering questions is about learning. It's about creating a permanent artifact to make the Internet better. It's about helping someone solve a problem in five minutes that would have taken them hours to solve on their own. It's not about working for free.

As soon as EE introduced the concept of money they forced everybody to think of their work on EE as just that -- work.

Making money from amazon bots

I saw that one of my old textbooks was selling for a nice price, so I listed it along with two other used copies. I priced it $1 cheaper than the lowest price offered, but within an hour both sellers had changed their prices to $.01 and $.02 cheaper than mine. I reduced it two times more by $1, and each time they beat my price by a cent or two. So what I did was reduce my price by a few dollars every hour for one day until everybody was priced under $5. Then I bought their books and changed my price back.

What running a business is like

While I like the sentiment here, I think the danger is that engineers might come to the mistaken conclusion that making pizzas is the primary limiting reagent to running a successful pizzeria. Running a successful pizzeria is more about schlepping to local hotels and leaving them 50 copies of your menu to put at the front desk, hiring drivers who will both deliver pizzas in a timely fashion and not embezzle your (razor-thin) profits while also costing next-to-nothing to employ, maintaining a kitchen in sufficient order to pass your local health inspector's annual visit (and dealing with 47 different pieces of paper related to that), being able to juggle priorities like "Do I take out a bank loan to build a new brick-oven, which will make the pizza taste better, in the knowledge that this will commit $3,000 of my cash flow every month for the next 3 years, or do I hire an extra cook?", sourcing ingredients such that they're available in quantity and quality every day for a fairly consistent price, setting prices such that they're locally competitive for your chosen clientele but generate a healthy gross margin for the business, understanding why a healthy gross margin really doesn't imply a healthy net margin and that the rent still needs to get paid, keeping good-enough records such that you know whether your business is dying before you can't make payroll and such that you can provide a reasonably accurate picture of accounts for the taxation authorities every year, balancing 50% off medium pizza promotions with the desire to not cannibalize the business of your regulars, etc etc, and by the way tomato sauce should be tangy but not sour and cheese should melt with just the faintest whisp of a crust on it.

Do you want to write software for a living? Google is hiring. Do you want to run a software business? Godspeed. Software is now 10% of your working life.

How to handle mismanagement?

The way I prefer to think of it is: it is not your job to protect people (particularly senior management) from the consequences of their decisions. Make your decisions in your own best interest; it is up to the organization to make sure that your interest aligns with theirs.

Google used to have a severe problem where code refactoring & maintenance was not rewarded in performance reviews while launches were highly regarded, which led to the effect of everybody trying to launch things as fast as possible and nobody cleaning up the messes left behind. Eventually launches started getting slowed down, Larry started asking "Why can't we have nice things?", and everybody responded "Because you've been paying us to rack up technical debt." As a result, teams were formed with the express purpose of code health & maintenance, those teams that were already working on those goals got more visibility, and refactoring contributions started counting for something in perf. Moreover, many ex-Googlers who were fed up with the situation went to Facebook and, I've heard, instituted a culture there where grungy engineering maintenance is valued by your peers.

None of this would've happened if people had just heroically fallen on their own sword and burnt out doing work nobody cared about. Sometimes it takes highly visible consequences before people with decision-making power realize there's a problem and start correcting it. If those consequences never happen, they'll keep believing it's not a problem and won't pay much attention to it.

Some downsides of immutability

Taking responsibility

The thing my grandfather taught me was that you live with all of your decisions for the rest of your life. When you make decisions which put other people at risk, you take on the risk that you are going to make someones life harder, possibly much harder. What is perhaps even more important is that no amount of "I'm so sorry I did that ..." will ever undo it. Sometimes its little things, like taking the last serving because you thought everyone had eaten, sometimes its big things like deciding that home is close enough that and you're sober enough to get there safely. They are all decisions we make every day. And as I've gotten older the weight of ones I wish I had made differently doesn't get any lighter. You can lie to yourself about your choices, rationalize them, but that doesn't change them either.

I didn't understand any of that when I was younger.

People who aren’t exactly lying

It took me too long to figure this out. There are some people to truly, and passionately, believe something they say to you, and realistically they personally can't make it happen so you can't really bank on that 'promise.'

I used to think those people were lying to take advantage, but as I've gotten older I have come to recognize that these 'yes' people get promoted a lot. And for some of them, they really do believe what they are saying.

As an engineer I've found that once I can 'calibrate' someone's 'yes-ness' I can then work with them, understanding that they only make 'wishful' commitments rather than 'reasoned' commitments.

So when someone, like Steve Jobs, says "we're going to make it an open standard!", my first question then is "Great, I've got your support in making this an open standard so I can count on you to wield your position influence to aid me when folks line up against that effort, right?" If the answer that that question is no, then they were lying.

The difference is subtle of course but important. Steve clearly doesn't go to standards meetings and vote etc, but if Manager Bob gets push back from accounting that he's going to exceed his travel budget by sending 5 guys to the Open Video Chat Working Group which is championing the Facetime protocol as an open standard, then Manager Bob goes to Steve and says "I need your help here, these 5 guys are needed to argue this standard and keep it from being turned into a turd by the 5 guys from Google who are going to attend." and then Steve whips off a one liner to accounting that says "Get off this guy's back we need this." Then its all good. If on the other hand he says "We gotta save money, send one guy." well in that case I'm more sympathetic to the accusation of prevarication.

What makes engineers productive?

For those who work inside Google, it's well worth it to look at Jeff & Sanjay's commit history and code review dashboard. They aren't actually all that much more productive in terms of code written than a decent SWE3 who knows his codebase.

The reason they have a reputation as rockstars is that they can apply this productivity to things that really matter; they're able to pick out the really important parts of the problem and then focus their efforts there, so that the end result ends up being much more impactful than what the SWE3 wrote. The SWE3 may spend his time writing a bunch of unit tests that catch bugs that wouldn't really have happened anyway, or migrating from one system to another that isn't really a large improvement, or going down an architectural dead end that'll just have to be rewritten later. Jeff or Sanjay (or any of the other folks operating at that level) will spend their time running a proposed API by clients to ensure it meets their needs, or measuring the performance of subsystems so they fully understand their building blocks, or mentally simulating the operation of the system before building it so they rapidly test out alternatives. They don't actually write more code than a junior developer (oftentimes, they write less), but the code they do write gives them more information, which makes them ensure that they write the rightcode.

I feel like this point needs to be stressed a whole lot more than it is, as there's a whole mythology that's grown up around 10x developers that's not all that helpful. In particular, people need to realize that these developers rapidly become 1x developers (or worse) if you don't let them make their own architectural choices - the reason they're excellent in the first place is because they know how to determine if certain work is going to be useless and avoid doing it in the first place. If you dictate that they do it anyway, they're going to be just as slow as any other developer

Do the work, be a hero

I got the hero speech too, once. If anyone ever mentions the word "heroic" again and there isn't a burning building involved, I will start looking for new employment immediately. It seems that in our industry it is universally a code word for "We're about to exploit you because the project is understaffed and under budgeted for time and that is exactly as we planned it so you'd better cowboy up."

Maybe it is different if you're writing Quake, but I guarantee you the 43rd best selling game that year also had programmers "encouraged onwards" by tales of the glory that awaited after the death march.

Learning English from watching movies

I was once speaking to a good friend of mine here, in English.
"Do you want to go out for yakitori?"
"Go fuck yourself!"
"... switches to Japanese Have I recently done anything very major to offend you?"
"No, of course not."
"Oh, OK, I was worried. So that phrase, that's something you would only say under extreme distress when you had maximal desire to offend me, or I suppose you could use it jokingly between friends, but neither you nor I generally talk that way."
"I learned it from a movie. I thought it meant ‘No.’"

Being smart and getting things done

True story: I went to a talk given by one of the 'engineering elders' (these were low Emp# engineers who were considered quite successful and were to be emulated by the workers :-) This person stated when they came to work at Google they were given the XYZ system to work on (sadly I'm prevented from disclosing the actual system). They remarked how they spent a couple of days looking over the system which was complicated and creaky, they couldn't figure it out so they wrote a new system. Yup, and they committed that. This person is a coding God are they not? (sarcasm) I asked what happened to the old system (I knew but was interested on their perspective) and they said it was still around because a few things still used it, but (quite proudly) nearly everything else had moved to their new system.

So if you were reading carefully, this person created a new system to 'replace' an existing system which they didn't understand and got nearly everyone to move to the new system. That made them uber because they got something big to put on their internal resume, and a whole crapload of folks had to write new code to adapt from the old system to this new system, which imperfectly recreated the old system (remember they didn't understand the original), such that those parts of the system that relied on the more obscure bits had yet to be converted (because nobody undersood either the dependent code or the old system apparently).

Was this person smart? Blindingly brilliant according to some of their peers. Did they get things done? Hell yes, they wrote the replacement for the XYZ system from scratch! One person? Can you imagine? Would I hire them? Not unless they were the last qualified person in my pool and I was out of time.

That anecdote encapsulates the dangerous side of smart people who get things done.

Public speaking tips

Some kids grow up on football. I grew up on public speaking (as behavioral therapy for a speech impediment, actually). If you want to get radically better in a hurry:

1) If you ever find yourself buffering on output, rather than making hesitation noises, just pause. People will read that as considered deliberation and intelligence. It's outrageously more effective than the equivalent amount of emm, aww, like, etc. Practice saying nothing. Nothing is often the best possible thing to say. (A great time to say nothing: during applause or laughter.)

2) People remember voice a heck of a lot more than they remember content. Not vocal voice, but your authorial voice, the sort of thing English teachers teach you to detect in written documents. After you have found a voice which works for you and your typical audiences, you can exploit it to the hilt.

I have basically one way to start speeches: with a self-deprecating joke. It almost always gets a laugh out of the crowd, and I can't be nervous when people are laughing with me, so that helps break the ice and warm us into the main topic.

3) Posture hacks: if you're addressing any group of people larger than a dinner table, pick three people in the left, middle, and right of the crowd. Those three people are your new best friends, who have come to hear you talk but for some strange reason are surrounded by great masses of mammals who are uninvolved in the speech. Funny that. Rotate eye contact over your three best friends as you talk, at whatever a natural pace would be for you. (If you don't know what a natural pace is, two sentences or so works for me to a first approximation.)

Everyone in the audience -- both your friends and the uninvolved mammals -- will perceive that you are looking directly at them for enough of the speech to feel flattered but not quite enough to feel creepy.

4) Podiums were invented by some sadist who hates introverts. Don't give him the satisfaction. Speak from a vantage point where the crowd can see your entire body.

5) Hands: pockets, no, pens, no, fidgeting, no. Gestures, yes. If you don't have enough gross motor control to talk and gesture at the same time (no joke, this was once a problem for me) then having them in a neutral position in front of your body works well.

6) Many people have different thoughts on the level of preparation or memorization which is required. In general, having strong control of the narrative structure of your speech without being wedded to the exact ordering of sentences is a good balance for most people. (The fact that you're coming to the conclusion shouldn't surprise you.)

7) If you remember nothing else on microtactical phrasing when you're up there, remember that most people do not naturally include enough transition words when speaking informally, which tends to make speeches loose narrative cohesion. Throw in a few more than you would ordinarily think to do. ("Another example of this...", "This is why...", "Furthermore...", etc etc.)

A reason a company can be a bad fit

I can relate to this, but I can also relate to the other side of the question. Sometimes it isn't me, its you. Take someone who gets things done and suddenly in your organization they aren't delivering. Could be them, but it could also be you.

I had this experience working at Google. I had a horrible time getting anything done there. Now I spent a bit of time evaluating that since it had never been the case in my career, up to that point, where I was unable to move the ball forward and I really wanted to understand that. The short answer was that Google had developed a number of people who spent much, if not all, of their time preventing change. It took me a while to figure out what motivated someone to be anti-change.

The fear was risk and safety. Folks moved around a lot and so you had people in charge of systems they didn't build, didn't understand all the moving parts of, and were apt to get a poor rating if they broke. When dealing with people in that situation one could either educate them and bring them along, or steam roll over them. Education takes time, and during that time the 'teacher' doesn't get anything done. This favors steamrolling evolutionarily :-)

So you can hire someone who gets stuff done, but if getting stuff done in your organization requires them to be an asshole, and they aren't up for that, well they aren't going to be nearly as successful as you would like them to be.

What working at Google is like

I can tell that this was written by an outsider, because it focuses on the perks and rehashes several cliches that have made their way into the popular media but aren't all that accurate.

Most Googlers will tell you that the best thing about working there is having the ability to work on really hard problems, with really smart coworkers, and lots of resources at your disposal. I remember asking my interviewer whether I could use things like Google's index if I had a cool 20% idea, and he was like "Sure. That's encouraged. Oftentimes I'll just grab 4000 or so machines and run a MapReduce to test out some hypothesis." My phone screener, when I asked him what it was like to work there, said "It's a place where really smart people go to be average," which has turned out to be both true and honestly one of the best things that I've gained from working there.

NSA vs. Black Hat

This entire event was a staged press op. Keith Alexander is a ~30 year veteran of SIGINT, electronic warfare, and intelligence, and a Four-Star US Army General --- which is a bigger deal than you probably think it is. He's a spy chief in the truest sense and a master politician. Anyone who thinks he walked into that conference hall in Caesars without a near perfect forecast of the outcome of the speech is kidding themselves.

Heckling Alexander played right into the strategy. It gave him an opportunity to look reasonable compared to his detractors, and, more generally (and alarmingly), to have the NSA look more reasonable compared to opponents of NSA surveillance. It allowed him to "split the vote" with audience reactions, getting people who probably have serious misgivings about NSA programs to applaud his calm and graceful handling of shouted insults; many of those people probably applauded simply to protest the hecklers, who after all were making it harder for them to follow what Alexander was trying to say.

There was no serious Q&A on offer at the keynote. The questions were pre-screened; all attendees could do was vote on them. There was no possibility that anything would come of this speech other than an effectively unchallenged full-throated defense of the NSA's programs.

Are deadlines necessary?

Interestingly one of the things that I found most amazing when I was working for Google was a nearly total inability to grasp the concept of 'deadline.' For so many years the company just shipped it by committing it to the release branch and having the code deploy over the course of a small number of weeks to the 'fleet'.

Sure there were 'processes', like "Canary it in some cluster and watch the results for a few weeks before turning it loose on the world." but being completely vertically integrated is a unique sort of situation.

Debugging on Windows vs. Linux

Being a very experienced game developer who tried to switch to Linux, I have posted about this before (and gotten flamed heavily by reactionary Linux people).

The main reason is that debugging is terrible on Linux. gdb is just bad to use, and all these IDEs that try to interface with gdb to "improve" it do it badly (mainly because gdb itself is not good at being interfaced with). Someone needs to nuke this site from orbit and build a new debugger from scratch, and provide a library-style API that IDEs can use to inspect executables in rich and subtle ways.

Productivity is crucial. If the lack of a reasonable debugging environment costs me even 5% of my productivity, that is too much, because games take so much work to make. At the end of a project, I just don't have 5% effort left any more. It requires everything. (But the current Linux situation is way more than a 5% productivity drain. I don't know exactly what it is, but if I were to guess, I would say it is something like 20%.)

What happens when you become rich?

What is interesting is that people don't even know they have a complex about money until they get "rich." I've watched many people, perhaps a hundred, go from "working to pay the bills" to "holy crap I can pay all my current and possibly my future bills with the money I now have." That doesn't include the guy who lived in our neighborhood and won the CA lottery one year.

It affects people in ways they don't expect. If its sudden (like lottery winning or sudden IPO surge) it can be difficult to process. But it is an important thing to realize that one is processing an exceptional event. Like having a loved one die or a spouse suddenly divorcing you.

Not everyone feels "guilty", not everyone feels "smug." A lot of millionaires and billionaires in the Bay Area are outwardly unchanged. But the bottom line is that the emotion comes from the cognitive dissonance between values and reality. What do you value? What is reality?

One woman I knew at Google was massively conflicted when she started work at Google. She always felt that she would help the homeless folks she saw, if she had more money than she needed. Upon becoming rich (on Google stock value), now she found that she wanted to save the money she had for her future kids education and needs. Was she a bad person? Before? After? Do your kids hate you if you give away their college education to the local foodbank? Do your peers hate you because you could close the current food gap at the foodbank and you don't?

Microsoft’s Skype acquisition

This is Microsoft's ICQ moment. Overpaying for a company at the moment when its core competency is becoming a commodity. Does anyone have the slightest bit of loyalty to Skype? Of course not. They're going to use whichever video chat comes built into their SmartPhone, tablet, computer, etc. They're going to use FaceBook's eventual video chat service or something Google offers. No one is going to actively seek out Skype when so many alternatives exist and are deeply integrated into the products/services they already use. Certainly no one is going to buy a Microsoft product simply because it has Skype integration. Who cares if it's FaceTime, FaceBook Video Chat, Google Video Chat? It's all the same to the user.

With $7B they should have just given away about 15 million Windows Mobile phones in the form of an epic PR stunt. It's not a bad product -- they just need to make people realize it exists. If they want to flush money down the toilet they might as well engage users in the process right?

What happened to Google Fiber?

I worked briefly on the Fiber team when it was very young (basically from 2 weeks before to 2 weeks after launch - I was on loan from Search specifically so that they could hit their launch goals). The bottleneck when I was there were local government regulations, and in fact Kansas City was chosen because it had a unified city/county/utility regulatory authority that was very favorable to Google. To lay fiber to the home, you either need right-of-ways on the utility poles (which are owned by Google's competitors) or you need permission to dig up streets (which requires a mess of permitting from the city government). In either case, the cable & phone companies were in very tight with local regulators, and so you had hostile gatekeepers whose approval you absolutely needed.

The technology was awesome (1G Internet and HDTV!), the software all worked great, and the economics of hiring contractors to lay the fiber itself actually worked out. The big problem was regulatory capture.

With Uber & AirBnB's success in hindsight, I'd say that the way to crack the ISP business is to provide your customers with the tools to break the law en masse. For example, you could imagine an ISP startup that basically says "Here's a box, a wire, and a map of other customers' locations. Plug into their jack, and if you can convince others to plug into yours, we'll give you a discount on your monthly bill based on how many you sign up." But Google in general is not willing to break laws - they'll go right up to the boundary of what the law allows, but if a regulatory agency says "No, you can't do that", they won't do it rather than fight the agency.

Indeed, Fiber is being phased out in favor of Google's acquisition of WebPass, which does basically exactly that but with wireless instead of fiber. WebPass only requires the building owner's consent, and leaves the city out of it.

What it's like to talk at Microsoft's TechEd

I've spoken at TechEds in the US and Europe, and been in the top 10 for attendee feedback twice.

I'd never speak at TechEd again, and I told Microsoft the same thing, same reasons. The event staff is overly demanding and inconsiderate of speaker time. They repeatedly dragged me into mandatory virtual and in-person meetings to cover inane details that should have been covered via email. They mandated the color of pants speakers wore. Just ridiculously micromanaged.

Why did Hertz suddenly become so flaky?

Hertz laid off nearly the entirety of their rank and file IT staff earlier this year.

In order to receive our severance, we were forced to train our IBM replacements, who were in India. Hertz's strategy of IBM and Austerity is the new SMT's solution for a balance sheet that's in shambles, yet they have rewarded themselves by increasing executive compensation 35% over the prior year, including a $6 million bonus to the CIO.

I personally landed in an Alphabet company, received a giant raise, and now I get to work on really amazing stuff, so I'm doing fine. But to this day I'm sad to think how our once-amazing Hertz team, staffed with really smart people, led by the best boss I ever had, and were really driving the innovation at Hertz, was just thrown away like yesterday's garbage.

Before startups put clauses in contracts forbidden, they sometimes blocked sales via backchannel communications

Don't count on definitely being able to sell the stock to finance the taxes. I left after seven years in very good standing (I believed) but when I went to sell the deal was shut down [1]. Luckily I had a backup plan and I was ok [2].

[1] Had a handshake deal with an investor in the company, then the investor went silent on me. When I followed up he said the deal was "just much too small." I reached out to the company for help, and they said they'd actually told him not to buy from me. I never would have known if they hadn't decided to tell me for some reason. The takeaway is that the markets for private company stock tend to be small, and the buyers care more about their relationships with the company than they do about having your shares. Even if the stock terms allow them to buy, and they might not.

An Amazon pilot program designed to reduce the cost of interviewing

I took the first test just like the OP, the logical reasoning part seemed kind of irrelevant and a waste of time for me. That was nothing compared to the second online test.

The environment of the second test was like a scenario out of Black Mirror. Not only did they want to have the webcam and microphone on the entire time, I also had to install their custom software so the proctors could monitor my screen and control my computer. They opened up the macOS system preferences so they could disable all shortcuts to take screenshots, and they also manually closed all the background services I had running (even f.lux!).

Then they asked me to pick up my laptop and show them around my room with the webcam. They specifically asked to see the contents of my desk and the walls and ceiling of my room. I had some pencil and paper on my desk to use as scratch paper for the obvious reasons and they told me that wasn't allowed. Obviously that made me a little upset because I use it to sketch out examples and concepts. They also saw my phone on the desk and asked me to put it out of arm's reach.

After that they told me I couldn't leave the room until the 5 minute bathroom break allowed half-way through the test. I had forgotten to tell my roommate I was taking this test and he was making a bit of a ruckus playing L4D2 online (obviously a bit distracting). I asked the proctor if I could briefly leave the room to ask him to quiet down. They said I couldn't leave until the bathroom break so there was nothing I could do. Later on, I was busy thinking about a problem and had adjusted how I was sitting in my chair and moved my face slightly out of the camera's view. The proctor messaged me again telling me to move so they could see my entire face.

Amazon interviews, part 2

The first part of the interview was exactly like the linked experience. No coding questions just reasoning. The second part I had to use ProctorU instead of Proctorio. Personally I thought the experience was super weird but understandable, I'll get to that later, somebody watched me through my webcam the entire time with my microphone on. They needed to check my ID before the test. They needed me to show them the entire room I was in (which was my bedroom). My desktop computer was on behind my laptop so I turned off my computer (I don't remember if I offered to or if they asked me to) but they also asked me to cover my monitors up with something which I thought was silly after I turned them off so I covered them with a towel. They then used LogMeIn to remote into my machine so they could check running programs. I quit all my personal chat programs and pretty much only had the Chrome window running.

...

I didn't talk a real person who actually worked at Amazon (by email or through webcam) until I received an offer.

What's getting acquired by Oracle like?

[M]y company got acquired by Oracle. We thought things would be OK. Nothing changed immediately. Slowly but surely they turned the screws. 5 year laptop replacement policy. You get the corporate standard laptop and you'll like it. Sales? Oh those guys can buy new Macs every two years, they get whatever they want. Then you understand where Software Engineers rank in the company hierarchy. Oracle took the average price of our product from $100k to $5 million for the same size deals. Our sales went from $5-7m to more than $40m with no increasing in engineering headcount (team of 15). Didn't matter when bonus time came, we all got stack-ranked and some people got nothing. As a top performer I got a few options, worth maybe $5k.

Oracle exists to extract the maximum amount of money possible from the Fortune 1000. Everyone else can fuck off. Your impotent internet rage is meaningless. If it doesn't piss off the CTO of $X then it doesn't matter. If it gets that CTO to cut a bigger check then it will be embraced with extreme enthusiasm.

The culture wears down a lot (but not all) of the good people, who then leave. What's left is a lot of mediocrity and architecture astronauts. The more complex the product the better - it means extra consulting dollars!

My relative works at a business dependent on Micros. When Oracle announced the acquisition I told them to start on the backup plan immediately because Oracle was going to screw them sooner or later. A few years on and that is proving true: Oracle is slowly excising the Micros dealers and ISVs out of the picture, gobbling up all the revenue while hiking prices.

How do you avoid hiring developers who do negative work?

In practice, we have to face that all that our quest for more stringent hiring standards is not really selecting the best, but just selecting fewer people, in ways that might, or might not, have anything to do with being good at a job. Let's go through a few examples in my career:

A guy that was the most prolific developer I have ever seen: He'd rewrite entire subsystems over a weekend. The problem is that said susbsytems were not necessarily better than they started, trading bugs for bugs, and anyone that wanted to work on them would have to relearn that programmer's idiosyncrasies of the week. He easily cost his project 12 man/months of work in 4 months, the length of time it took for management to realize that he had to be let go.

A company's big UI framework was quite broken, and a new developer came in and fixed it. Great, right? Well, he was handed code review veto to changes into the framework, and his standards and his demeanor made people stop contributing after two or three attempts. In practice, the framework died as people found it antiquated, and they decided to build a new one: Well, the same developer was tasked with building new framwork, which was made mandatory for 200+ developers to use. Total contribution was clearly negative.

A developer that was very fast, and wrote working code, had been managing a rather large 500K line codebase, and received some developers as help. He didn't believe in internal documentation or on keeping interfaces stable. He also didn't believe in writing code that wasn't brittle, or in unit tests: Code changes from the new developers often broke things, the veteran would come in, fix everything in the middle of the emergency, and look absolutely great, while all the other developers looked to management as if they were incompetent. They were not, however: they were quite successful when moved to other teams. It just happens that the original developer made sure nobody else could touch anything. Eventually, the experiment was retried after the original developer was sent to do other things. It took a few months, but the new replacement team managed to modularize the code, and new people could actually modify the codebase productively.

All of those negative value developers could probably be very valuable in very specific conditions, and they'd look just fine in a tough job interview. They were still terrible hires. In my experience, if anything, a harder process that demands people to appear smarter or work faster in an interview have the opposite effect of what I'd want: They end up selecting for people that think less and do more quickly, building debt faster.

My favorite developers ever all do badly in your typical stringent Silicon Valley intervew. They work slower, do more thinking, and consider every line of code they write technical debt. They won't have a million algorithms memorized: They'll go look at sources more often than not, and will spend a lot of time on tests that might as well be documentation. Very few of those traits are positive in an interview, but I think they are vital in creating good teams, but few select for them at all.

Linux and the demise of Solaris

I worked on Solaris for over a decade, and for a while it was usually a better choice than Linux, especially due to price/performance (which includes how many instances it takes to run a given workload). It was worth fighting for, and I fought hard. But Linux has now become technically better in just about every way. Out-of-box performance, tuned performance, observability tools, reliability (on patched LTS), scheduling, networking (including TCP feature support), driver support, application support, processor support, debuggers, syscall features, etc. Last I checked, ZFS worked better on Solaris than Linux, but it's an area where Linux has been catching up. I have little hope that Solaris will ever catch up to Linux, and I have even less hope for illumos: Linux now has around 1,000 monthly contributors, whereas illumos has about 15.

In addition to technology advantages, Linux has a community and workforce that's orders of magnitude larger, staff with invested skills (re-education is part of a TCO calculation), companies with invested infrastructure (rewriting automation scripts is also part of TCO), and also much better future employment prospects (a factor than can influence people wanting to work at your company on that OS). Even with my considerable and well-known Solaris expertise, the employment prospects with Solaris are bleak and getting worse every year. With my Linux skills, I can work at awesome companies like Netflix (which I highly recommend), Facebook, Google, SpaceX, etc.

Large technology-focused companies, like Netflix, Facebook, and Google, have the expertise and appetite to make a technology-based OS decision. We have dedicated teams for the OS and kernel with deep expertise. On Netflix's OS team, there are three staff who previously worked at Sun Microsystems and have more Solaris expertise than they do Linux expertise, and I believe you'll find similar people at Facebook and Google as well. And we are choosing Linux.

The choice of an OS includes many factors. If an OS came along that was better, we'd start with a thorough internal investigation, involving microbenchmarks (including an automated suite I wrote), macrobenchmarks (depending on the expected gains), and production testing using canaries. We'd be able to come up with a rough estimate of the cost savings based on price/performance. Most microservices we have run hot in user-level applications (think 99% user time), not the kernel, so it's difficult to find large gains from the OS or kernel. Gains are more likely to come from off-CPU activities, like task scheduling and TCP congestion, and indirect, like NUMA memory placement: all areas where Linux is leading. It would be very difficult to find a large gain by changing the kernel from Linux to something else. Just based on CPU cycles, the target that should have the most attention is Java, not the OS. But let's say that somehow we did find an OS with a significant enough gain: we'd then look at the cost to switch, including retraining staff, rewriting automation software, and how quickly we could find help to resolve issues as they came up. Linux is so widely used that there's a good chance someone else has found an issue, had it fixed in a certain version or documented a workaround.

What's left where Solaris/SmartOS/illumos is better? 1. There's more marketing of the features and people. Linux develops great technologies and has some highly skilled kernel engineers, but I haven't seen any serious effort to market these. Why does Linux need to? And 2. Enterprise support. Large enterprise companies where technology is not their focus (eg, a breakfast cereal company) and who want to outsource these decisions to companies like Oracle and IBM. Oracle still has Solaris enterprise support that I believe is very competitive compared to Linux offerings.~

Why wasn't RethinkDB more sucessful?

I'd argue that where RethinkDB fell down is on a step you don't list, "Understand the context of the problem", which you'd ideally do before figuring out how many people it's a problem for. Their initial idea was a MySQL storage engine for SSDs - the environmental change was that SSD prices were falling rapidly, SSDs have wildly different performance characteristics from disk, and so they figured there was an opportunity to catch the next wave. Only problem is that the biggest corporate buyers of SSDs are gigantic tech companies (eg. Google, Amazon) with large amounts of proprietary software, and so a generic MySQL storage engine isn't going to be useful to them anyway.

Unfortunately they'd already taken funding, built a team, and written a lot of code by the time they found that out, and there's only so far you can pivot when you have an ecosystem like that.

On falsehoods programmers believe about X

This unfortunately follows the conventions of the genre called "Falsehood programmers believe about X": ...

I honestly think this genre is horrible and counterproductive, even though the writer's intentions are good. It gives no examples, no explanations, no guidelines for proper implementations - just a list of condescending gotchas, showing off the superior intellect and perception of the author.

What does it mean if a company rescinds an offer because you tried to negotiate?

It happens sometimes. Usually it's because of one of two situations:

1) The company was on the fence about wanting you anyway, and negotiating takes you from the "maybe kinda sorta want to work with" to the "don't want to work with" pile.

2) The company is looking for people who don't question authority and don't stick up for their own interests.

Both of these are red flags. It's not really a matter of ethics - they're completely within their rights to withdraw an offer for any reason - but it's a matter of "Would you really want to work there anyway?" For both corporations and individuals, it usually leads to a smoother life if you only surround yourself with people who really value you.

HN comments

I feel like this is every HN discussion about "rates---comma---raising them": a mean-spirited attempt to convince the audience on the site that high rates aren't really possible, because if they were, the person telling you they're possible would be wealthy beyond the dreams of avarice. Once again: Patrick is just offering a more refined and savvy version of advice me and my Matasano friends gave him, and our outcomes are part of the record of a reasonable large public company.

This, by the way, is why I'll never write this kind of end-of-year wrap-up post (and, for the same reasons, why I'll never open source code unless I absolutely have to). It's also a big part of what I'm trying to get my hands around for the Starfighter wrap-up post. When we started Starfighter, everyone said "you're going to have such an amazing time because of all the HN credibility you have". But pretty much every time Starfighter actually came up on HN, I just wanted to hide under a rock. Even when the site is civil, it's still committed to grind away any joy you take either in accomplishing something near or even in just sharing something interesting you learned . You could sort of understand an atavistic urge to shit all over someone sharing an interesting experience that was pleasant or impressive. There's a bad Morrissey song about that. But look what happens when you share an interesting story that obviously involved significant unpleasantness and an honest accounting of one's limitations: a giant thread full of people piling on to question your motives and life choices. You can't win.

On the journalistic integrity of Quartz

I was the first person to be interviewed by this journalist (Michael Thomas @curious_founder). He approached me on Twitter to ask questions about digital nomad and remote work life (as I founded Nomad List and have been doing it for years).

I told him it'd be great to see more honest depictions as most articles are heavily idealized making it sound all great, when it's not necessarily. It's ups and downs (just like regular life really).

What happened next may surprise you. He wrote a hit piece on me changing my entire story that I told him over Skype into a clickbait article of how digital nomadism doesn't work and one of the main people doing it for awhile (en public) even settled down and gave up altogether.

I didn't settle down. I spent the summer in Amsterdam. Cause you know, it's a nice place! But he needed to say this to make a polarized hit piece with an angle. And that piece became viral. Resulting in me having to tell people daily that I didn't and getting lots of flack. You may understand it doesn't help if your entire startup is about something and a journalist writes a viral piece how you yourself don't even believe in that anymore. I contacted the journalist and Quartz but they didn't change a thing.

It's great this meant his journalistic breakthrough but it hurt me in the process.

I'd argue journalists like this are the whole problem we have these days. The articles they write can't be balanced because they need to get pageviews. Every potential to write something interesting quickly turns into clickbait. It turned me off from being interviewed ever again. Doing my own PR by posting comment sections of Hacker News or Reddit seems like a better idea (also see how Elon Musk does exactly this, seems smarter).

How did Click and Clack always manage to solve the problem?

Hope this doesn't ruin it for you, but I knew someone who had a problem presented on the show. She called in and reached an answering machine. Someone called her and qualified the problem. Then one of the brothers called and talked to her for a while. Then a few weeks later (there might have been some more calls, I don't know) both brothers called her and talked to her for a while. Her parts of that last call was edited into the radio show so it sounded like she had called and they just figured out the answer on the spot.

Why are so many people down on blockchain?

Blockchain is the world's worst database, created entirely to maintain the reputations of venture capital firms who injected hundreds of millions of dollars into a technology whose core defining insight was "You can improve on a Ponzi scam by making it self-organizing and distributed; that gets vastly more distribution, reduces the single point of failure, and makes it censorship-resistant."

That's more robust than I usually phrase things on HN, but you did ask. In slightly more detail:

Databases are wonderful things. We have a number which are actually employed in production, at a variety of institutions. They run the world. Meaningful applications run on top of Postgres, MySQL, Oracle, etc etc.

No meaningful applications run on top of "blockchain", because it is a marketing term. You cannot install blockchain just like you cannot install database. (Database sounds much cooler without the definitive article, too.) If you pick a particular instantiation of a blockchain-style database, it is a horrible, horrible database.

Can I pick on Bitcoin? Let me pick on Bitcoin. Bitcoin is claimed to be a global financial network and ready for production right now. Bitcoin cannot sustain 5 transactions per second, worldwide.

You might be sensibly interested in Bitcoin governance if, for some reason, you wanted to use Bitcoin. Bitcoin is a software artifact; it matters to users who makes changes to it and by what process. (Bitcoin is a software artifact, not a protocol, even though the Bitcoin community will tell you differently. There is a single C++ codebase which matters. It is essentially impossible to interoperate with Bitcoin without bugs-and-all replicating that codebase.) Bitcoin governance is captured by approximately ~5 people. This is a robust claim and requires extraordinary evidence.

Ordinary evidence would be pointing you, in a handwavy fashion, about the depth of acrimony with regards to raising the block size, which would let Bitcoin scale to the commanding heights of 10 or, nay, 100 transactions per second worldwide.

Extraordinary evidence might be pointing you to the time where the entire Bitcoin network was de-facto shut down based on the consensus of N people in an IRC channel. c.f. https://news.ycombinator.com/item?id=9320989 This was back in 2013. Long story short: a software update went awry so they rolled back global state by a few hours by getting the right two people to agree to it on a Skype call.

But let's get back to discussing that sole technical artifact. Bitcoin has a higher cost-to-value ratio than almost any technology conceivable; the cost to date is the market capitalization of Bitcoin. Because Bitcoin enters through a seigniorage mechanism, every Bitcoin existing was minted as compensation for "security the integrity of the blockchain" (by doing computationally expensive makework).

This cost is high. Today, routine maintenance of the Bitcoin network will cost the network approximately $1.5 million. That's on the order of $3 per write on a maximum committed capacity basis. It will cost another $1.5 million tomorrow, exchange rate depending.

(Bitcoin has successfully shifted much of the cost of operating its database to speculators rather than people who actually use Bitcoin for transaction processing. That game of musical chairs has gone on for a while.)

Bitcoin has some properties which one does not associate with many databases. One is that write acknowledgments average 5 minutes. Another is that they can stop, non-deterministically, for more than an hour at a time, worldwide, for all users simultaneously. This behavior is by design.

How big is the proprietary database market?

1. The database market is NOT closed. In fact, we are in a database boom. Since 2009 (the year RethinkDB was founded), there have been over 100 production grade databases released in the market. These span document stores, Key/Value, time series, MPP, relational, in-memory, and the ever increasing "multi model databases."

2. Since 2009, over $600 MILLION dollars (publicly announced) has been invested in these database companies (RethinkDB represents 12.2M or about 2%). That's aside from money invested in the bigger established databases.

3. Almost all of the companies that have raised funding in this period generate revenue from one of more of the following areas:

a) exclusive hosting (meaning AWS et al. do not offer this product) b) multi-node/cluster support c) product enhancements c) enterprise support

Looking at each of the above revenue paths as executed by RethinkDB:

a) RethinkDB never offered a hosted solution. Compose offered a hosted solution in October of 2014. b) RethinkDB didn't support true high availability until the 2.1 release in August 2015. It was released as open source and to my knowledge was not monetized. c/d) I've heard that an enterprise version of RethinkDB was offered near the end. Enterprise Support is, empirically, a bad approach for a venture backed company. I don't know that RethinkDB ever took this avenue seriously. Correct me if I am wrong.

A model that is not popular among RECENT databases but is popular among traditional databases is a standard licensing model (e.g. Oracle, Microsoft SQL Server). Even these are becoming more rare with the advent of A, but never underestimate the licensing market.

Again, this is complete conjecture, but I believe RethinkDB failed for a few reasons:

1) not pursuing one of the above revenue models early enough. This has serious affects on the order of the feature enhancements (for instance, the HA released in 2015 could have been released earlier at a premium or to help facilitate a hosted solution).

2) incorrect priority of enhancements:

2a) general database performance never reached the point it needed to. RethinkDB struggled with both write and read performance well into 2015. There was no clear value add in this area compared to many write or read focused databases released around this time.

2b) lack of (proper) High Availability for too long.

2c) ReQL was not necessary - most developers use ORMs when interacting with SQL. When you venture into analytical queries, we actually seem to make great effort to provide SQL: look at the number of projects or companies that exist to bring SQL to databases and filesystems that don't support it (Hive, Pig, Slam Data, etc).

2d) push notifications. This has not been demonstrated to be a clear market need yet. There are a small handful of companies that promoting development stacks around this, but no database company is doing the same.

2e) lack of focus. What was RethinkDB REALLY good at? It push ReQL and joins at first, but it lacked HA until 2015, struggled with high write or read loads into 2015. It then started to focus on real time notifications. Again, there just aren't many databases focusing on these areas.

My final thought is that RethinkDB didn't raise enough capital. Perhaps this is because of previous points, but without capital, the above can't be corrected. RethinkDB actually raised far less money than basically any other venture backed company in this space during this time.

Again, I've never run a database company so my thoughts are just from an outsider. However, I am the founder of a company that provides database integration products so I monitor this industry like I hawk. I simply don't agree that the database market has been "captured."

I expect to see even bigger growth in databases in the future. I'm happy to share my thoughts about what types of databases are working and where the market needs solutions. Additionally, companies are increasingly relying on third part cloud services for data they previously captured themselves. Anything from payment processes, order fulfillment, traffic analytics etc is now being handled by someone else.

A Google Maps employee's opinion on the Google Maps pricing change

I was a googler working on Google maps at the time of the API self immolation.

There were strong complaints from within about the price changes. Obviously everyone couldn't believe what was being planned, and there were countless spreadsheets and reports and SQL queries showing how this was going to shit all over a lot of customers that we'd be guaranteed to lose to a competitor.

Management didn't give a shit.

I don't know what the rationale was apart from some vague claim about "charging for value". A lot of users of the API apparently were basically under the free limits or only spending less than 100 USD on API usage so I can kind of understand the line of thought, but I still.thibk they went way too far.

I don't know what happened to the architects of the plan. I presume promo.

Edit: I should add that this was not a knee-jerk thing or some exec just woke up one day with an idea in their dreams. It was a planned change that took many months to plan and prepare for with endless preparations and reporting and so on.

A response to a PL-centric view of what's wrong with Protobufs

Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways.

This article appears to be written by a programming language design theorist who, unfortunately, does not understand (or, perhaps, does not value) practical software engineering. Type theory is a lot of fun to think about, but being simple and elegant from a type theory perspective does not necessarily translate to real value in real systems. Protobuf has undoubtedly, empirically proven its real value in real systems, despite its admittedly large number of warts.

The main thing that the author of this article does not seem to understand -- and, indeed, many PL theorists seem to miss -- is that the main challenge in real-world software engineering is not writing code but changing code once it is written and deployed. In general, type systems can be both helpful and harmful when it comes to changing code -- type systems are invaluable for detecting problems introduced by a change, but an overly-rigid type system can be a hindrance if it means common types of changes are difficult to make.

This is especially true when it comes to protocols, because in a distributed system, you cannot update both sides of a protocol simultaneously. I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths, leading to ugly and hard-to-test code. Inevitably, developers are pushed towards hacks in order to avoid protocol changes, which makes things worse.

I don't have time to address all the author's points, so let me choose a few that I think are representative of the misunderstanding.

Make all fields in a message required. This makes messages product types.

Promote oneof fields to instead be standalone data types. These are coproduct types.

This seems to miss the point of optional fields. Optional fields are not primarily about nullability but about compatibility. Protobuf's single most important feature is the ability to add new fields over time while maintaining compatibility. This has proven -- in real practice, not in theory -- to be an extremely powerful way to allow protocol evolution. It allows developers to build new features with minimal work.

Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time, hence the "required considered harmful" manifesto. In practice, you want to declare all fields optional to give yourself maximum flexibility for change.

The author dismisses this later on:

What protobuffers are is permissive. They manage to not shit the bed when receiving messages from the past or from the future because they make absolutely no promises about what your data will look like. Everything is optional! But if you need it anyway, protobuffers will happily cook up and serve you something that typechecks, regardless of whether or not it's meaningful.

In real world practice, the permissiveness of Protocol Buffers has proven to be a powerful way to allow for protocols to change over time.

Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is. Certainly the usual proposals I see seem like steps backwards. I'd love to be proven wrong, but not on the basis of perceived elegance and simplicity, but rather in real-world use.

oneof fields can't be repeated.

(background: A "oneof" is essentially a tagged union -- a "sum type" for type theorists. A "repeated field" is an array.)

Two things:

1. It's that way because the "oneof" pattern long-predates the "oneof" language construct. A "oneof" is actually syntax sugar for a bunch of "optional" fields where exactly one is expected to be filled in. Lots of protocols used this pattern before I added "oneof" to the language, and I wanted those protocols to be able to upgrade to the new construct without breaking compatibility.

You might argue that this is a side-effect of a system evolving over time rather than being designed, and you'd be right. However, there is no such thing as a successful system which was designed perfectly upfront. All successful systems become successful by evolving, and thus you will always see this kind of wart in anything that works well. You should want a system that thinks about its existing users when creating new features, because once you adopt it, you'll be an existing user.

1. You actually do not want a oneof field to be repeated!

Here's the problem: Say you have your repeated "oneof" representing an array of values where each value can be one of 10 different types. For a concrete example, let's say you're writing a parser and they represent tokens (number, identifier, string, operator, etc.).

Now, at some point later on, you realize there's some additional piece of data you want to attach to every element. In our example, it could be that you now want to record the original source location (line and column number) where the token appeared.

How do you make this change without breaking compatibility? Now you wish that you had defined your array as an array of messages, each containing a oneof, so that you could add a new field to that message. But because you didn't, you're probably stuck creating a parallel array to store your new field. That sucks.

In every single case where you might want a repeated oneof, you always want to wrap it in a message (product type), and then repeat that. That's exactly what you can do with the existing design.

The author's complaints about several other features have similar stories.

One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Surely that's a win, isn't it?

Granted, on paper it's a cool feature. But I've never once seen an application that will actually preserve that property.

OK, well, I've worked on lots of systems -- across three different companies -- where this feature is essential.

On the last bit, some comments note "I wonder if author uses Chrome, which depends heavily on this in its Sync feature", "When I worked on Chrome Sync, we spent some time making sure that unknown fields were preserved properly. Glad to see that someone noticed, cheers!", "Yeah, most big Google services -- including Search -- rely pretty heavily on unknown field retention. Google has been building large services out of microservices since a decade before anyone ever said the word "microservice". When one service is updated to emit a new field, and another service is updated to consume it, it's important that the feature can then work, without updating all the middlemen.", etc.

Why do people keep getting incorrectly pulled over after being tagged by Flock cameras?

This is exactly why we took our cameras down in nearby Oak Park in Chicagoland.

People talk online as if this is a Flock technology issue, but it only peripherally is. The core problem is that the hotlist systems states set up were never intended for real-time enforcement; there's a great deal of imprecision in them that tends not to matter in human-scale enforcement, but matters a great deal when enforcement is automated.

This distinction is important because there's basically not much Flock can do about it. The source of truth about which cars should be curbed isn't Flock's and can't be Flock's; it's core law enforcement data that belongs to and is the responsibility of the state. Companies like Flock can presumably rig up Rube Goldberg devices to double-check and cross-check hotlist data, but their goals are in tension: they have a window of just minutes to detect and interdict a car. Our police department pledged to do voice confirmations on these hits, and everyone knew that wasn't going to work.

In our specific case, we curbed more innocent vehicles than stolen ones (though not at gunpoint). In each bogus case, Illinois LEADS had left a car recorded "stolen" weeks after it had been "recovered"; "recovered" in scare-quotes because many (most?) of these cars were never stolen to begin with, but rather driven off by a relative without notice. More context on why these data sources are unsuited for real-time enforcement!

Unfortunately, every law enforcement ALPR deployment in the country is a dragnet for all curb-able offenses, from stolen cars to failure-to-appear warrants (the plurality of arrests under our system, which turned our police force into municipal debt collectors for neighboring suburbs). The starting point for mitigating this problem would be to drastically reduce the set of offenses you alert on --- and that's tricky, because your instinct is to reduce them to the most serious offenses, but what you really need is the set of offenses least likely to result in false positives. We settled on "stolen license plate" (no innocent explanation) before giving up on the endeavor entirely.

???

How did HN get get the commenter base that it has? If you read HN, on any given week, there are at least as many good, substantial, comments as there are posts. This is different from every other modern public news aggregator I can find out there, and I don’t really know what the ingredients are that make HN successful.

For the last couple years (ish?), the moderation regime has been really active in trying to get a good mix of stories on the front page and in tamping down on gratuitously mean comments. But there was a period of years where the moderation could be described as sparse, arbitrary, and capricious, and while there are fewer “bad” comments now, it doesn’t seem like good moderation actually generates more “good” comments.

The ranking scheme seems to penalize posts that have a lot of comments on the theory that flamebait topics will draw a lot of comments. That sometimes prematurely buries stories with good discussion, but much more often, it buries stories that draw pointless flamewars. If you just read HN, it’s hard to see the effect, but if you look at forums that use comments as a positive factor in ranking, the difference is dramatic -- those other forums that boost topics with many comments (presumably on theory that vigorous discussion should be highlighted) often have content-free flame wars pinned at the top for long periods of time.

Something else that HN does that’s different from most forums is that user flags are weighted very heavily. On reddit, a downvote only cancels out an upvote, which means that flamebait topics that draw a lot of upvotes like “platform X is cancer” “Y is doing some horrible thing” often get pinned to the top of r/programming for a an entire day, since the number of people who don’t want to see that is drowned out by the number of people who upvote outrageous stories. If you read the comments for one of the "X is cancer" posts on r/programming, the top comment will almost inevitably that the post has no content, that the author of the post is a troll who never posts anything with content, and that we'd be better off with less flamebait by the author at the top of r/programming. But the people who will upvote outrage porn outnumber the people who will downvote it, so that kind of stuff dominates aggregators that use raw votes for ranking. Having flamebait drop off the front page quickly is significant, but it doesn’t seem sufficient to explain why there are so many more well-informed comments on HN than on other forums with roughly similar traffic.

Maybe the answer is that people come to HN for the same reason people come to Silicon Valley -- despite all the downsides, there’s a relatively large concentration of experts there across a wide variety of CS-related disciplines. If that’s true, and it’s a combination of path dependence on network effects, that’s pretty depressing since that’s not replicable.

If you liked this curated list of comments, you'll probably also like this list of books and this list of blogs.

This is part of an experiment where I write up thoughts quickly, without proofing or editing. Apologies if this is less clear than a normal post. This is probably going to be the last post like this, for now, since, by quickly writing up a post whenever I have something that can be written up quickly, I'm building up a backlog of post ideas that require re-reading the literature in an area or running experiments.

P.S. Please suggest other good comments! By their nature, HN comments are much less discoverable than stories, so there are a lot of great coments that I haven't seen.

A couple years ago, I took a road trip from Wisconsin to Washington and mostly stayed in rural hotels on the way. I expected the internet in rural areas too sparse to have cable internet to be slow, but I was still surprised that a large fraction of the web was inaccessible. Some blogs with lightweight styling were readable, as were pages by academics who hadn’t updated the styling on their website since 1995. But very few commercial websites were usable (other than Google). When I measured my connection, I found that the bandwidth was roughly comparable to what I got with a 56k modem in the 90s. The latency and packetloss were significantly worse than the average day on dialup: latency varied between 500ms and 1000ms and packetloss varied between 1% and 10%. Those numbers are comparable to what I’d see on dialup on a bad day.

Despite my connection being only a bit worse than it was in the 90s, the vast majority of the web wouldn’t load. Why shouldn’t the web work with dialup or a dialup-like connection? It would be one thing if I tried to watch youtube and read pinterest. It’s hard to serve videos and images without bandwidth. But my online interests are quite boring from a media standpoint. Pretty much everything I consume online is plain text, even if it happens to be styled with images and fancy javascript. In fact, I recently tried using w3m (a terminal-based web browser that, by default, doesn’t support css, javascript, or even images) for a week and it turns out there are only two websites I regularly visit that don’t really work in w3m (twitter and zulip, both fundamentally text based sites, at least as I use them)¹.

More recently, I was reminded of how poorly the web works for people on slow connections when I tried to read a joelonsoftware post while using a flaky mobile connection. The HTML loaded but either one of the five CSS requests or one of the thirteen javascript requests timed out, leaving me with a broken page. Instead of seeing the article, I saw three entire pages of sidebar, menu, and ads before getting to the title because the page required some kind of layout modification to display reasonably. Pages are often designed so that they're hard or impossible to read if some dependency fails to load. On a slow connection, it's quite common for at least one depedency to fail. After refreshing the page twice, the page loaded as it was supposed to and I was able to read the blog post, a fairly compelling post on eliminating dependencies.

Complaining that people don’t care about performance like they used to and that we’re letting bloat slow things down for no good reason is “old man yells at cloud” territory; I probably sound like that dude who complains that his word processor, which used to take 1MB of RAM, takes 1GB of RAM. Sure, that could be trimmed down, but there’s a real cost to spending time doing optimization and even a $300 laptop comes with 2GB of RAM, so why bother? But it’s not quite the same situation -- it’s not just nerds like me who care about web performance. When Microsoft looked at actual measured connection speeds, they found that half of Americans don't have broadband speed. Heck, AOL had 2 million dial-up subscribers in 2015, just AOL alone. Outside of the U.S., there are even more people with slow connections. I recently chatted with Ben Kuhn, who spends a fair amount of time in Africa, about his internet connection:

I've seen ping latencies as bad as ~45 sec and packet loss as bad as 50% on a mobile hotspot in the evenings from Jijiga, Ethiopia. (I'm here now and currently I have 150ms ping with no packet loss but it's 10am). There are some periods of the day where it ~never gets better than 10 sec and ~10% loss. The internet has gotten a lot better in the past ~year; it used to be that bad all the time except in the early mornings.

…

Speedtest.net reports 2.6 mbps download, 0.6 mbps upload. I realized I probably shouldn't run a speed test on my mobile data because bandwidth is really expensive.

Our server in Ethiopia is has a fiber uplink, but it frequently goes down and we fall back to a 16kbps satellite connection, though I think normal people would just stop using the Internet in that case.

If you think browsing on a 56k connection is bad, try a 16k connection from Ethiopia!

Everything we’ve seen so far is anecdotal. Let’s load some websites that programmers might frequent with a variety of simulated connections to get data on page load times. webpagetest lets us see how long it takes a web site to load (and why it takes that long) from locations all over the world. It even lets us simulate different kinds of connections as well as load sites on a variety of mobile devices. The times listed in the table below are the time until the page is “visually complete”; as measured by webpagetest, that’s the time until the above-the-fold content stops changing.

Each row is a website. For sites that support both plain HTTP as well as HTTPS, both were tested; URLs are HTTPS except where explicitly specified as HTTP. The first two columns show the amount of data transferred over the wire in MB (which includes headers, handshaking, compression, etc.) and the number of TCP connections made. The rest of the columns show the time in seconds to load the page on a variety of connections from fiber (FIOS) to less good connections. “Bad” has the bandwidth of dialup, but with 1000ms ping and 10% packetloss, which is roughly what I saw when using the internet in small rural hotels. “😱” simulates a 16kbps satellite connection from Jijiga, Ethiopia. Rows are sorted by the measured amount of data transferred.

The timeout for tests was 6 minutes; anything slower than that is listed as FAIL. Pages that failed to load are also listed as FAIL. A few things that jump out from the table are:

1. A large fraction of the web is unusable on a bad connection. Even on a good (0% packetloss, no ping spike) dialup connection, some sites won’t load.
2. Some sites will use a lot of data!

The web on bad connections

As commercial websites go, Google is basically as good as it gets for people on a slow connection. On dialup, the 50%-ile page load time is a minute and a half. But at least it loads -- when I was on a slow, shared, satellite connection in rural Montana, virtually no commercial websites would load at all. I could view websites that only had static content via Google cache, but the live site had no hope of loading.

Some sites will use a lot of data

Although only two really big sites were tested here, there are plenty of sites that will use 10MB or 20MB of data. If you’re reading this from the U.S., maybe you don’t care, but if you’re browsing from Mauritania, Madagascar, or Vanuatu, loading codinghorror once will cost you more than 10% of the daily per capita GNI.

Page weight matters

Despite the best efforts of Maciej, the meme that page weight doesn’t matter keeps getting spread around. AFAICT, the top HN link of all time on web page optimization is to an article titled “Ludicrously Fast Page Loads - A Guide for Full-Stack Devs”. At the bottom of the page, the author links to another one of his posts, titled “Page Weight Doesn’t Matter”.

Usually, the boogeyman that gets pointed at is bandwidth: users in low-bandwidth areas (3G, developing world) are getting shafted. But the math doesn’t quite work out. Akamai puts the global connection speed average at 3.9 megabits per second.

The “ludicrously fast” guide fails to display properly on dialup or slow mobile connections because the images time out. On reddit, it also fails under load: "Ironically, that page took so long to load that I closed the window.", "a lot of … gifs that do nothing but make your viewing experience worse", "I didn't even make it to the gifs; the header loaded then it just hung.", etc.

The flaw in the “page weight doesn’t matter because average speed is fast” is that if you average the connection of someone in my apartment building (which is wired for 1Gbps internet) and someone on 56k dialup, you get an average speed of 500 Mbps. That doesn’t mean the person on dialup is actually going to be able to load a 5MB website. The average speed of 3.9 Mbps comes from a 2014 Akamai report, but it’s just an average. If you look at Akamai’s 2016 report, you can find entire countries where more than 90% of IP addresses are slower than that!

Yes, there are a lot of factors besides page weight that matter, and yes it's possible to create a contrived page that's very small but loads slowly, as well as a huge page that loads ok because all of the weight isn't blocking, but total page weight is still pretty decently correlated with load time.

Since its publication, the "ludicrously fast" guide was updated with some javascript that only loads images if you scroll down far enough. That makes it look a lot better on webpagetest if you're looking at the page size number (if webpagetest isn't being scripted to scroll), but it's a worse user experience for people on slow connections who want to read the page. If you're going to read the entire page anyway, the weight increases, and you can no longer preload images by loading the site. Instead, if you're reading, you have to stop for a few minutes at every section to wait for the images from that section to load. And that's if you're lucky and the javascript for loading images didn't fail to load.

The average user fallacy

Just like many people develop with an average connection speed in mind, many people have a fixed view of who a user is. Maybe they think there are customers with a lot of money with fast connections and customers who won't spend money on slow connections. That is, very roughly speaking, perhaps true on average, but sites don't operate on average, they operate in particular domains. Jamie Brandon writes the following about his experience with Airbnb:

I spent three hours last night trying to book a room on airbnb through an overloaded wifi and presumably a satellite connection. OAuth seems to be particularly bad over poor connections. Facebook's OAuth wouldn't load at all and Google's sent me round a 'pick an account' -> 'please reenter you password' -> 'pick an account' loop several times. It took so many attempts to log in that I triggered some 2fa nonsense on airbnb that also didn't work (the confirmation link from the email led to a page that said 'please log in to view this page') and eventually I was just told to send an email to account.disabled@airbnb.com, who haven't replied.

It's particularly galling that airbnb doesn't test this stuff, because traveling is pretty much the whole point of the site so they can't even claim that there's no money in servicing people with poor connections.

What about tail latency?

My original plan for this was post was to show 50%-ile, 90%-ile, 99%-ile, etc., tail load times. But the 50%-ile results are so bad that I don’t know if there’s any point to showing the other results. If you were to look at the 90%-ile results, you’d see that most pages fail to load on dialup and the “Bad” and “😱” connections are hopeless for almost all sites.

HTTP vs HTTPs

You can see that for a very small site that doesn’t load many blocking resources, HTTPS is noticeably slower than HTTP, especially on slow connections. Practically speaking, this doesn’t matter today because virtually no sites are that small, but if you design a web site as if people with slow connections actually matter, this is noticeable.

How to make pages usable on slow connections

The long version is, to really understand what’s going on, considering reading high-performance browser networking, a great book on web performance that’s avaiable for free.

The short version is that most sites are so poorly optimized that someone who has no idea what they’re doing can get a 10x improvement in page load times for a site whose job is to serve up text with the occasional image. When I started this blog in 2013, I used Octopress because Jekyll/Octopress was the most widely recommended static site generator back then. A plain blog post with one or two images took 11s to load on a cable connection because the Octopress defaults included multiple useless javascript files in the header (for never-used-by-me things like embedding flash videos and delicious integration), which blocked page rendering. Just moving those javascript includes to the footer halved page load time, and making a few other tweaks decreased page load time by another order of magnitude. At the time I made those changes, I knew nothing about web page optimization, other than what I heard during a 2-minute blurb on optimization from a 40-minute talk on how the internet works and I was able to get a 20x speedup on my blog in a few hours. You might argue that I’ve now gone too far and removed too much CSS, but I got a 20x speedup for people on fast connections before making changes that affected the site’s appearance (and the speedup on slow connections was much larger).

That’s normal. Popular themes for many different kinds of blogging software and CMSs contain anti-optimizations so blatant that any programmer, even someone with no front-end experience, can find large gains by just pointing webpagetest at their site and looking at the output.

What about browsers?

While it's easy to blame page authors because there's a lot of low-hanging fruit on the page side, there's just as much low-hanging fruit on the browser side. Why does my browser open up 6 TCP connections to try to download six images at once when I'm on a slow satellite connection? That just guarantees that all six images will time out! Even if I tweak the timeout on the client side, servers that are configured to protect against DoS attacks won't allow long lived connections that aren't doing anything. I can sometimes get some images to load by refreshing the page a few times (and waiting ten minutes each time), but why shouldn't the browser handle retries for me? If you think about it for a few minutes, there are a lot of optimiztions that browsers could do for people on slow connections, but because they don't, the best current solution for users appears to be: use w3m when you can, and then switch to a browser with ad-blocking when that doesn't work. But why should users have to use two entirely different programs, one of which has a text-based interface only computer nerds will find palatable?

Conclusion

When I was at Google, someone told me a story about a time that “they” completed a big optimization push only to find that measured page load times increased. When they dug into the data, they found that the reason load times had increased was that they got a lot more traffic from Africa after doing the optimizations. The team’s product went from being unusable for people with slow connections to usable, which caused so many users with slow connections to start using the product that load times actually increased.

Last night, at a presentation on the websockets protocol, Gary Bernhardt made the observation that the people who designed the websockets protocol did things like using a variable length field for frame length to save a few bytes. By contrast, if you look at the Alexa top 100 sites, almost all of them have a huge amount of slop in them; it’s plausible that the total bandwidth used for those 100 sites is probably greater than the total bandwidth for all websockets connections combined. Despite that, if we just look at the three top 35 sites tested in this post, two send uncompressed javascript over the wire, two redirect the bare domain to the www subdomain, and two send a lot of extraneous information by not compressing images as much as they could be compressed without sacrificing quality. If you look at twitter, which isn’t in our table but was mentioned above, they actually do an anti-optimization where, if you upload a PNG which isn’t even particularly well optimized, they’ll re-encode it as a jpeg which is larger and has visible artifacts!

“Use bcrypt” has become the mantra for a reasonable default if you’re not sure what to do when storing passwords. The web would be a nicer place if “use webpagetest” caught on in the same way. It’s not always the best tool for the job, but it sure beats the current defaults.

Appendix: experimental caveats

The above tests were done by repeatedly loading pages via a private webpagetest image in AWS west 2, on a c4.xlarge VM, with simulated connections on a first page load in Chrome with no other tabs open and nothing running on the VM other than the webpagetest software and the browser. This is unrealistic in many ways.

In relative terms, this disadvantages sites that have a large edge presence. When I was in rural Montana, I ran some tests and found that I had noticeably better latency to Google than to basically any other site. This is not reflected in the test results. Furthermore, this setup means that pages are nearly certain to be served from a CDN cache. That shouldn't make any difference for sites like Google and Amazon, but it reduces the page load time of less-trafficked sites that aren't "always" served out of cache. For example, when I don't have a post trending on social media, between 55% and 75% of traffic is served out of a CDN cache, and when I do have something trending on social media, it's more like 90% to 99%. But the test setup means that the CDN cache hit rate during the test is likely to be > 99% for my site and other blogs which aren't so widely read that they'd normally always have a cached copy available.

All tests were run assuming a first page load, but it’s entirely reasonable for sites like Google and Amazon to assume that many or most of their assets are cached. Testing first page load times is perhaps reasonable for sites with a traffic profile like mine, where much of the traffic comes from social media referrals of people who’ve never visited the site before.

A c4.xlarge is a fairly powerful machine. Today, most page loads come from mobile and even the fastest mobile devices aren’t as fast as a c4.xlarge; most mobile devices are much slower than the fastest mobile devices. Most desktop page loads will also be from a machine that’s slower than a c4.xlarge. Although the results aren’t shown, I also ran a set of tests using a t2.micro instance: for simple sites, like mine, the difference was negligible, but for complex sites, like Amazon, page load times were as much as 2x worse. As you might expect, for any particular site, the difference got smaller as the connection got slower.

As Joey Hess pointed out, many dialup providers attempt to do compression or other tricks to reduce the effective weight of pages and none of these tests take that into account.

Firefox, IE, and Edge often have substantially different performance characteristics from Chrome. For that matter, different versions of Chrome can have different performance characteristics. I just used Chrome because it’s the most widely used desktop browser, and running this set of tests took over a full day of VM time with a single-browser.

The simulated bad connections add a constant latency and fixed (10%) packetloss. In reality, poor connections have highly variable latency with peaks that are much higher than the simulated latency and periods of much higher packetloss than can last for minutes, hours, or days. Putting 😱 at the rightmost side of the table may make it seem like the worst possible connection, but packetloss can get much worse.

Similarly, while codinghorror happens to be at the bottom of the page, it's nowhere to being the slowest loading page. Just for example, I originally considered including slashdot in the table but it was so slow that it caused a significant increase in total test run time because it timed out at six minutes so many times. Even on FIOS it takes 15s to load by making a whopping 223 requests over 100 TCP connections despite weighing in at "only" 1.9MB. Amazingly, slashdot also pegs the CPU at 100% for 17 entire seconds while loading on FIOS. In retrospect, this might have been a good site to include because it's pathologically mis-optimized sites like slashdot that allow the "page weight doesn't matter" meme to sound reasonable.

The websites compared don't do the same thing. Just looking at the blogs, some blogs put entire blog entries on the front page, which is more convenient in some ways, but also slower. Commercial sites are even more different -- they often can't reasonably be static sites and have to have relatively large javascrit payloads in order to work well.

Appendix: irony

The main table in this post is almost 50kB of HTML (without compression or minification); that’s larger than everything else in this post combined. That table is curiously large because I used a library (pandas) to generate the table instead of just writing a script to do it by hand, and as we know, the default settings for most libraries generate a massive amount of bloat. It didn’t even save time because every single built-in time-saving feature that I wanted to use was buggy, which forced me to write all of the heatmap/gradient/styling code myself anyway! Due to laziness, I left the pandas table generating scaffolding code, resulting in a table that looks like it’s roughly an order of magnitude larger than it needs to be.

This isn't a criticism of pandas. Pandas is probably quite good at what it's designed for; it's just not designed to produce slim websites. The CSS class names are huge, which is reasonable if you want to avoid accidental name collisions for generated CSS. Almost every td, th, and tr element is tagged with a redundant rowspan=1 or colspan=1, which is reasonable for generated code if you don't care about size. Each cell has its own CSS class, even though many cells share styling with other cells; again, this probably simplified things on the code generation. Every piece of bloat is totally reasonable. And unfortunately, there's no tool that I know of that will take a bloated table and turn it into a slim table. A pure HTML minifier can't change the class names because it doesn't know that some external CSS or JS doesn't depend on the class name. An HTML minifier could theoretically determine that different cells have the same styling and merge them, except for the aforementioned problem with potential but non-existent external depenencies, but that's beyond the capability of the tools I know of.

For another level of ironic, consider that while I think of a 50kB table as bloat, this page is 12kB when gzipped, even with all of the bloat. Google's AMP currently has > 100kB of blocking javascript that has to load before the page loads! There's no reason for me to use AMP pages because AMP is slower than my current setup of pure HTML with a few lines of embedded CSS and the occasional image, but, as a result, I'm penalized by Google (relative to AMP pages) for not "accelerating" (deccelerating) my page with AMP.

Thanks to Leah Hanson, Jason Owen, Ethan Willis, and Lindsey Kuper for comments/corrections

I often talk to startups that claim that their compensation package has a higher expected value than the equivalent package at a place like Facebook, Google, Twitter, or Snapchat. One thing I don’t understand about this claim is, if the claim is true, why shouldn’t the startup go to an investor, sell their options for what they claim their options to be worth, and then pay me in cash? The non-obvious value of options combined with their volatility is a barrier for recruiting.

Additionally, given my risk function and the risk function of VCs, this appears to be a better deal for everyone. Like most people, extra income gives me diminishing utility, but VCs have an arguably nearly linear utility in income. Moreover, even if VCs shared my risk function, because VCs hold a diversified portfolio of investments, the same options would be worth more to them than they are to me because they can diversify away downside risk much more effectively than I can. If these startups are making a true claim about the value of their options, there should be a trade here that makes all parties better off.

In a classic series of essays written a decade ago, seemingly aimed at convincing people to either found or join startups, Paul Graham stated "If you wanted to get rich, how would you do it? I think your best bet would be to start or join a startup. That's been a reliable way to get rich for hundreds of years" and "Risk and reward are always proportionate." This risk-reward assertion is used to back the claim that people can make more money, in expectation, by joining startups and taking risky equity packages than they can by taking jobs that pay cash or cash plus public equity. However, the premise — that risk and reward are always proportionate — isn’t true in the general case. It's basic finance 101 that only assets whose risk cannot be diversified away carry a risk premium (on average). Since VCs can and do diversify risk away, there’s no reason to believe that an individual employee who “invests” in startup options by working at a startup is getting a deal because of the risk involved. And by the way, when you look at historical returns, VC funds don’t appear to outperform other investment classes even though they get to buy a kind of startup equity that has less downside risk than the options you get as a normal employee.

So how come startups can’t or won’t take on more investment and pay their employees in cash? Let’s start by looking at some cynical reasons, followed by some less cynical reasons.

Cynical reasons

One possible answer, perhaps the simplest possible answer, is that options aren’t worth what startups claim they’re worth and startups prefer options because their lack of value is less obvious than it would be with cash. A simplistic argument that this might be the case is, if you look at the amount investors pay for a fraction of an early-stage or mid-stage startup and look at the extra cash the company would have been able to raise if they gave their employee option pool to investors, it usually isn’t enough to pay employees competitive compensation packages. Given that VCs don’t, on average, have outsized returns, this seems to imply that employee options aren’t worth as much as startups often claim. Compensation is much cheaper if you can convince people to take an arbirary number of lottery tickets in a lottery of unknown value instead of cash.

Some common ways that employee options are misrepresented are:

Strike price as value

A company that gives you 1M options with a strike price of $10 might claim that those are “worth” $10M. However, if the share price stays at $10 for the lifetime of the option, the options will end up being worth $0 because an option with a $10 strike price is an option to buy the stock at $10, which is not the same as a grant of actual shares worth $10 a piece.

Public valuation as value

Let’s say a company raised $300M by selling 30% of the company, giving the company an implied valuation of $1B. The most common misrepresentation I see is that the company will claim that because they’re giving an option for, say, 0.1% of the company, your option is worth $1B * 0.001 = $1M. A related, common, misrepresentation is that the company raised money last year and has increased in value since then, e.g., the company has since doubled in value, so your option is worth $2M. Even if you assume the strike price was $0 and and go with the last valuation at which the company raised money, the implied value of your option isn’t $1M because investors buy a different class of stock than you get as an employee.

There are a lot of differences between the preferred stock that VCs get and the common stock that employees get; let’s look at a couple of concrete scenarios.

Let’s say those investors that paid $300M for 30% of the company have a straight (1x) liquidation preference, and the company sells for $500M. The 1x liquidation preference means that the investors will get 1x of their investment back before lowly common stock holders get anything, so the investors will get $300M for their 30% of the company. The other 70% of equity will split $200M: your 0.1% common stock option with a $0 strike price is worth $285k (instead of the $500k you might expect it to be worth if you multiply $500M by 0.001).

The preferred stock VCs get usually has at least a 1x liquidation preference. Let’s say the investors had a 2x liquidation preference in the above scenario. They would get 2x their investment back before the common stockholders split the rest of the company. Since 2 * $300M is greater than $500M, the investors would get everything and the remaining equity holders would get $0.

Another difference between your common stock and preferred stock is that preferred stock sometimes comes with an anti-dilution clause, which you have no chance of getting as a normal engineering hire. Let’s look at an actual example of dilution at a real company. Mayhar got 0.4% of a company when it was valued at $5M. By the time the company was worth $1B, Mayhar’s share of the company was diluted by 8x, which made his share of the company worth less than $500k (minus the cost of exercising his options) instead of $4M (minus the cost of exercising his options).

This story has a few additional complications which illustrate other reasons options are often worth less than they seem. Mayhar couldn’t afford to exercise his options (by paying the strike price times the number of shares he had an option for) when he joined, which is common for people who take startup jobs out of college who don’t come from wealthy families. When he left four years later, he could afford to pay the cost of exercising the options, but due to a quirk of U.S. tax law, he either couldn’t afford the tax bill or didn’t want to pay that cost for what was still a lottery ticket — when you exercise your options, you’re effectively taxed on the difference between the current valuation and the strike price. Even if the company has a successful IPO for 10x as much in a few years, you’re still liable for the tax bill the year you exercise (and if the company stays private indefinitely or fails, you get nothing but a future tax deduction). Because, like most options, Mayhar’s option has a 90-day exercise window, he didn’t get anything from his options.

While that’s more than the average amount of dilution, there are much worse cases, for example, cases where investors and senior management basically get to keep their equity and everyone else gets diluted to the point where their equity is worthless.

Those are just a few of the many ways in which the differences between preferred and common stock can cause the value of options to be wildly different from a value naively calculated from a public valuation. I often see both companies and employees use public preferred stock valuations as a benchmark in order to precisely value common stock options, but this isn’t possible, even in principle, without access to a company’s cap table (which shows how much of the company different investors own) as well as access to the specific details of each investment. Even if you can get that (which you usually can’t), determining the appropriate numbers to plug into a model that will give you the expected value is non-trivial because it requires answering questions like “what’s the probability that, in an acquisition, upper management will collude with investors to keep everything and leave the employees with nothing?”

Black-Scholes valuation as value

Because of the issues listed above, people will sometimes try to use a model to estimate the value of options. Black-Scholes is commonly used because well known and has an easy to use closed form solution, it’s the most commonly used model. Unfortunately, most of the major assumptions for Black-Scholes are false for startup options, making the relationship between the output between Black-Scholes and the actual value of your options non-obvious.

Options are often free to the company

A large fraction of options get returned to the employee option pool when employees leave, either voluntarily or involuntarily. I haven’t been able to find comprehensive numbers on this, but anecdotally, I hear that more than 50% of options end up getting taken back from employees and returned to the general pool. Dan McKinley points out an (unvetted) analysis that shows that only 5% of employee grants are exercised. Even with a conservative estimate, a 50% discount on options granted sounds pretty good. A 20x discount sounds amazing, and would explain why companies like options so much.

Present value of a future sum of money

When someone says that a startup’s compensation package is worth as much as Facebook’s, they often mean that the total value paid out over N years is similar. But a fixed nominal amount of money is worth more the sooner you get it because you can (at a minimum) invest it in a low-risk asset, like Treasury bonds, and get some return on the money.

That’s an abstract argument you’ll hear in an econ 101 class, but in practice, if you live somewhere with a relatively high cost of living, like SF or NYC, there’s an even greater value to getting paid sooner rather than later because it lets you live in a relatively nice place (however you define nice) without having to cram into a space with more roommates than would be considered reasonable elsewhere in the U.S. Many startups from the last two generations seem to be putting off their IPOs; for folks in those companies with contracts that prevent them from selling options on a secondary market, that could easily mean that the majority of their potential wealth is locked up for the first decade of their working life. Even if the startup’s compensation package is worth more when adjusting for inflation and interest, it’s not clear if that’s a great choice for most people who aren’t already moderately well off.

Non-cynical reasons

We’ve looked at some cynical reasons companies might want to offer options instead of cash, namely that they can claim that their options are worth more than they’re actually worth. Now, let’s look at some non-cynical reasons companies might want to give out stock options.

From an employee standpoint, one non-cynical reason might have been stock option backdating, at least until that loophole was mostly closed. Up until late early 2000s, many companies backdated the date of options grants. Let’s look at this example, explained by Jessie M. Fried

Options covering 1.2 million shares were given to Reyes. The reported grant date was October 1, 2001, when the firm's stock was trading at around $13 per share, the lowest closing price for the year. A week later, the stock was trading at $20 per share, and a month later the stock closed at almost $26 per share.

Brocade disclosed this grant to investors in its 2002 proxy statement in a table titled "Option Grants in the Last Fiscal Year, prepared in the format specified by SEC rules. Among other things, the table describes the details of this and other grants to executives, including the number of shares covered by the option grants, the exercise price, and the options' expiration date. The information in this table is used by analysts, including those assembling Standard & Poor's well-known ExecuComp database, to calculate the Black Scholes value for each option grant on the date of grant. In calculating the value, the analysts assumed, based on the firm's representations about its procedure for setting exercise prices, that the options were granted at-the-money. The calculated value was then widely used by shareholders, researchers, and the media to estimate the CEO's total pay. The Black Scholes value calculated for Reyes' 1.2 million stock option grant, which analysts assumed was at-the-money, was $13.2 million.

However, the SEC has concluded that the option grant to Reyes was backdated, and the market price on the actual date of grant may have been around $26 per share. Let us assume that the stock was in fact trading at $26 per share when the options were actually granted. Thus, if Brocade had adhered to its policy of giving only at-the-money options, it should have given Reyes options with a strike price of $26 per share. Instead, it gave Reyes options with a strike price of $13 per share, so that the options were $13 in the money. And it reported the grant as if it had given Reyes at-the-money options when the stock price was $13 per share.

Had Brocade given Reyes at-the-money options at a strike price of $26 per share, the Black Scholes value of the option grant would have been approximately $26 million. But because the options were $13 million in the money, they were even more valuable. According to one estimate, they were worth $28 million. Thus, if analysts had been told that Reyes received options with a strike price of $13 when the stock was trading for $26, they would have reported their value as $28 million rather than $13.2 million. In short, backdating this particular option grant, in the scenario just described, would have enabled Brocade to give Reyes $2 million more in options (Black Scholes value) while reporting an amount that was $15 million less.

While stock options backdating isn’t (easily) possible anymore, there might be other loopholes or consequences of tax law that make options a better deal than cash. I could only think of one reason off the top of my head, so I spent a couple weeks asking folks (including multiple founders) for their non-cynical reasons why startups might prefer options to an equivalent amount of cash.

Tax benefit of ISOs

In the U.S., Incentive stock options (ISOs) have the property that, if held for one year after the exercise date and two years after the grant date, the owner of the option pays long-term capital gains tax instead of ordinary income tax on the difference between the exercise price and the strike price. In general, the capital gains has a lower tax rate than ordinary income.

This isn’t quite as good as it sounds because the difference between the exercise price and the strike price is subject to the Alternative Minimum Tax (AMT). I don’t find this personally relevant since I prefer to sell employer stock as quickly as possible in order to be as diversified as possible, but if you’re interested in figuring out how the AMT affects your tax bill when you exercise ISOs, see this explanation for more details. For people in California, California also has a relatively poor treatment of capital gains at the state level, which also makes this difference smaller than you might expect from looking at capital gains vs. ordinary income tax rates.

Tax benefit of QSBS

There’s a certain class of stock that is exempt from federal capital gains tax and state tax in many states (though not in CA). This is interesting, but it seems like people rarely take advantage of this when eligible, and many startups aren’t eligible.

Tax benefit of other options

The IRS says:

Most nonstatutory options don't have a readily determinable fair market value. For nonstatutory options without a readily determinable fair market value, there's no taxable event when the option is granted but you must include in income the fair market value of the stock received on exercise, less the amount paid, when you exercise the option. You have taxable income or deductible loss when you sell the stock you received by exercising the option. You generally treat this amount as a capital gain or loss.

Valuations are bogus

One quirk of stock options is that, to qualify as ISOs, the strike price must be at least the fair market value. That’s easy to determine for public companies, but the fair market value of a share in a private company is somewhat arbitrary. For ISOs, my reading of the requirement is that companies must make “an attempt, made in good faith” to determine the fair market value. For other types of options, there’s other regulation which which determines the definition of fair market value. Either way, startups usually go to an outside firm between 1 and N times a year to get an estimate of the fair market value for their common stock. This results in at least two possible gaps between a hypothetical “real” valuation and the fair market value for options purposes.

First, the valuation is updated relatively infrequently. A common pitch I’ve heard is that the company hasn’t had its valuation updated for ages, and the company is worth twice as much now, so you’re basically getting a 2x discount.

Second, the firms doing the valuations are poorly incentivized to produce “correct” valuations. The firms are paid by startups, which gain something when the legal valuation is as low as possible.

I don’t really believe that these things make options amazing, because I hear these exact things from startups and founders, which means that their offers take these into account and are priced accordingly. However, if there’s a large gap between the legal valuation and the “true” valuation and this allows companies to effectively give out higher compensation, the way stock option backdating did, I could see how this would tilt companies towards favoring options.

Control

Even if employees got the same class of stock that VCs get, founders would retain less control if they transferred the equity from employees to VCs because employee-owned equity is spread between a relatively large number of people.

Retention

This answer was commonly given to me as a non-cynical reason. The idea is that, if you offer employees options and have a clause that prevents them from selling options on a secondary market, many employees won’t be able to leave without walking away from the majority of their compensation. Personally, this strikes me as a cynical reason, but that’s not how everyone sees it. For example, Andreessen Horowitz managing partner Scott Kupor recently proposed a scheme under which employees would lose their options under all circumstances if they leave before a liquidity event, supposedly in order to help employees.

Whether or not you view employers being able to lock in employees for indeterminate lengths of time as good or bad, options lock-in appears to be a poor retention mechanism — companies that pay cash seem to have better retention. Just for example, Netflix pays salaries that are comparable to the total compensation in the senior band at places like Google and, anecdotally, they seem to have less attrition than trendy Bay Area startups. In fact, even though Netflix makes a lot of noise about showing people the door if they’re not a good fit, they don’t appear to have a higher involuntary attrition rate than trendy Bay Area startups — they just seem more honest about it, something which they can do because their recruiting pitch doesn’t involve you walking away with below-market compensation if you leave. If you think this comparison is unfair because Netflix hasn’t been a startup in recent memory, you can compare to finance startups, e.g. Headlands, which was founded in the same era as Uber, Airbnb, and Stripe. They (and some other finance startups) pay out hefty sums of cash and this does not appear to result in higher attrition than similarly aged startups which give out illiquid option grants.

In the cases where this results in the employee staying longer than they otherwise would, options lock-in is often a bad deal for all parties involved. The situation is obviously bad for employees and, on average, companies don’t want unhappy people who are just waiting for a vesting cliff or liquidity event.

Incentive alignment

Another commonly stated reason is that, if you give people options, they’ll work harder because they’ll do well when the company does well. This was the reason that was given most vehemently (“you shouldn’t trust someone who’s only interested in a paycheck”, etc.)

However, as far as I can tell, paying people in options almost totally decouples job performance and compensation. If you look at companies that have made a lot of people rich, like Microsoft, Google, Apple, and Facebook, almost none of the employees who became rich had an instrumental role in the company’s success. Google and Microsoft each made thousands of people rich, but the vast majority of those folks just happened to be in the right place at the right time and could have just as easily taken a different job where they didn't get rich. Conversely, the vast majority of startup option packages end up being worth little to nothing, but nearly none of the employees whose options end up being worthless were instrumental in causing their options to become worthless.

If options are a large fraction of compensation, choosing a company that’s going to be successful is much more important than working hard. For reference, Microsoft is estimated to have created roughly 10^3 millionaires by 1992 (adjusted for inflation, that's $1.75M). The stock then went up by more than 20x. Microsoft was legendary for making people who didn't particularly do much rich; all told, it's been estimated that they made 10^4 people rich by the late 90s. The vast majority of those people were no different from people in similar roles at Microsoft's competitors. They just happened to pick a winning lottery ticket. This is the opposite of what founders claim they get out of giving options. As above, companies that pay cash, like Netflix, don’t seem to have a problem with employee productivity.

By the way, a large fraction of the people who were made rich by working at Microsoft joined after their IPO, which was in 1986. The same is true of Google, and while Facebook is too young for us to have a good idea what the long-term post-IPO story is, the folks who joined a year or two after the IPO (5 years ago, in 2012) have done quite well for themselves. People who joined pre-IPO have done better, but as mentioned above, most people have diminishing returns to individual wealth. The same power-law-like distribution that makes VC work also means that it's entirely plausible that Microsoft alone made more post-IPO people rich from 1986-1999 than all pre-IPO tech companies combined during that period. Something similar is plausibly true for Google from 2004 until FB's IPO in 2012, even including the people who got rich from FB's IPO as people who were made rich by a pre-IPO company, and you can do a similar calculation for Apple.

VC firms vs. the market

There are several potential counter-arguments to the statement that VC returns (and therefore startup equity) don’t beat the market.

One argument is, when people say that, they typically mean that after VCs take their fees, returns to VC funds don’t beat the market. As an employee who gets startup options, you don’t (directly) pay VC fees, which means you can beat the market by keeping the VC fees for yourself.

Another argument is that, some investors (like YC) seem to consistently do pretty well. If you join a startup that’s funded by a savvy investors, you too can do pretty well. For this to make sense, you have to realize that the company is worth more than “expected” while the company doesn’t have the same realization because you need the company to give you an option package without properly accounting for its value. For you to have that expectation and get a good deal, this requires the founders to not only not be overconfident in the company’s probability of success, but actually requires that the founders are underconfident. While this isn’t impossible, the majority of startup offers I hear about have the opposite problem.

Investing

This section is an update written in 2020. This post was originally written when I didn't realize that it was possible for people who aren't extremely wealthy to invest in startups. But once I moved to SF, I found that it's actually very easy to invest in startups and that you don't have to be particularly wealthy (for a programmer) to do so — people will often take small checks (as small as $5k or sometimes even less) in seed rounds. If you can invest directly in a seed round, this is a strictly better deal than joining as an early employee.

As of this writing, it's quite common for companies to raise a seed round at a $10M valuation. This meeans you'd have to invest $100k to get 1%, or about as much equity as you'd expect to get as a very early employee. However, if you were to join the company, your equity would vest over four years, you'd get a worse class of equity, and you'd (typically) get much less information about the share structure of the company. As an investor, you only need to invest $25k to get 1 year's worth of early employee equity. Morever, you can invest in multiple companies, which gives you better risk adjusted return. At rates big companies are paying today (mid-band of perhaps $380k/yr for senior engineer, $600k/yr for staff engineer), working at a big company and spending $25k/yr investing in startups is strictly superior to working at a startup from the standpoint of financial return.

Conclusion

There are a number of factors that can make options more or less valuable than they seem. From an employee standpoint, the factors that make options more valuable than they seem can cause equity to be worth tens of percent more than a naive calculation. The factors that make options less valuable than they seem do so in ways that mostly aren’t easy to quantify.

Whether or not the factors that make options relatively more valuable dominate or the factors that make options relatively less valuable dominate is an empirical question. My intuition is that the factors that make options relatively less valuable are stronger, but that’s just a guess. A way to get an idea about this from public data would be to go through through successful startup S-1 filing. Since this post is already ~5k words, I’ll leave that for another post, but I’ll note that in my preliminary skim of a handful of 99%-ile exits (> $1B), the median employee seems to do worse than someone who’s on the standard Facebook/Google/Amazon career trajectory.

From a company standpoint, there are a couple factors that allow companies to retain more leverage/control by giving relatively more options to employees and relatively less equity to investors.

All of this sounds fine for founders and investors, but I don’t see what’s in it for employees. If you have additional reasons that I’m missing, I’d love to hear them.

_If you liked this post, you may also like this other post on the tradeoff between working at a big company and working at a startup.

Appendix: caveats

Many startups don’t claim that their offers are financially competitive. As time goes on, I hear less “If you wanted to get rich, how would you do it? I think your best bet would be to start or join a startup. That's been a reliable way to get rich for hundreds of years.” and more “we’re not financially competitive with Facebook, but ... ”. I’ve heard from multiple founders that joining as an early employee is an incredibly bad deal when you compare early-employee equity and workload vs. founder equity and workload.

Some startups are giving out offers that are actually competitive with large company offers. Something I’ve seen from startups that are trying to give out compelling offers is that, for “senior” folks, they’re willing to pay substantially higher salaries than public companies because it’s understood that options aren’t great for employees because of their timeline, risk profile, and expected value.

There’s a huge amount of variation in offers, much of which is effectively random. I know of cases where an individual got a more lucrative offer from a startup (that doesn’t tend to give particular strong offers) than from Google, and if you ask around you’ll hear about a lot of cases like that. It’s not always true that startup offers are lower than Google/Facebook/Amazon offers, even at startups that don’t pay competitively (on average).

Anything in this post that’s related to taxes is U.S. specific. For example, I’m told that in Canada, “you can defer the payment of taxes when exercising options whose strike price is way below fair market valuation until disposition, as long as the company is Canadian-controlled and operated in Canada”.

You might object that the same line of reasoning we looked at for options can be applied to RSUs, even RSUs for public companies. That’s true, although the largest downsides of startup options are mitigated or non-existent, cash still has significant advantages to employees over RSUs. Unfortunately, the only non-finance company I know of that uses this to their advantage in recruiting is Netflix; please let me know if you can think of other tech companies that use the same compensation model.

Some startups have a sliding scale that lets you choose different amounts of option/salary compensation. I haven't seen an offer that will let you put the slider to 100% cash and 0% options (or 100% options and 0% cash), but someone out there will probably be willing to give you an all-cash offer.

In the current environment, looking at public exits may bias the data towards less sucessful companies. The most sucessful startups from the last couple generations of startups that haven't exited by acquisition have so far chosen not to IPO. It's possible that, once all the data are in, the average returns to joining a startup will look quite different (although I doubt the median return will change much).

BTW, I don't have anything against taking a startup offer, even if it's low. When I graduated from college, I took the lowest offer I had, and my partner recently took the lowest offer she got (nearly a 2x difference over the highest offer). There are plenty of reasons you might want to take an offer that isn't the best possible financial offer. However, I think you should know what you're getting into and not take an offer that you think is financially great when it's merely mediocre or even bad.

Appendix: non-counterarguments

The most common objection I’ve heard to this is that most startups don’t have enough money to pay equivalent cash and couldn’t raise that much money by selling off what would “normally” be their employee option pool. Maybe so, but that’s not a counter-argument — it’s an argument that the most startups don’t have options that are valuable enough to be exchanged for the equivalent sum of money, i.e., that the options simply aren’t as valuable as claimed. This argument can be phrased in a variety of ways (e.g., paying salary instead of options increases burn rate, reduces runway, makes the startup default dead, etc.), but arguments of this form are fundamentally equivalent to admitting that startup options aren’t worth much because they wouldn't hold up if the options were worth enough that a typical compensation package was worth as much as a typical "senior" offer at Google or Facebook.

If you don't buy this, imagine a startup with a typical valuation that's at a stage where they're giving out 0.1% equity in options to new hires. Now imagine that some irrational bystander is willing to make a deal where they take 0.1% of the company for $1B. Is it worth it to take the money and pay people out of the $1B cash pool instead of paying people with 0.1% slices of the option pool? Your answer should be yes, unless you believe that the ratio between the value of cash on hand and equity is nearly infinite. Absolute statements like "options are preferred to cash because paying cash increases burn rate, making the startup default dead" at any valuation are equivalent to stating that the correct ratio is infinity. That's clearly nonsensical; there's some correct ratio, and we might disagree over what the correct ratio is, but for typical startups it should not be the case that the correct ratio is infinite. Since this was such a common objection, if you have this objection, my question to you is, why don't you argue that startups should pay even less cash and even more options? Is the argument that the current ratio is exactly optimal, and if so, why? Also, why does the ratio vary so much between different companies at the same stage which have raised roughly the same amount of money? Are all of those companies giving out optimal deals?

The second most common objection is that startup options are actually worth a lot, if you pick the right startup and use a proper model to value the options. Perhaps, but if that’s true, why couldn’t they have raised a bit more money by giving away more equity to VCs at its true value, and then pay cash?

Another common objection is something like "I know lots of people who've made $1m from startups". Me too, but I also know lots of people who've made much more than that working at public companies. This post is about the relative value of compensation packages, not the absolute value.

Acknowledgements

Thanks to Leah Hanson, Ben Kuhn, Tim Abbott, David Turner, Nick Bergson-Shilcock, Peter Fraenkel, Joe Ardent, Chris Ball, Anton Dubrau, Sean Talts, Danielle Sucher, Dan McKinley, Bert Muthalaly, Dan Puttick, Indradhanush Gupta, and Gaxun for comments and corrections.

Which is faster, keyboard or mouse? A large number of programmers believe that the keyboard is faster for all (programming-related) tasks. However, there are a few widely cited webpages on AskTog which claim that Apple studies show that using the mouse is faster than using the keyboard for everything and that people who think that using the keyboard is faster are just deluding themselves. This might sound extreme, but, just for example, one page says that the author has “never seen [the keyboard] outperform the mouse”.

But it can’t be the case that the mouse is faster for everything — almost no one is faster at clicking on an on-screen keyboard with a mouse than typing at a physical keyboard. Conversely, there are tasks for which mice are much better suited than keyboards (e.g., aiming in FPS games). For someone without an agenda, the question shouldn’t be, which is faster at all tasks, but which tasks are faster with a keyboard, which are faster with a mouse, and which are faster when both are used?

You might ask if any of this matters. It depends! One of the best programmrers I know is a hunt-and-peck typist, so it's clearly possible to be a great programmer without having particularly quick input speed. But I'm in the middle of an easy data munging task where I'm limited by the speed at which I can type in a large amount of boring code. If I were quicker, this task would be quicker, and there are tasks that I don't do that I might do. I can type at > 100 wpm, which isn't bad, but I can talk at > 400 wpm and I can think much faster than I can talk. I'm often rate limited even when talking; typing is much worse and the half-a-second here and one-second there I spent on navigation certainly doesn't help. When I first got started in tech, I had a mundane test/verification/QA role where my primary job was to triage test failures. Even before I started automating tasks, I could triage nearly twice as many bugs per day as other folks in the same role because I took being efficient at basic navigation tasks seriously. Nowadays, my jobs aren't 90% rote anymore, but my guess is that about a third of the time I spend in front of a computer is spent on mindless tasks that are rate-limited by my input and navigation speed. If I could get faster at those mundane tasks and have to spend less time on them and more time doing things that are fun, that would be great.

Anyway, to start, let’s look at the cited studies to see where the mouse is really faster. Most references on the web, when followed all the way back, point to the AskTog, a site by Bruce Tognazzini, who describes himself as a "recognized leader in human/computer interaction design".

The most cited AskTog page on the topic claims that they've spent $50M of R&D and done all kinds of studies; the page claims that, among other things, the $50M in R&D showed “Test subjects consistently report that keyboarding is faster than mousing” and “The stopwatch consistently proves mousing is faster than keyboarding. ”. The claim is that this both proves that the mouse is faster than the keyboard, and explains why programmers think the keyboard is faster than the mouse even though it’s slower. However, the result is unreproducible because “Tog” not only doesn’t cite the details of the experiments, Tog doesn’t even describe the experiments and just makes a blanket claim.

The second widely cited AskTog page is in response to a response to the previous page, and it simply repeats that the first page showed that keyboard shortcuts are slower. While there’s a lot of sarcasm, like “Perhaps we have all been misled these years. Perhaps the independent studies that show over and over again that Macintosh users are more productive, can learn quicker, buy more software packages, etc., etc., etc., are somehow all flawed. Perhaps....” no actual results are cited, as before. There is, however, a psuedo-scientific explanation of why the mouse is faster than the keyboard:

Command Keys Aren’t Faster. As you know from my August column, it takes just as long to decide upon a command key as it does to access the mouse. The difference is that the command-key decision is a high-level cognitive function of which there is no long-term memory generated. Therefore, subjectively, keys seem faster when in fact they usually take just as long to use.

Since mouse acquisition is a low-level cognitive function, the user need not abandon cognitive process on the primary task during the acquisition period. Therefore, the mouse acquirer achieves greater productivity.

One question this raises is, why should typing on the keyboard be any different from using command keys? There certainly are people who aren’t fluent at touch typing who have to think about which key they’re going to press when they type. Those people are very slow typists, perhaps even slower than someone who’s quick at using the mouse to type via an on screen keyboard. But there are also people who are fluent with the keyboard and can type without consciously thinking about which keys they’re going to press. The implicit claim here is that it’s not possible to be fluent with command keys in the same way it’s possible to be fluent with the keyboard for typing. It’s possible that’s true, but I find the claim to be highly implausible, both in principle, and from having observed people who certainly seem to be fluent with command keys, and the claim has no supporting evidence.

The third widely cited AskTog page cites a single experiment, where the author typed a paragraph and then had to replace every “e” with a “|”, either using cursor keys or the mouse. The author found that the average time for using cursor keys was 99.43 seconds and the average time for the mouse was 50.22 seconds. No information about the length of the paragraph or the number of “e”s was given. The third page was in response to a user who cited specific editing examples where they found that they were faster with a keyboard than with a mouse.

My experience with benchmarking is that the vast majority of microbenchmarks have wrong or misleading results because they’re difficult to set up properly, and even when set up properly, understanding how the microbenchmark results relate to real-world world results requires a deep understanding of the domain. As a result, I’m deeply skeptical of broad claims that come from microbenchmarks unless the author has a demonstrated, deep, understanding of benchmarking their particular domain, and even then I’ll ask why they believe their result generalizes. The opinion that microbenchmarks are very difficult to interpret properly is widely shared among people who understand benchmarking.

The e -> | replacement task described is not only a microbenchmark, it's a bizarrely artificial microbenchmark.

Based on the times given in the result, the task was either for very naive users, or disallowed any kind of search and replace functionality. This particular AskTog column is in response to a programmer who mentioned editing tasks, so the microbenchmark is meaningless unless that programmer is trapped in an experiment where they’re not allowed to use their editor’s basic functionality. Moreover, the replacement task itself is unrealistic — how often do people replace e with |?

I timed this task without the bizarre no-search-and-replace restriction removed and got the following results:

• Keyboard shortcut: 1.26s
• M-x, “replace-string” (instead of using mapped keyboard shortcut): 2.8s
• Navigate to search and replace with mouse: 5.39s

The first result was from using a keyboard shortcut. The second result is something I might do if I were in someone else’s emacs setup, which has different keyboard shortcuts mapped; emacs lets you run a command by hitting “M-x” and typing the entire name of the command. That’s much slower than using a keyboard shortcut directly, but still faster than using the mouse (at least for me, here) Does this mean that keyboards are great and mice are terrible? No, the result is nearly totally meaningless because I spend almost none of my time doing single-character search-and-replace, making the speed of single-character search-and-replace irrelevant.

Also, since I’m used to using the keyboard, the mouse speed here is probably unusually slow. That’s doubly true here because my normal editor setup (emacs -nw) doesn’t allow for mouse usage, so I ended up using an unfamiliar editor, TextEdit, for the mouse test. I did each task once in order to avoid “practicing” the exact task, which could unrealistically make the keyboard-shortcut version nearly instantaneous because it’s easy to hit a practiced sequence of keys very quickly. However, this meant that I was using an unfamiliar mouse in an unfamiliar set of menus for the mouse. Furthermore, like many people who’ve played video games in the distant past, I’m used to having “mouse acceleration” turned off, but the Mac has this on by default and I didn’t go through the rigmarole necessary to disable mouse acceleration. Additionally, recording program I used (quicktime) made the entire machine laggy, which probably affects mousing speed more than keyboard speed, and the menu setup for the program I happened to use forced me to navigate through two levels of menus.

That being said, despite not being used to the mouse, if I want to find a microbenchmark where I’m faster with the mouse than with the keyboard, that’s easy: let me try selecting a block of text that’s on the screen but not near my cursor:

• Keyboard: 1.8s
• Mouse: 0.7s

I tend to do selection of blocks in emacs by searching for something at the start of the block, setting a mark, and then searching for something at the end of the mark. I typically type three characters to make sure that I get a unique chunk of text (and I’ll type more if it’s text where I don’t think three characters will cut it). This makes the selection task somewhat slower than the replacement task because the replacement task used single characters and this task used multiple characters.

The mouse is so much better suited for selecting a block of text that even with an unfamiliar mouse setup where I end up having to make a correction instead of being able to do the selection in one motion, the mouse is over twice as fast. But, if I wanted select something that was off screen and the selection was so large that it wouldn’t fit on one screen, the keyboard time wouldn’t change and the mouse time would get much slower, making the keyboard faster.

In addition to doing the measurements, I also (informally) polled people to ask if they thought the keyboard or the mouse would be faster for specific tasks. Both search-and-replace and select-text are tasks where the result was obvious to most people. But not all tasks are obvious; scrolling was one where people didn’t have strong opinions one way or another. Let’s look at scrolling, which is a task both the keyboard and the mouse are well suited for. To have something concrete, let’s look at scrolling down 4 pages:

• Keyboard: 0.49s
• Mouse: 0.57s

While there’s some difference, and I suspect that if I repeated the experiment enough times I could get a statistically significant result, but the difference is small enough that the difference isn’t of practical significance.

Contra Tog’s result, which was that everyone believes the keyboard was faster even though the mouse is faster, I find that people are pretty good at estimating what’s which device is faster for which tasks and also at estimate when both devices will give a similar result. One possible reason is that I’m polling programmers, and in particular, programmers at RC, who are probably a different population than whoever Tog might’ve studied in his studies. He was in a group that was looking at how to design the UI for a general purpose computer in the 80s, where it would have been actually been unreasonable to focus on studying people, many of whom grew up using computers, and then chose a career where you use computers all day. The equivalent population would’ve had to start using computers in the 60s or even earlier, but even if they had, input devices were quite different (the ball mouse wasn’t invented until 1972, and it certainly wasn’t in wide use the moment it was invented). There’s nothing wrong with studying populations who aren’t relatively expert at using computer input devices, but there is something wrong with generalizing those results to people who are relatively expert.

Unlike claims by either keyboard or mouse advocates, when I do experiments myself, the results are mixed. Some tasks are substantially faster if I use the keyboard and some are substantially faster if I use the mouse. Moreover, most of the results are easily predictable (when the results are similar, the prediction is that it would be hard to predict). If we look at the most widely cited, authoritative, results on the web, we find that they make very strong claims that the mouse is much faster than the keyboard but back up the claim with nothing but a single, bogus, experiment. It’s possible that some of the vaunted $50M in R&D went into valid experiments, but those experiments, if they exist, aren’t cited.

I spent some time reviewing the literature on the subject, but couldn’t find anything conclusive. Rather than do a point-by-point summary of each study (like I did here for here for another controversial topic), I’ll mention the high-level issues that make the studies irrelevant to me. All studies I could find had at least one of the issues listed below; if you have a link to a study that isn’t irrelevant for one of the following reasons, I’d love to hear about it!

1. Age of study: it’s unclear how a study on interacting with computers from the mid-80s transfers to how people interact with computers today. Even ignoring differences in editing programs, there are large differences in the interface. Mice are more precise and a decent modern optical mouse can be moved as fast as a human can move it without the tracking becoming erratic, something that isn’t true of any mouse I’ve tried from the 80s and was only true of high quality mice from the 90s when the balls were recently cleaned and the mouse was on a decent quality mousepad. Keyboards haven’t improved as much, but even so, I can type substantially faster a modern, low-travel, keyboard than on any keyboard I’ve tried from the 80s.
2. Narrow microbenchmarking: not all of these are as irrelevant as the e -> | without search and replace task, but even in the case of tasks that aren’t obviously irrelevant, it’s not clear what the impact of the result is on actual work I might do.
3. Not keyboard vs. mouse: a tiny fraction of published studies are on keyboard vs. mouse interaction. When a study is on device interaction, it’s often about some new kind of device or a new interaction model.
4. Vague description: a lot of studies will say something like they found a 7.8% improvement, with results being significant with p < 0.005, without providing enough information to tell if the results are actually significant or merely statistically significant (recall that the practically insignificant scrolling result was a 0.08s difference, which could also be reported as a 16.3% improvement).
5. Unskilled users: in one, typical, paper, they note that it can take users as long as two seconds to move the mouse from one side of the screen to a scrollbar on the other side of the screen. While there’s something to be said for doing studies on unskilled users in order to figure out what sorts of interfaces are easiest for users who have the hardest time, a study on users who take 2 seconds to get their mouse onto the scrollbar doesn’t appear to be relevant to my user experience. When I timed this for myself, it took 0.21s to get to the scrollbar from the other side of the screen and scroll a short distance, despite using an unfamiliar mouse with different sensitivity than I’m used to and running a recording program which made mousing more difficult than usual.
6. Seemingly unreasonable results: some studies claim to show large improvements in overall productivity when switching from type of device to another (e.g., a 20% total productivity gain from switching types of mice).

Conclusion

It’s entirely possible that the mysterious studies Tog’s org spent $50M on prove that the mouse is faster than the keyboard for all tasks other than raw text input, but there doesn’t appear to be enough information to tell what the actual studies were. There are many public studies on user input, but I couldn’t find any that are relevant to whether or not I should use the mouse more or less at the margin.

When I look at various tasks myself, the results are mixed, and they’re mixed in the way that most programmers I polled predicted. This result is so boring that it would barely be worth mentioning if not for the large groups of people who believe that either the keyboard is always faster than the mouse or vice versa.

Please let me know if there are relevant studies on this topic that I should read! I’m not familiar with the relevant fields, so it’s possible that I’m searching with the wrong keywords and reading the wrong papers.

Appendix: note to self

I didn't realize that scrolling was so fast relative to searching (not explicitly mentioned in the blog post, but 1/2 of the text selection task). I tend to use search to scroll to things that are offscreen, but it appears that I should consider scrolling instead when I don't want to drop my cursor in a specific position.

Thanks to Leah Hanson, Quentin Pradet, Alex Wilson, and Gaxun for comments/corrections on this post and to Annie Cherkaev, Chris Ball, Stefan Lesser, and David Isaac Lee for related discussion.

There’s a great MSR demo from 2012 that shows the effect of latency on the experience of using a tablet. If you don’t want to watch the three minute video, they basically created a device which could simulate arbitrary latencies down to a fraction of a millisecond. At 100ms (1/10th of a second), which is typical of consumer tablets, the experience is terrible. At 10ms (1/100th of a second), the latency is noticeable, but the experience is ok, and at < 1ms the experience is great, as good as pen and paper. If you want to see a mini version of this for yourself, you can try a random Android tablet with a stylus vs. the current generation iPad Pro with the Apple stylus. The Apple device has well above 10ms end-to-end latency, but the difference is still quite dramatic -- it’s enough that I’ll actually use the new iPad Pro to take notes or draw diagrams, whereas I find Android tablets unbearable as a pen-and-paper replacement.

You can also see something similar if you try VR headsets with different latencies. 20ms feels fine, 50ms feels laggy, and 150ms feels unbearable.

Curiously, I rarely hear complaints about keyboard and mouse input being slow. One reason might be that keyboard and mouse input are quick and that inputs are reflected nearly instantaneously, but I don’t think that’s true. People often tell me that’s true, but I think it’s just the opposite. The idea that computers respond quickly to input, so quickly that humans can’t notice the latency, is the most common performance-related fallacy I hear from professional programmers.

When people measure actual end-to-end latency for games on normal computer setups, they usually find latencies in the 100ms range.

If we look at Robert Menzel’s breakdown of the the end-to-end pipeline for a game, it’s not hard to see why we expect to see 100+ ms of latency:

• ~2 msec (mouse)
• 8 msec (average time we wait for the input to be processed by the game)
• 16.6 (game simulation)
• 16.6 (rendering code)
• 16.6 (GPU is rendering the previous frame, current frame is cached)
• 16.6 (GPU rendering)
• 8 (average for missing the vsync)
• 16.6 (frame caching inside of the display)
• 16.6 (redrawing the frame)
• 5 (pixel switching)

Note that this assumes a gaming mouse and a pretty decent LCD; it’s common to see substantially slower latency for the mouse and for pixel switching.

It’s possible to tune things to get into the 40ms range, but the vast majority of users don’t do that kind of tuning, and even if they do, that’s still quite far from the 10ms to 20ms range, where tablets and VR start to feel really “right”.

Keypress-to-display measurements are mostly done in games because gamers care more about latency than most people, but I don’t think that most applications are all that different from games in terms of latency. While games often do much more work per frame than “typical” applications, they’re also much better optimized than “typical” applications. Menzel budgets 33ms to the game, half for game logic and half for rendering. How much time do non-game applications take? Pavel Fatin measured this for text editors and found latencies ranging from a few milliseconds to hundreds of milliseconds and he did this with an app he wrote that we can use to measure the latency of other applications that uses java.awt.Robot to generate keypresses and do screen captures.

Personally, I’d like to see the latency of different terminals and shells for a couple of reasons. First, I spend most of my time in a terminal and usually do editing in a terminal, so the latency I see is at least the latency of the terminal. Second, the most common terminal benchmark I see cited (by at least two orders of magnitude) is the rate at which a terminal can display output, often measured by running cat on a large file. This is pretty much as useless a benchmark as I can think of. I can’t recall the last task I did which was limited by the speed at which I can cat a file to stdout on my terminal (well, unless I’m using eshell in emacs), nor can I think of any task for which that sub-measurement is useful. The closest thing that I care about is the speed at which I can ^C a command when I’ve accidentally output too much to stdout, but as we’ll see when we look at actual measurements, a terminal’s ability to absorb a lot of input to stdout is only weakly related to its responsiveness to ^C. The speed at which I can scroll up or down an entire page sounds related, but in actual measurements the two are not highly correlated (e.g., emacs-eshell is quick at scrolling but extremely slow at sinking stdout). Another thing I care about is latency, but knowing that a particular terminal has high stdout throughput tells me little to nothing about its latency.

Let’s look at some different terminals to see if any terminals add enough latency that we’d expect the difference to be noticeable. If we measure the latency from keypress to internal screen capture on my laptop, we see the following latencies for different terminals

These graphs show the distribution of latencies for each terminal. The y-axis has the latency in milliseconds. The x-axis is the percentile (e.g., 50 means represents 50%-ile keypress i.e., the median keypress). Measurements are with macOS unless otherwise stated. The graph on the left is when the machine is idle, and the graph on the right is under load. If we just look at median latencies, some setups don’t look too bad -- terminal.app and emacs-eshell are at roughly 5ms unloaded, small enough that many people wouldn’t notice. But most terminals (st, alacritty, hyper, and iterm2) are in the range where you might expect people to notice the additional latency even when the machine is idle. If we look at the tail when the machine is idle, say the 99.9%-ile latency, every terminal gets into the range where the additional latency ought to be perceptible, according to studies on user interaction. For reference, the internally generated keypress to GPU memory trip for some terminals is slower than the time it takes to send a packet from Boston to Seattle and back, about 70ms.

All measurements were done with input only happening on one terminal at a time, with full battery and running off of A/C power. The loaded measurements were done while compiling Rust (as before, with full battery and running off of A/C power, and in order to make the measurements reproducible, each measurement started 15s after a clean build of Rust after downloading all dependencies, with enough time between runs to avoid thermal throttling interference across runs).

If we look at median loaded latencies, other than emacs-term, most terminals don’t do much worse than at idle. But as we look at tail measurements, like 90%-ile or 99.9%-ile measurements, every terminal gets much slower. Switching between macOS and Linux makes some difference, but the difference is different for different terminals.

These measurements aren't anywhere near the worst case (if we run off of battery when the battery is low, and wait 10 minutes into the compile in order to exacerbate thermal throttling, it’s easy to see latencies that are multiple hundreds of ms) but even so, every terminal has tail latency that should be observable. Also, recall that this is only a fraction of the total end-to-end latency.

Why don’t people complain about keyboard-to-display latency the way they complain stylus-to-display latency or VR latency? My theory is that, for both VR and tablets, people have a lot of experience with a much lower latency application. For tablets, the “application” is pen-and-paper, and for VR, the “application” is turning your head without a VR headset on. But input-to-display latency is so bad for every application that most people just expect terrible latency.

An alternate theory might be that keyboard and mouse input are fundamentally different from tablet input in a way that makes latency less noticeable. Even without any data, I’d find that implausible because, when I access a remote terminal in a way that adds tens of milliseconds of extra latency, I find typing to be noticeably laggy. And it turns out that when extra latency is A/B tested, people can and do notice latency in the range we’re discussing here.

Just so we can compare the most commonly used benchmark (throughput of stdout) to latency, let’s measure how quickly different terminals can sink input on stdout:

Alacritty was tested with tmux because alacritty doesn’t support scrolling back up, and the docs indicate that you should use tmux if you want to be able to scroll up. Just to have another reference, terminal.app was also tested with tmux. For most terminals, tmux doesn’t appear to reduce stdout speed, but alacritty and terminal.app are fast enough that they’re actually limited by the speed of tmux.

Emacs-eshell is technically not a terminal, but I also tested eshell because it can be used as a terminal alternative for some use cases. Emacs, with both eshell and term, is actually slow enough that I care about the speed at which it can sink stdout. When I’ve used eshell or term in the past, I find that I sometimes have to wait for a few thousand lines of text to scroll by if I run a command with verbose logging to stdout or stderr. Since that happens very rarely, it’s not really a big deal to me unless it’s so slow that I end up waiting half a second or a second when it happens, and no other terminal is slow enough for that to matter.

Conversely, I type individual characters often enough that I’ll notice tail latency. Say I type at 120wpm and that results in 600 characters per minute, or 10 characters per second of input. Then I’d expect to see the 99.9% tail (1 in 1000) every 100 seconds!

Anyway, the cat “benchmark” that I care about more is whether or not I can ^C a process when I’ve accidentally run a command that outputs millions of lines to the screen instead of thousands of lines. For that benchmark, every terminal is fine except for hyper and emacs-eshell, both of which hung for at least ten minutes (I killed each process after ten minutes, rather than waiting for the terminal to catch up).

Memory usage at startup is also included in the table for reference because that's the other measurement I see people benchmark terminals with. While I think that it's a bit absurd that terminals can use 40MB at startup, even the three year old hand-me-down laptop I'm using has 16GB of RAM, so squeezing that 40MB down to 2MB doesn't have any appreciable affect on user experience. Heck, even the $300 chromebook we recently got has 16GB of RAM.

Conclusion

Most terminals have enough latency that the user experience could be improved if the terminals concentrated more on latency and less on other features or other aspects of performance. However, when I search for terminal benchmarks, I find that terminal authors, if they benchmark anything, benchmark the speed of sinking stdout or memory usage at startup. This is unfortunate because most “low performance” terminals can already sink stdout many orders of magnitude faster than humans can keep up with, so further optimizing stdout throughput has a relatively small impact on actual user experience for most users. Likewise for reducing memory usage when an idle terminal uses 0.01% of the memory on my old and now quite low-end laptop.

If you work on a terminal, perhaps consider relatively more latency and interactivity (e.g., responsiveness to ^C) optimization and relatively less throughput and idle memory usage optimization.

Update: In response to this post, the author of alacritty explains where alacritty's latency comes from and describes how alacritty could reduce its latency

Appendix: negative results

Tmux and latency: I tried tmux and various terminals and found that the the differences were within the range of measurement noise.

Shells and latency: I tried a number of shells and found that, even in the quickest terminal, the difference between shells was within the range of measurement noise. Powershell was somewhat problematic to test with the setup I was using because it doesn’t handle colors correctly (the first character typed shows up with the color specified by the terminal, but other characters are yellow regardless of setting, which appears to be an open issue), which confused the image recognition setup I used. Powershell also doesn’t consistently put the cursor where it should be -- it jumps around randomly within a line, which also confused the image recognition setup I used. However, despite its other problems, powershell had comparable performance to other shells.

Shells and stdout throughput: As above, the speed difference between different shells was within the range of measurement noise.

Single-line vs. multiline text and throughput: Although some text editors bog down with extremely long lines, throughput was similar when I shoved a large file into a terminal whether the file was all one line or was line broken every 80 characters.

Head of line blocking / coordinated omission: I ran these tests with input at a rate of 10.3 characters per second. But it turns out this doesn't matter much and input rates that humans are capapable of and the latencies are quite similar to doing input once every 10.3 seconds. It's possible to overwhelm a terminal, and hyper is the first to start falling over at high input rates, but the speed necessary to make the tail latency worse is beyond the rate at which any human I know of can type.

Appendix: experimental setup

All tests were done on a dual core 2.6GHz 13” Mid-2014 Macbook pro. The machine has 16GB of RAM and a 2560x1600 screen. The OS X version was 10.12.5. Some tests were done in Linux (Lubuntu 16.04) to get a comparison between macOS and Linux. 10k keypresses were for each latency measurements.

Latency measurements were done with the . key and throughput was done with default base32 output, which is all plain ASCII text. George King notes that different kinds of text can change output speed:

I’ve noticed that Terminal.app slows dramatically when outputting non-latin unicode ranges. I’m aware of three things that might cause this: having to load different font pages, and having to parse code points outside of the BMP, and wide characters.

The first probably boils down to a very complicated mix of lazy loading of font glyphs, font fallback calculations, and caching of the glyph pages or however that works.

The second is a bit speculative, but I would bet that Terminal.app uses Cocoa’s UTF16-based NSString, which almost certainly hits a slow path when code points are above the BMP due to surrogate pairs.

Terminals were fullscreened before running tests. This affects test results, and resizing the terminal windows can and does significantly change performance (e.g., it’s possible to get hyper to be slower than iterm2 by changing the window size while holding everything else constant). st on macOS was running as an X client under XQuartz. To see if XQuartz is inherently slow, I tried runes, another "native" Linux terminal that uses XQuartz; runes had much better tail latency than st and iterm2.

The “idle” latency tests were done on a freshly rebooted machine. All terminals were running, but input was only fed to one terminal at a time.

The “loaded” latency tests were done with rust compiling in the background, 15s after the compilation started.

Terminal bandwidth tests were done by creating a large, pseudo-random, text file with

timeout 64 sh -c 'cat /dev/urandom | base32 > junk.txt'

and then running

timeout 8 sh -c 'cat junk.txt | tee junk.term_name'

Terminator and urxvt weren’t tested because they weren’t completely trivial to install on mac and I didn’t want to futz around to make them work. Terminator was easy to build from source, but it hung on startup and didn’t get to a shell prompt. Urxvt installed through brew, but one of its dependencies (also installed through brew) was the wrong version, which prevented it from starting.

Thanks to Kamal Marhubi, Leah Hanson, Wesley Aptekar-Cassels, David Albert, Vaibhav Sagar, Indradhanush Gupta, Rudi Chen, Laura Lindzey, Ahmad Jarara, George King, Tim Dierks, Nikith Naide, Veit Heller, and Nick Bergson-Shilcock for comments/corrections/discussion.

I recently had a problem where part of the solution was to do a series of pointer accesses that would walk around a chunk of memory in pseudo-random order. Sattolo's algorithm provides a solution to this because it produces a permutation of a list with exactly one cycle, which guarantees that we will reach every element of the list even though we're traversing it in random order.

However, the explanations of why the algorithm worked that I could find online either used some kind of mathematical machinery (Stirling numbers, assuming familiarity with cycle notation, etc.), or used logic that was hard for me to follow. I find that this is common for explanations of concepts that could, but don't have to, use a lot of mathematical machinery. I don't think there's anything wrong with using existing mathematical methods per se -- it's a nice mental shortcut if you're familiar with the concepts. If you're taking a combinatorics class, it makes sense to cover Stirling numbers and then rattle off a series of results whose proofs are trivial if you're familiar with Stirling numbers, but for people who are only interested in a single result, I think it's unfortunate that it's hard to find a relatively simple explanation that doesn't require any background. When I was looking for a simple explanation, I also found a lot of people who were using Sattolo's algorithm in places where it wasn't appropriate and also people who didn't know that Sattolo's algorithm is what they were looking for, so here's an attempt at an explanation of why the algorithm works that doesn't assume an undergraduate combinatorics background.

Before we look at Sattolo's algorithm, let's look at Fisher-Yates, which is an in-place algorithm that produces a random permutation of an array/vector, where every possible permutation occurs with uniform probability.

We'll look at the code for Fisher-Yates and then how to prove that the algorithm produces the intended result.

def shuffle(a):
    n = len(a)
    for i in range(n - 1):  # i from 0 to n-2, inclusive.
        j = random.randrange(i, n)  # j from i to n-1, inclusive.
        a[i], a[j] = a[j], a[i]  # swap a[i] and a[j].

shuffle takes an array and produces a permutation of the array, i.e., it shuffles the array. We can think of this loop as placing each element of the array, a, in turn, from a[0] to a[n-2]. On some iteration, i, we choose one of n-i elements to swap with and swap element i with some random element. The last element in the array, a[n-1], is skipped because it would always be swapped with itself. One way to see that this produces every possible permutation with uniform probability is to write down the probability that each element will end up in any particular location¹. Another way to do it is to observe two facts about this algorithm:

1. Every output that Fisher-Yates produces is produced with uniform probability
2. Fisher-Yates produces as many outputs as there are permutations (and each output is a permutation)

(1) For each random choice we make in the algorithm, if we make a different choice, we get a different output. For example, if we look at the resultant a[0], the only way to place the element that was originally in a[k] (for some k) in the resultant a[0] is to swap a[0] with a[k] in iteration 0. If we choose a different element to swap with, we'll end up with a different resultant a[0]. Once we place a[0] and look at the resultant a[1], the same thing is true of a[1] and so on for each a[i]. Additionally, each choice reduces the range by the same amount -- there's a kind of symmetry, in that although we place a[0] first, we could have placed any other element first; every choice has the same effect. This is vaguely analogous to the reason that you can pick an integer uniformly at random by picking digits uniformly at random, one at a time.

(2) How many different outputs does Fisher-Yates produce? On the first iteration, we fix one of n possible choices for a[0], then given that choice, we fix one of n-1 choices for a[1], then one of n-2 for a[2], and so on, so there are n * (n-1) * (n-2) * ... 2 * 1 = n! possible different outputs.

This is exactly the same number of possible permutations of n elements, by pretty much the same reasoning. If we want to count the number of possible permutations of n elements, we first pick one of n possible elements for the first position, n-1 for the second position, and so on resulting in n! possible permutations.

Since Fisher-Yates only produces unique permutations and there are exactly as many outputs as there are permutations, Fisher-Yates produces every possible permutation. Since Fisher-Yates produces each output with uniform probability, it produces all possible permutations with uniform probability.

Now, let's look at Sattolo's algorithm, which is almost identical to Fisher-Yates and also produces a shuffled version of the input, but produces something quite different:

def sattolo(a):
    n = len(a)
    for i in range(n - 1):
        j = random.randrange(i+1, n)  # i+1 instead of i
        a[i], a[j] = a[j], a[i]

Instead of picking an element at random to swap with, like we did in Fisher-Yates, we pick an element at random that is not the element being placed, i.e., we do not allow an element to be swapped with itself. One side effect of this is that no element ends up where it originally started.

Before we talk about why this produces the intended result, let's make sure we're on the same page regarding terminology. One way to look at an array is to view it as a description of a graph where the index indicates the node and the value indicates where the edge points to. For example, if we have the list 0 2 3 1, this can be thought of as a directed graph from its indices to its value, which is a graph with the following edges:

0 -> 0
1 -> 2
2 -> 3
3 -> 1

Node 0 points to itself (because the value at index 0 is 0), node 1 points to node 2 (because the value at index 1 is 2), and so on. If we traverse this graph, we see that there are two cycles. 0 -> 0 -> 0 ... and 1 -> 2 -> 3 -> 1....

Let's say we swap the element in position 0 with some other element. It could be any element, but let's say that we swap it with the element in position 2. Then we'll have the list 3 2 0 1, which can be thought of as the following graph:

0 -> 3
1 -> 2
2 -> 0
3 -> 1

If we traverse this graph, we see the cycle 0 -> 3 -> 1 -> 2 -> 0.... This is an example of a permutation with exactly one cycle.

If we swap two elements that belong to different cycles, we'll merge the two cycles into a single cycle. One way to see this is when we swap two elements in the list, we're essentially picking up the arrow-heads pointing to each element and swapping where they point (rather than the arrow-tails, which stay put). Tracing the result of this is like tracing a figure-8. Just for example, say if we swap 0 with an arbitrary element of the other cycle, let's say element 2, we'll end up with 3 2 0 1, whose only cycle is 0 -> 3 -> 1 -> 2 -> 0.... Note that this operation is reversible -- if we do the same swap again, we end up with two cycles again. In general, if we swap two elements from the same cycle, we break the cycle into two separate cycles.

If we feed a list consisting of 0 1 2 ... n-1 to Sattolo's algorithm we'll get a permutation with exactly one cycle. Furthermore, we have the same probability of generating any permutation that has exactly one cycle. Let's look at why Sattolo's generates exactly one cycle. Afterwards, we'll figure out why it produces all possible cycles with uniform probability.

For Sattolo's algorithm, let's say we start with the list 0 1 2 3 ... n-1, i.e., a list with n cycles of length 1. On each iteration, we do one swap. If we swap elements from two separate cycles, we'll merge the two cycles, reducing the number of cycles by 1. We'll then do n-1 iterations, reducing the number of cycles from n to n - (n-1) = 1.

Now let's see why it's safe to assume we always swap elements from different cycles. In each iteration of the algorithm, we swap some element with index > i with the element at index i and then increment i. Since i gets incremented, the element that gets placed into index i can never be swapped again, i.e., each swap puts one of the two elements that was swapped into its final position, i.e., for each swap, we take two elements that were potentially swappable and render one of them unswappable.

When we start, we have n cycles of length 1, each with 1 element that's swappable. When we swap the initial element with some random element, we'll take one of the swappable elements and render it unswappable, creating a cycle of length 2 with 1 swappable element and leaving us with n-2 other cycles, each with 1 swappable element.

The key invariant that's maintained is that each cycle has exactly 1 swappable element. The invariant holds in the beginning when we have n cycles of length 1. And as long as this is true, every time we merge two cycles of any length, we'll take the swappable element from one cycle and swap it with the swappable element from the other cycle, rendering one of the two elements unswappable and creating a longer cycle that still only has one swappable element, maintaining the invariant.

Since we cannot swap two elements from the same cycle, we merge two cycles with every swap, reducing the number of cycles by 1 with each iteration until we've run n-1 iterations and have exactly one cycle remaining.

To see that we generate each cycle with equal probability, note that there's only one way to produce each output, i.e., changing any particular random choice results in a different output. In the first iteration, we randomly choose one of n-1 placements, then n-2, then n-3, and so on, so for any particular cycle, we produce it with probability (n-1) * (n-2) * (n-3) ... * 2 * 1 = (n-1)!. If we can show that there are (n-1)! permutations with exactly one cycle, then we'll know that we generate every permutation with exactly one cycle with uniform probability.

Let's say we have an arbitrary list of length n that has exactly one cycle and we add a single element, there are n ways to extend that to become a cycle of length n+1 because there are n places we could add in the new element and keep the cycle, which means that the number of cycles of length n+1, cycles(n+1), is n * cycles(n).

For example, say we have a cycle that produces the path 0 -> 1 -> 2 -> 0 ... and we want to add a new element, 3. We can substitute -> 3 -> for any -> and get a cycle of length 4 instead of length 3.

In the base case, there's one cycle of length 2, the permutation 1 0 (the other permutation of length two, 0 1, has two cycles of length one instead of having a cycle of length 2), so we know that cycles(2) = 1. If we apply the recurrence above, we get that cycles(n) = (n-1)!, which is exactly the number of different permutations that Sattolo's algorithm generates, which means that we generate all possible permutations with one cycle. Since we know that we generate each cycle with uniform probability, we now know that we generate all possible one-cycle permutations with uniform probability.

An alternate way to see that there are (n-1)! permutations with exactly one cycle, is that we rotate each cycle around so that 0 is at the start and write it down as 0 -> i -> j -> k -> .... The number of these is the same as the number of permutations of elements to the right of the 0 ->, which is (n-1)!.

Conclusion

We've looked at two algorithms that are identical, except for a two character change. These algorithms produce quite different results -- one algorithm produces a random permutation and the other produces a random permutation with exactly one cycle. I think these algorithms are neat because they're so simple, just a double for loop with a swap.

In practice, you probably don't "need" to know how these algorithms work because the standard library for most modern languages will have some way of producing a random shuffle. And if you have a function that will give you a shuffle, you can produce a permutation with exactly one cycle if you don't mind a non-in-place algorithm that takes an extra pass. I'll leave that as an exercise for the reader, but if you want a hint, one way to do it parallels the "alternate" way to see that there are (n-1)! permutations with exactly one cycle.

Although I said that you probably don't need to know this stuff, you do actually need to know it if you're going to implement a custom shuffling algorithm! That may sound obvious, but there's a long history of people implementing incorrect shuffling algorithms. This was common in games and on online gambling sites in the 90s and even the early 2000s and you still see the occasional mis-implemented shuffle, e.g., when Microsoft implemented a bogus shuffle and failed to properly randomize a browser choice poll. At the time, the top Google hit for javascript random array sort was the incorrect algorithm that Microsoft ended up using. That site has been fixed, but you can still find incorrect tutorials floating around online.

Appendix: generating a random derangement

A permutation where no element ends up in its original position is called a derangement. When I searched for uses of Sattolo's algorithm, I found many people using Sattolo's algorithm to generate random derangements. While Sattolo's algorithm generates derangements, it only generates derangements with exactly one cycle, and there are derangements with more than one cycle (e.g., 3 2 1 0), so it can't possibly generate random derangements with uniform probability.

One way to generate random derangements is to generate random shuffles using Fisher-Yates and then retry until we get a derangement:

def derangement(n):
    assert n != 1, "can't have a derangement of length 1"
    a = list(range(n))
    while not is_derangement(a):
        shuffle(a)
    return a

This algorithm is simple, and is overwhelmingly likely to eventually return a derangement (for n != 1), but it's not immediately obvious how long we should expect this to run before it returns a result. Maybe we'll get a derangement on the first try and run shuffle once, or maybe it will take 100 tries and we'll have to do 100 shuffles before getting a derangement.

To figure this out, we'll want to know the probability that a random permutation (shuffle) is a derangement. To get that, we'll want to know, given a list of of length n, how many permutations there are and how many derangements there are.

Since we're deep in the appendix, I'll assume that you know the number of permutations of a n elements is n! what binomial coefficients are, and are comfortable with Taylor series.

To count the number of derangements, we can start with the number of permutations, n!, and subtract off permutations where an element remains in its starting position, (n choose 1) * (n - 1)!. That isn't quite right because this double subtracts permutations where two elements remain in the starting position, so we'll have to add back (n choose 2) * (n - 2)!. That isn't quite right because we've overcorrected elements with three permutations, so we'll have to add those back, and so on and so forth, resulting in ∑ (−1)ᵏ (n choose k)(n−k)!. If we expand this out and divide by n! and cancel things out, we get ∑ (−1)ᵏ (1 / k!). If we look at the limit as the number of elements goes to infinity, this looks just like the Taylor series for e^x where x = -1, i.e., 1/e, i.e., in the limit, we expect that the fraction of permutations that are derangements is 1/e, i.e., we expect to have to do e times as many swaps to generate a derangement as we do to generate a random permutation. Like many alternating series, this series converges quickly. It gets within 7 significant figures of e when k = 10!

One silly thing about our algorithm is that, if we place the first element in the first location, we already know that we don't have a derangement, but we continue placing elements until we've created an entire permutation. If we reject illegal placements, we can do even better than a factor of e overhead. It's also possible to come up with a non-rejection based algorithm, but I really enjoy the naive rejection based algorithm because I find it delightful when basic randomized algorithms that consist of "keep trying again" work well.

Appendix: wikipedia's explanation of Sattolo's algorithm

I wrote this explanation because I found the explanation in Wikipedia relatively hard to follow, but if you find the explanation above difficult to understand, maybe you'll prefer wikipedia's version:

The fact that Sattolo's algorithm always produces a cycle of length n can be shown by induction. Assume by induction that after the initial iteration of the loop, the remaining iterations permute the first n - 1 elements according to a cycle of length n - 1 (those remaining iterations are just Sattolo's algorithm applied to those first n - 1 elements). This means that tracing the initial element to its new position p, then the element originally at position p to its new position, and so forth, one only gets back to the initial position after having visited all other positions. Suppose the initial iteration swapped the final element with the one at (non-final) position k, and that the subsequent permutation of first n - 1 elements then moved it to position l; we compare the permutation π of all n elements with that remaining permutation σ of the first n - 1 elements. Tracing successive positions as just mentioned, there is no difference between σ and π until arriving at position k. But then, under π the element originally at position k is moved to the final position rather than to position l, and the element originally at the final position is moved to position l. From there on, the sequence of positions for π again follows the sequence for σ, and all positions will have been visited before getting back to the initial position, as required.

As for the equal probability of the permutations, it suffices to observe that the modified algorithm involves (n-1)! distinct possible sequences of random numbers produced, each of which clearly produces a different permutation, and each of which occurs--assuming the random number source is unbiased--with equal probability. The (n-1)! different permutations so produced precisely exhaust the set of cycles of length n: each such cycle has a unique cycle notation with the value n in the final position, which allows for (n-1)! permutations of the remaining values to fill the other positions of the cycle notation

Thanks to Mathieu Guay-Paquet, Leah Hanson, Rudi Chen, Kamal Marhubi, Michael Robert Arntzenius, Heath Borders, Shreevatsa R, @chozu@fedi.absturztau.be, and David Turner for comments/corrections/discussion.

This is a pseudo-transcript for a talk on branch prediction given at Two Sigma on 8/22/2017 to kick off "localhost", a talk series organized by RC.

How many of you use branches in your code? Could you please raise your hand if you use if statements or pattern matching?

Most of the audience raises their hands

I won’t ask you to raise your hands for this next part, but my guess is that if I asked, how many of you feel like you have a good understanding of what your CPU does when it executes a branch and what the performance implications are, and how many of you feel like you could understand a modern paper on branch prediction, fewer people would raise their hands.

The purpose of this talk is to explain how and why CPUs do “branch prediction” and then explain enough about classic branch prediction algorithms that you could read a modern paper on branch prediction and basically know what’s going on.

Before we talk about branch prediction, let’s talk about why CPUs do branch prediction. To do that, we’ll need to know a bit about how CPUs work.

For the purposes of this talk, you can think of your computer as a CPU plus some memory. The instructions live in memory and the CPU executes a sequence of instructions from memory, where instructions are things like “add two numbers”, “move a chunk of data from memory to the processor”. Normally, after executing one instruction, the CPU will execute the instruction that’s at the next sequential address. However, there are instructions called “branches” that let you change the address next instruction comes from.

Here’s an abstract diagram of a CPU executing some instructions. The x-axis is time and the y-axis distinguishes different instructions.

Here, we execute instruction A, followed by instruction B, followed by instruction C, followed by instruction D.

One way you might design a CPU is to have the CPU do all of the work for one instruction, then move on to the next instruction, do all of the work for the next instruction, and so on. There’s nothing wrong with this; a lot of older CPUs did this, and some modern very low-cost CPUs still do this. But if you want to make a faster CPU, you might make a CPU that works like an assembly line. That is, you break the CPU up into two parts, so that half the CPU can do the “front half” of the work for an instruction while half the CPU works on the “back half” of the work for an instruction, like an assembly line. This is typically called a pipelined CPU.

If you do this, the execution might look something like the above. After the first half of instruction A is complete, the CPU can work on the second half of instruction A while the first half of instruction B runs. And when the second half of A finishes, the CPU can start on both the second half of B and the first half of C. In this diagram, you can see that the pipelined CPU can execute twice as many instructions per unit time as the unpipelined CPU above.

There’s no reason that a CPU can only be broken up into two parts. We could break the CPU into three parts, and get a 3x speedup, or four parts and get a 4x speedup. This isn’t strictly true, and we generally get less than a 3x speedup for a three-stage pipeline or 4x speedup for a 4-stage pipeline because there’s overhead in breaking the CPU up into more parts and having a deeper pipeline.

One source of overhead is how branches are handled. One of the first things the CPU has to do for an instruction is to get the instruction; to do that, it has to know where the instruction is. For example, consider the following code:

if (x == 0) {
  // Do stuff
} else {
  // Do other stuff (things)
}
  // Whatever happens later

This might turn into assembly that looks something like

branch_if_not_equal x, 0, else_label
// Do stuff
goto end_label
else_label:
// Do things
end_label:
// whatever happens later

In this example, we compare x to 0. if_not_equal, then we branch to else_label and execute the code in the else block. If that comparison fails (i.e., if x is 0), we fall through, execute the code in the if block, and then jump to end_label in order to avoid executing the code in else block.

The particular sequence of instructions that’s problematic for pipelining is

branch_if_not_equal x, 0, else_label
???

The CPU doesn’t know if this is going to be

branch_if_not_equal x, 0, else_label
// Do stuff

or

branch_if_not_equal x, 0, else_label
// Do things

until the branch has finished (or nearly finished) executing. Since one of the first things the CPU needs to do for an instruction is to get the instruction from memory, and we don’t know which instruction ??? is going to be, we can’t even start on ??? until the previous instruction is nearly finished.

Earlier, when we said that we’d get a 3x speedup for a 3-stage pipeline or a 20x speedup for a 20-stage pipeline, that assumed that you could start a new instruction every cycle, but in this case the two instructions are nearly serialized.

One way around this problem is to use branch prediction. When a branch shows up, the CPU will guess if the branch was taken or not taken.

In this case, the CPU predicts that the branch won’t be taken and starts executing the first half of stuff while it’s executing the second half of the branch. If the prediction is correct, the CPU will execute the second half of stuff and can start another instruction while it’s executing the second half of stuff, like we saw in the first pipeline diagram.

If the prediction is wrong, when the branch finishes executing, the CPU will throw away the result from stuff.1 and start executing the correct instructions instead of the wrong instructions. Since we would’ve stalled the processor and not executed any instructions if we didn’t have branch prediction, we’re no worse off than we would’ve been had we not made a prediction (at least at the level of detail we’re looking at).

What’s the performance impact of doing this? To make an estimate, we’ll need a performance model and a workload. For the purposes of this talk, our cartoon model of a CPU will be a pipelined CPU where non-branches take an average of one instruction per clock, unpredicted or mispredicted branches take 20 cycles, and correctly predicted branches take one cycle.

If we look at the most commonly used benchmark of “workstation” integer workloads, SPECint, the composition is maybe 20% branches, and 80% other operations. Without branch prediction, we then expect the “average” instruction to take branch_pct * 1 + non_branch_pct * 20 = 0.2 * 20 + 0.8 * 1 = 4 + 0.8 = 4.8 cycles. With perfect, 100% accurate, branch prediction, we’d expect the average instruction to take 0.8 * 1 + 0.2 * 1 = 1 cycle, a 4.8x speedup! Another way to look at it is that if we have a pipeline with a 20-cycle branch misprediction penalty, we have nearly a 5x overhead from our ideal pipelining speedup just from branches alone.

Let’s see what we can do about this. We’ll start with the most naive things someone might do and work our way up to something better.

Predict taken

Instead of predicting randomly, we could look at all branches in the execution of all programs. If we do this, we’ll see that taken and not not-taken branches aren’t exactly balanced -- there are substantially more taken branches than not-taken branches. One reason for this is that loop branches are often taken.

If we predict that every branch is taken, we might get 70% accuracy, which means we’ll pay the the misprediction cost for 30% of branches, making the cost of of an average instruction (0.8 + 0.7 * 0.2) * 1 + 0.3 * 0.2 * 20 = 0.94 + 1.2. = 2.14. If we compare always predicting taken to no prediction and perfect prediction, always predicting taken gets a large fraction of the benefit of perfect prediction despite being a very simple algorithm.

Backwards taken forwards not taken (BTFNT)

Predicting branches as taken works well for loops, but not so great for all branches. If we look at whether or not branches are taken based on whether or not the branch is forward (skips over code) or backwards (goes back to previous code), we can see that backwards branches are taken more often than forward branches, so we could try a predictor which predicts that backward branches are taken and forward branches aren’t taken (BTFNT). If we implement this scheme in hardware, compiler writers will conspire with us to arrange code such that branches the compiler thinks will be taken will be backwards branches and branches the compiler thinks won’t be taken will be forward branches.

If we do this, we might get something like 80% prediction accuracy, making our cost function (0.8 + 0.8 * 0.2) * 1 + 0.2 * 0.2 * 20 = 0.96 + 0.8 = 1.76 cycles per instruction.

Used by

• PPC 601(1993): also uses compiler generated branch hints
• PPC 603

One-bit

So far, we’ve look at schemes that don’t store any state, i.e., schemes where the prediction ignores the program’s execution history. These are called static branch prediction schemes in the literature. These schemes have the advantage of being simple but they have the disadvantage of being bad at predicting branches whose behavior change over time. If you want an example of a branch whose behavior changes over time, you might imagine some code like

if (flag) {
  // things
  }

Over the course of the program, we might have one phase of the program where the flag is set and the branch is taken and another phase of the program where flag isn’t set and the branch isn’t taken. There’s no way for a static scheme to make good predictions for a branch like that, so let’s consider dynamic branch prediction schemes, where the prediction can change based on the program history.

One of the simplest things we might do is to make a prediction based on the last result of the branch, i.e., we predict taken if the branch was taken last time and we predict not taken if the branch wasn’t taken last time.

Since having one bit for every possible branch is too many bits to feasibly store, we’ll keep a table of some number of branches we’ve seen and their last results. For this talk, let’s store not taken as 0 and taken as 1.

In this case, just to make things fit on a diagram, we have a 64-entry table, which mean that we can index into the table with 6 bits, so we index into the table with the low 6 bits of the branch address. After we execute a branch, we update the entry in the prediction table (highlighted below) and the next time the branch is executed again, we index into the same entry and use the updated value for the prediction.

It’s possible that we’ll observe aliasing and two branches in two different locations will map to the same location. This isn’t ideal, but there’s a tradeoff between table speed & cost vs. size that effectively limits the size of the table.

If we use a one-bit scheme, we might get 85% accuracy, a cost of (0.8 + 0.85 * 0.2) * 1 + 0.15 * 0.2 * 20 = 0.97 + 0.6 = 1.57 cycles per instruction.

Used by

• DEC EV4 (1992)
• MIPS R8000 (1994)

Two-bit

A one-bit scheme works fine for patterns like TTTTTTTT… or NNNNNNN… but will have a misprediction for a stream of branches that’s mostly taken but has one branch that’s not taken, ...TTTNTTT... This can be fixed by adding second bit for each address and implementing a saturating counter. Let’s arbitrarily say that we count down when a branch is not taken and count up when it’s taken. If we look at the binary values, we’ll then end up with:

00: predict Not
01: predict Not
10: predict Taken
11: predict Taken

The “saturating” part of saturating counter means that if we count down from 00, instead of underflowing, we stay at 00, and similar for counting up from 11 staying at 11. This scheme is identical to the one-bit scheme, except that each entry in the prediction table is two bits instead of one bit.

Compared to a one-bit scheme, a two-bit scheme can have half as many entries at the same size/cost (if we only consider the cost of storage and ignore the cost of the logic for the saturating counter), but even so, for most reasonable table sizes a two-bit scheme provides better accuracy.

Despite being simple, this works quite well, and we might expect to see something like 90% accuracy for a two bit predictor, which gives us a cost of 1.38 cycles per instruction.

One natural thing to do would be to generalize the scheme to an n-bit saturating counter, but it turns out that adding more bits has a relatively small effect on accuracy. We haven’t really discussed the cost of the branch predictor, but going from 2 bits to 3 bits per branch increases the table size by 1.5x for little gain, which makes it not worth the cost in most cases. The simplest and most common things that we won’t predict well with a two-bit scheme are patterns like NTNTNTNTNT... or NNTNNTNNT…, but going to n-bits won’t let us predict those patterns well either!

Used by

• LLNL S-1 (1977)
• CDC Cyber? (early 80s)
• Burroughs B4900 (1982): state stored in instruction stream; hardware would over-write instruction to update branch state
• Intel Pentium (1993)
• PPC 604 (1994)
• DEC EV45 (1993)
• DEC EV5 (1995)
• PA 8000 (1996): actually a 3-bit shift register with majority vote

Two-level adaptive, global (1991)

If we think about code like

for (int i = 0; i < 3; ++i) {
  // code here.
  }

That code will produce a pattern of branches like TTTNTTTNTTTN....

If we know the last three executions of the branch, we should be able to predict the next execution of the branch:

TTT:N
TTN:T
TNT:T
NTT:T

The previous schemes we’ve considered use the branch address to index into a table that tells us if the branch is, according to recent history, more likely to be taken or not taken. That tells us which direction the branch is biased towards, but it can’t tell us that we’re in the middle of a repetitive pattern. To fix that, we’ll store the history of the most recent branches as well as a table of predictions.

In this example, we concatenate 4 bits of branch history together with 2 bits of branch address to index into the prediction table. As before, the prediction comes from a 2-bit saturating counter. We don’t want to only use the branch history to index into our prediction table since, if we did that, any two branches with the same history would alias to the same table entry. In a real predictor, we’d probably have a larger table and use more bits of branch address, but in order to fit the table on a slide, we have an index that’s only 6 bits long.

Below, we’ll see what gets updated when we execute a branch.

The bolded parts are the parts that were updated. In this diagram, we shift new bits of branch history in from right to left, updating the branch history. Because the branch history is updated, the low bits of the index into the prediction table are updated, so the next time we take the same branch again, we’ll use a different entry in the table to make the prediction, unlike in previous schemes where the index is fixed by the branch address. The old entry’s value is updated so that the next time we take the same branch again with the same branch history, we’ll have the updated prediction.

Since the history in this scheme is global, this will correctly predict patterns like NTNTNTNT… in inner loops, but may not always correct make predictions for higher-level branches because the history is global and will be contaminated with information from other branches. However, the tradeoff here is that keeping a global history is cheaper than keeping a table of local histories. Additionally, using a global history lets us correctly predict correlated branches. For example, we might have something like:

if x > 0:
  x -= 1
if y > 0:
  y -= 1
if x * y > 0:
  foo()

If either the first branch or the next branch isn’t taken, then the third branch definitely will not be taken.

With this scheme, we might get 93% accuracy, giving us 1.27 cycles per instruction.

Used by

• Pentium MMX (1996): 4-bit global branch history

Two-level adaptive, local [1992]

As mentioned above, an issue with the global history scheme is that the branch history for local branches that could be predicted cleanly gets contaminated by other branches.

One way to get good local predictions is to keep separate branch histories for separate branches.

Instead of keeping a single global history, we keep a table of local histories, index by the branch address. This scheme is identical to the global scheme we just looked at, except that we keep multiple branch histories. One way to think about this is that having global history is a special case of local history, where the number of histories we keep track of is 1.

With this scheme, we might get something like 94% accuracy, which gives us a cost of 1.23 cycles per instruction.

Used by

• Pentium Pro (1996): 4 bit local branch history, low bits of PC used for index. Note that is under some dispute and Agner Fog claims that the PPro and follow-on processors use 4-bit global history
• Pentium II (1997): same as PPro
• Pentium III (1999): same as PPro

gshare

One tradeoff a global two-level scheme has to make is that, for a prediction table of a fixed size, bits must be dedicated to either the branch history or the branch address. We’d like to give more bits to the branch history because that allows correlations across greater “distance” as well as tracking more complicated patterns and we’d like to give more bits to the branch address to avoid interference between unrelated branches.

We can try to get the best of both worlds by hashing both the branch history and the branch address instead of concatenating them. One of the simplest reasonable things one might do, and the first proposed mechanism was to xor them together. This two-level adaptive scheme, where we xorthe bits together is called gshare.

With this scheme, we might see something like 94% accuracy. That’s the accuracy we got from the local scheme we just looked at, but gshare avoids having to keep a large table of local histories; getting the same accuracy while having to track less state is a significant improvement.

Used by

• MIPS R12000 (1998): 2K entries, 11 bits of PC, 8 bits of history
• UltraSPARC-3 (2001): 16K entries, 14 bits of PC, 12 bits of history

agree (1997)

One reason for branch mispredictions is interference between different branches that alias to the same location. There are many ways to reduce interference between branches that alias to the same predictor table entry. In fact, the reason this talk only runs into schemes invented in the 90s is because a wide variety of interference-reducing schemes were proposed and there are too many to cover in half an hour.

We’ll look at one scheme which might give you an idea of what an interference-reducing scheme could look like, the “agree” predictor. When two branch-history pairs collide, the predictions either match or they don’t. If they match, we’ll call that neutral interference and if they don’t, we’ll call that negative interference. The idea is that most branches tend to be strongly biased (that is, if we use two-bit entries in the predictor table, we expect that, without interference, most entries will be 00 or 11 most of the time, not 01 or 10). For each branch in the program, we’ll store one bit, which we call the “bias”. The table of predictions will, instead of storing the absolute branch predictions, store whether or not the prediction matches or does not match the bias.

If we look at how this works, the predictor is identical to a gshare predictor, except that we make the changes mentioned above -- the prediction is agree/disagree instead of taken/not-taken and we have a bias bit that’s indexed by the branch address, which gives us something to agree or disagree with. In the original paper, they propose using the first thing you see as the bias and other people have proposed using profile-guided optimization (basically running the program and feeding the data back to the compiler) to determine the bias.

Note that, when we execute a branch and then later come back around to the same branch, we’ll use the same bias bit because the bias is indexed by the branch address, but we’ll use a different predictor table entry because that’s indexed by both the branch address and the branch history.

If it seems weird that this would do anything, let’s look at a concrete example. Say we have two branches, branch A which is taken with 90% probability and branch B which is taken with 10% probability. If those two branches alias and we assume the probabilities that each branch is taken are independent, the probability that they disagree and negatively interfere is P(A taken) * P(B not taken) + P(A not taken) + P(B taken) = (0.9 * 0.9) + (0.1 * 0.1) = 0.82.

If we use the agree scheme, we can re-do the calculation above, but the probability that the two branches disagree and negatively interfere is P(A agree) * P(B disagree) + P(A disagree) * P(B agree) = P(A taken) * P(B taken) + P(A not taken) * P(B taken) = (0.9 * 0.1) + (0.1 * 0.9) = 0.18. Another way to look at it is, to have destructive interference, one of the branches must disagree with its bias. By definition, if we’ve correctly determined the bias, this cannot be likely to happen.

With this scheme, we might get something like 95% accuracy, giving us 1.19 cycles per instruction.

Used by

• PA-RISC 8700 (2001)

Hybrid (1993)

As we’ve seen, local predictors can predict some kinds of branches well (e.g., inner loops) and global predictors can predict some kinds of branches well (e.g., some correlated branches). One way to try to get the best of both worlds is to have both predictors, then have a meta predictor that predicts if the local or the global predictor should be used. A simple way to do this is to have the meta-predictor use the same scheme as the two-bit predictor above, except that instead of predicting taken or not taken it predicts local predictor or global predictor

Just as there are many possible interference-reducing schemes, of which the agree predictor, above is one, there are many possible hybrid schemes. We could use any two predictors, not just a local and global predictor, and we could even use more than two predictors.

If we use a local and global predictor, we might get something like 96% accuracy, giving us 1.15 cycles per instruction.

Used by

• DEC EV6 (1998): combination of local (1k entries, 10 history bits, 3 bit counter) & global (4k entries, 12 history bits, 2 bit counter) predictors
• IBM POWER4 (2001): local (16k entries) & gshare (16k entries, 11 history bits, xor with branch address, 16k selector table)
• IBM POWER5 (2004): combination of bimodal (not covered) and two-level adaptive
• IBM POWER7 (2010)

Not covered

There are a lot of things we didn’t cover in this talk! As you might expect, the set of material that we didn’t cover is much larger than what we did cover. I’ll briefly describe a few things we didn’t cover, with references, so you can look them up if you’re interested in learning more.

One major thing we didn’t talk about is how to predict the branch target. Note that this needs to be done even for some unconditional branches (that is, branches that don’t need directional prediction because they’re always taken), since (some) unconditional branches have unknown branch targets.

Branch target prediction is expensive enough that some early CPUs had a branch prediction policy of “always predict not taken” because a branch target isn’t necessary when you predict the branch won’t be taken! Always predicting not taken has poor accuracy, but it’s still better than making no prediction at all.

Among the interference reducing predictors we didn’t discuss are bi-mode, gskew, and YAGS. Very briefly, bi-mode is somewhat like agree in that it tries to seperate out branches based on direction, but the mechanism used in bi-mode is that we keep multiple predictor tables and a third predictor based on the branch address is used to predict which predictor table gets use for the particular combination of branch and branch history. Bi-mode appears to be more successful than agree in that it's seen wider use. With gskew, we keep at least three predictor tables and use a different hash to index into each table. The idea is that, even if two branches alias, those two branches will only alias in one of the tables, so we can use a vote and the result from the other two tables will override the potentially bad result from the aliasing table. I don't know how to describe YAGS very briefly :-).

Because we didn't take about speed (as in latency), a prediction strategy we didn't talk about is to have a small/fast predictor that can be overridden by a slower and more accurate predictor when the slower predictor computes its result.

Some modern CPUs have completely different branch predictors; AMD Zen (2017) and AMD Bulldozer (2011) chips appear to use perceptron based branch predictors. Perceptrons are single-layer neural nets.

It’s been argued that Intel Haswell (2013) uses a variant of a TAGE predictor. TAGE stands for TAgged GEometric history length predictor. If we look at the predictors we’ve covered and look at actual executions of programs to see which branches we’re not predicting correctly, one major class of branches are branches that need a lot of history -- a significant number of branches need tens or hundreds of bits of history and some even need more than a thousand bits of branch history. If we have a single predictor or even a hybrid predictor that combines a few different predictors, it’s counterproductive to keep a thousand bits of history because that will make predictions worse for the branches which need a relatively small amount of history (especially relative to the cost), which is most branches. One of the ideas in the TAGE predictor is that, by keeping a geometric series of history lengths, each branch can use the appropriate history. That explains the GE. The TA part is that branches are tagged, which is a mechanism we don’t discuss that the predictor uses to track which branches should use which set of history.

Modern CPUs often have specialized predictors, e.g., a loop predictor can accurately predict loop branches in cases where a generalized branch predictor couldn’t reasonably store enough history to make perfect predictions for every iteration of the loop.

We didn’t talk at all about the tradeoff between using up more space and getting better predictions. Not only does changing the size of the table change the performance of a predictor, it also changes which predictors are better relative to each other.

We also didn’t talk at all about how different workloads affect different branch predictors. Predictor performance varies not only based on table size but also based on which particular program is run.

We’ve also talked about branch misprediction cost as if it’s a fixed thing, but it is not, and for that matter, the cost of non-branch instructions also varies widely between different workloads.

I tried to avoid introducing non-self-explanatory terminology when possible, so if you read the literature, terminology will be somewhat different.

Conclusion

We’ve looked at a variety of classic branch predictors and very briefly discussed a couple of newer predictors. Some of the classic predictors we discussed are still used in CPUs today, and if this were an hour long talk instead of a half-hour long talk, we could have discussed state-of-the-art predictors. I think that a lot of people have an idea that CPUs are mysterious and hard to understand, but I think that CPUs are actually easier to understand than software. I might be biased because I used to work on CPUs, but I think that this is not a result of my bias but something fundamental.

If you think about the complexity of software, the main limiting factor on complexity is your imagination. If you can imagine something in enough detail that you can write it down, you can make it. Of course there are cases where that’s not the limiting factor and there’s something more practical (e.g., the performance of large scale applications), but I think that most of us spend most of our time writing software where the limiting factor is the ability to create and manage complexity.

Hardware is quite different from this in that there are forces that push back against complexity. Every chunk of hardware you implement costs money, so you want to implement as little hardware as possible. Additionally, performance matters for most hardware (whether that’s absolute performance or performance per dollar or per watt or per other cost), and adding complexity makes hardware slower, which limits performance. Today, you can buy an off-the-shelf CPU for $300 which can be overclocked to 5 GHz. At 5 GHz, one unit of work is one-fifth of one nanosecond. For reference, light travels roughly one foot in one nanosecond. Another limiting factor is that people get pretty upset when CPUs don’t work perfectly all of the time. Although CPUs do have bugs, the rate of bugs is much lower than in almost all software, i.e., the standard to which they’re verified/tested is much higher. Adding complexity makes things harder to test and verify. Because CPUs are held to a higher correctness standard than most software, adding complexity creates a much higher test/verification burden on CPUs, which makes adding a similar amount of complexity much more expensive in hardware than in software, even ignoring the other factors we discussed.

A side effect of these factors that push back against chip complexity is that, for any particular “high-level” general purpose CPU feature, it is generally conceptually simple enough that it can be described in a half-hour or hour-long talk. CPUs are simpler than many programmers think! BTW, I say “high-level” to rule out things like how transistors and circuit design, which can require a fair amount of low-level (physics or solid-state) background to understand.

CPU internals series

• New CPU features since the 80s
• The cost of branches and integer overflow checking in real code
• CPU bugs
• A brief history of branch prediction
• Why CPU development is hard
• Verilog sucks, part 1
• Verilog sucks, part 2

Thanks to Leah Hanson, Hari Angepat, and Nick Bergson-Shilcock for reviewing practice versions of the talk and to Fred Clausen Jr for finding a typo in this post. Apologies for the somewhat slapdash state of this post -- I wrote it quickly so that people who attended the talk could refer to the “transcript ” soon afterwards and look up references, but this means that there are probably more than the usual number of errors and that the organization isn’t as nice as it would be for a normal blog post. In particular, things that were explained using a series of animations in the talk are not explained in the same level of detail and on skimming this, I notice that there’s less explanation of what sorts of branches each predictor doesn’t handle well, and hence less motivation for each predictor. I may try to go back and add more motivation, but I’m unlikely to restructure the post completely and generate a new set of graphics that better convey concepts when there are a couple of still graphics next to text. Thanks to Julien Vivenot, Ralph Corderoy, Vaibhav Sagar, Mindy Preston, Stefan Kanthak, and Uri Shaked for catching typos in this hastily written post.

If you look at “gaming" keyboards, a lot of them sell for $100 or more on the promise that they’re fast. Ad copy that you’ll see includes:

• a custom designed keycap that has been made shorter to reduce the time it takes for your actions to register
• 8x FASTER - Polling Rate of 1000Hz: Response time 0.1 milliseconds
• Wield the ultimate performance advantage over your opponents with light operation 45g key switches and an actuation 40% faster than standard Cherry MX Red switches
• World's Fastest Ultra Polling 1000Hz
• World's Fastest Gaming Keyboard, 1000Hz Polling Rate, 0.001 Second Response Time

Despite all of these claims, I can only find one person who’s publicly benchmarked keyboard latency and they only tested two keyboards. In general, my belief is that if someone makes performance claims without benchmarks, the claims probably aren’t true, just like how code that isn’t tested (or otherwise verified) should be assumed broken.

The situation with gaming keyboards reminds me a lot of talking to car salesmen:

Salesman: this car is super safe! It has 12 airbags! Me: that’s nice, but how does it fare in crash tests? Salesman: 12 airbags!

Sure, gaming keyboards have 1000Hz polling, but so what?

Two obvious questions are:

1. Does keyboard latency matter?
2. Are gaming keyboards actually quicker than other keyboards?

Does keyboard latency matter?

A year ago, if you’d asked me if I was going to build a custom setup to measure keyboard latency, I would have said that’s silly, and yet here I am, measuring keyboard latency with a logic analyzer.

It all started because I had this feeling that some old computers feel much more responsive than modern machines. For example, an iMac G4 running macOS 9 or an Apple 2 both feel quicker than my 4.2 GHz Kaby Lake system. I never trust feelings like this because there’s decades of research showing that users often have feelings that are the literal opposite of reality, so got a high-speed camera and started measuring actual keypress-to-screen-update latency as well as mouse-move-to-screen-update latency. It turns out the machines that feel quick are actually quick, much quicker than my modern computer -- computers from the 70s and 80s commonly have keypress-to-screen-update latencies in the 30ms to 50ms range out of the box, whereas modern computers are often in the 100ms to 200ms range when you press a key in a terminal. It’s possible to get down to the 50ms range in well optimized games with a fancy gaming setup, and there’s one extraordinary consumer device that can easily get below 50ms, but the default experience is much slower. Modern computers have much better throughput, but their latency isn’t so great.

Anyway, at the time I did these measurements, my 4.2 GHz kaby lake had the fastest single-threaded performance of any machine you could buy but had worse latency than a quick machine from the 70s (roughly 6x worse than an Apple 2), which seems a bit curious. To figure out where the latency comes from, I started measuring keyboard latency because that’s the first part of the pipeline. My plan was to look at the end-to-end pipeline and start at the beginning, ruling out keyboard latency as a real source of latency. But it turns out keyboard latency is significant! I was surprised to find that the median keyboard I tested has more latency than the entire end-to-end pipeline of the Apple 2. If this doesn’t immedately strike you as absurd, consider that an Apple 2 has 3500 transistors running at 1MHz and an Atmel employee estimates that the core used in a number of high-end keyboards today has 80k transistors running at 16MHz. That's 20x the transistors running at 16x the clock speed -- keyboards are often more powerful than entire computers from the 70s and 80s! And yet, the median keyboard today adds as much latency as the entire end-to-end pipeline as a fast machine from the 70s.

Let’s look at the measured keypress-to-USB latency on some keyboards: