RSS Feeds

Work-life balance at Bioware
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2008-05-31 00:00:00 | Created: 2026-07-23 05:18:40

This is an archive of some posts in a forum thread titled "Beware of Bioware" in a now defunct forum, with comments from that forum as well as blog comments from a now defunct blog that archived that made the first attempt to archive this content. The original posts were deleted shortly after being posted, replaced with "big scary company vs. li'l ol' me."

Although the comments below seem to be about Bioware's main studio in Edmonton, I knew someone at Bioware Austin during the time period under discussion, which is how I learned about the term "sympathy crunch", where you're required to be at the office because other teams are "crunching". I'd never heard of this concept before, so I looked it up and found the following thread around 2008 or so.

Searching for "sympathy crunch" today, in 2024, doesn't return many hits. One of the few hits is a 2011 blog post by a former director at BioWare titled "Loving the Crunch", which has this to say about sympathy crunches:

If you find yourself working sympathy crunch, in that even though you have no bugs of your own to take care of, don’t be pissed off about it. Play test the game you made! And enjoy it for what it is, something you contributed to. And if that’s not enough to make you happy then be satisfied that every bug you send to one of your co-workers will make them more miserable. (Though do try and be constructive.)

Another one of the few hits is a 2013 recruiting marketing blog post on Bioware blog, where a developer notes that "We are clearly moving away from the concept of 'sympathy crunch'. In the 2008 thread below, there's a heated debate over the promise by leadership that sympathy crunch had been abolished was kept or not. Even if you ignore the comments from the person I knew at Bioware, these later comments from Bioware employees, especially the recruiting marketing post in 2013, seem to indicate that sympathy crunch was not abolished until well after 2008.


Milez5858

EA gets a lot of the grief for their employment history.

For anyone considering work at Bioware, beware of them as well.

They use seriously cult like tactics to keep their employees towing the company line, but don't be fooled. The second you don't tow that line you'll be walked out the doors.

They love out of country workers because they don't understand the Canadian labour laws. They continually fire people with out warning. This is illegal in Canada. You must warn people of performance problems and give them a certain amount of warnings before you are allowed to fire someone.

They are smarter about it than EA by offering food and free ice cream and other on site amenities, but it all adds up to a lot of extra hours with no extra pay.


Milez5858

BTW you could say my post is somewhat personal, but I've not worked for Bioware. I have several friends that do. Three have been walked out the door in the last year for refusing to work more than the 40 hours they are paid for and wanting to spend time with their wives and kids.

The friends I have remaining there are from out of country and feel as though they are somewhat trapped. They are unhappy but will only admit it behind very closed doors because they have it in their head they will get black listed or something.

I'm an outside observer. I get paid more than these poor kids, and I only work about 35 hours a week. I've always put MY life ahead of my employers, but work very hard and dedicated at my job. When I see what in my opinion amounts to cult like mind control over these young men who are enamoured by the legend of Ray and Greg, the founders of bioware, I'm almost sickened.

I used to think it would be cool to work in the gaming industry, and now I'm just happy as hell I'm not in it.

I will certainly be pointing them to this site as a resource and hope, that like any other entertainment industry, they get organized in some fashion. It's absolutely dehumanizing what this industry does to people.


Arty

Hell is happy?

And, didn't EA buy BioWare?

Just asking, not complaining. Welcome aboard!


Anguirel

Yes, Bioware and Pandemic were bought by EA. However, it sounds like this started well before that acquisition. It actually sounds more like what amounts to a Cult of Personality (such as you see at Blizzard, Maxis, id, or Ion Storm) where a single person or a few people have such a huge reputation that they can get a more-or-less unlimited supply of reasonably good new hires -- and thus, someone (possibly someone in between said person and the average worker) takes advantage and pushes the employees much harder than they should.

In some cases the cults that build up around certain individuals insulates them from bad working conditions (I've heard Maxis enjoyed that happy fate while Will Wright was there, keeping the people inside in much better conditions than the remainder of EA), but in many cases it results in an attitude of "if you don't like it, we can get any number of people to replace you." Which is what EA certainly had for a long time, and several other studios still have, though the people who work at them tend to be quieter about it (and I don't think any other studios ever got to EA's legendary status of continuous crunch).


Milez5858

Yes, these things were happening well ahead of the EA purchase of Bioware.

You are exactly accurate about the Cult of Personality thing. People think that Ray and Greg are looking out for them. Bioware Pandemic just sold for nearly a billion dollars. People are marked for assination for far smaller stakes. Can anyone really believe that the owners are in there saying... "ya know.. I can't accept your billion dollars unless I know the employees are really well taken care of". It defies logic.

I suspect it will just get worse. Now people that have been forced to leave the company are also being told the the MUST sell their stocks back to the company.

I don't know enough about it to say if it's legal or not, but it sure sounds fishy to me. Again I've recommended to people that they check with a lawyer first, but nobody wants any more hastle than they already have. Unfortunately it's this attitude that keeps the gaming industry from getting organized.

I would love to be able to say this sort of putrid intimidation is anomalous, but that would just be the utterest of untruths plaguing the industry. Maybe we should extract this ubiquitous haughtiness from the development process, I'm crazy enough to believe this can happen without the U-word.


Anonymous

The truth is life at Bioware is not as bad and as bad as implied in this article. But mostly not as bad.

The original article states that Bioware uses cult tactics. I dont know what cult tactics are so I cant give a simple answer but I know that Bioware uses traditional tactics to make for good morale. They give you the free breakfast. They bring you dinner if you are working overtime. They give out Oilers tickets or tickets to theatre or gokarts and the like. They let you wander away from your desk for an hour to go to the lunch room and take a nap on the sofa or play video games. Or they let you leave to run errands or go out for an hour for coffee. They also have company meetings where they highlight the development of the various projects ongoing. On this last one some think its a great way to keep up on other projects while some think its just a way to keep employees excited about everything. Is any of this a cult tactic?

Bioware is notoriously loyal. And the managers are notoriously wimpy and avoid confrontation. There isnt a way to emphasise this enough. In fact people joke that they havent done good work in months and yet they receive a strong review and a raise or more stock options. Getting fired as a fulltime employee from Bioware is a shocking rare event that there has been company meetings or emails sent out to explain the firing. Employees are given so many chances to get it right when they do get a bad review. This is all different for contract employees. A contract employee who sucks wont be fired but they will have their contract not renewed. A good contract employee is always offered an opening for fulltime (since contract employees do get overtime and making them fulltime saves money) or if there is none has their contract renewed.

I dont know much about HR so I dont know about the hiring tactics but I can agree that a lot of employees come from around the world or outside of the Edmonton area. It was always assumed that they couldnt find good talent around Edmonton. I cant comment much more than that because I dont know. I know there are lots of recruiting drives all over.

Hours are definitely a problem and I can agree whole hearted:

Some think it gets better each project cycle but its just transposed. In the early days the staff might work 30 to 40 hour weeks for a year or two and then come to the end and realize there was too much to be done. They spend the last few months working at least double the hours. In later projects things were more controlled by project managers and producers and it turned into what some call death crunch or death marches. Employees start working 50 hour weeks a year or two before ship. It isnt much extra hours and noone complains much but there are problems if an employee wants to make plans and never knows if she has to work or not. Managers are good about making sure employees can take time off and get lots of extra time off as compensation for the work but employees are still asked a lot of them.

When the time comes closer to release the employees have their hours scaled up more and more. It might start as 9-9 on Tuesday and Thursday, and then become 9-9 Monday until Thursday. Then it's Saturday 10-4. Then it's 9-9 Friday nights too. Then it's 9-5 Saturday. And in desperate times they even say maybe 12-4 on Sunday.

Morale gets low when employees think the game is awful and they cant get it done right. Thats when management tells the staff that they have decided to not ship the game until it is done and that they are extending the release date. This is good for the title but still hurts morale when employees think of having to work so much longer when they were working so hard to make a date. For example to use the most recent title Mass Effect employees were told the game would ship in Christmas of 2006. Then it was pushes to February of 2007 and then later spring and then June. But we know the game didnt get shipped until November. That does wear on employees.

Each time it gets pushed back the management cuts hours for a few weeks or give out breaks of a few extra days off or a four days weekend to recharge employees. Plus people say "in the old days we worked 90 hours or 100 or more. Now its only 50 or 60 or maybe in rare times sometimes 70 so this is great." which is meant to make you remember that you are making a game and should be happy to hang out at Bioware for only 50 or 60 hours a week to make “the best games ever”.

After game is shipped people take long breaks weeks at a time. Then they slowly ease back into it. They get maybe 20 or 30 hours of work assigned per week. This can go on for months before employee returns to regular 40 hour weeks. And that happens for many months or a year before the project pushes for release and the cycle starts again.

This is not an indictment of Bioware. All companies in video gaming do this. It's unfair to point at Bioware as any exception but a good one to me.

People leave all the time for these reasons. More have left in the last year. Maybe two years. Maybe three. Im not sure. But Bioware hires so many people and the ones who leave are generally not as good as the ones who are hired so it works out. Many people in Bioware wish more people would leave. Maybe with less dead weight since no one gets fired ever there could be a stronger staff who works more efficiently.

When EA Spouse was out everyone at Bioware got curious about it. But then the reports about what EA Spouse's husband was working and the situations he was put into and everyone at Bioware realised that they had nothing anywhere as close to as bad as what that was. Bioware employees enjoy comfort and support by managers and long projects but not 100 hour weeks. And some employees do manage to hold on to 9-5 for very long stretches if they can get their work done to highest standards.

But still people hoped that the industry in general would improve. Many people at Bioware would be thrilled to have nothing else change other than to never have to work more than 40 or 45 hours. But also many people at Bioware are workaholics who would never work less than 50 or 60 and they always create a dangerous precedent and control the pace unfairly.

Everyone at Bioware is aware that EA owns them now but no one is more thoughtful about it. Nothing has changed. Life is the same in every way except for more money. Its easy to forget that Bioware was ever bought by EA because the culture is unchanged and no one from EA is coming in and yelling about how things have to be changed. Thats kind of amazing considering Bioware doesnt make really big selling titles and probably deserves someone like EA coming in and saying this is how to do it. Bioware titles eventually after a year or two sell a million or two m but they never have that 5 m in the first month kind of release that the big titles get and that Bioware sorely wants.

If you want to work video games you are going to work lots of hours no matter where you go but Bioware is a great place to do it because they do treat you well. Many people leave Bioware and write back to say they regret it. Some come back. Some also do find a better life elsewhere but say that Bioware life is good too and that they miss many facets of it. I think more people leave Bioware to get away from Deadmonton then any other reason!!

Bioware censoring that article if they did isnt surprising to me because they are very controlling of their image. They only want positive talk about them and want to protect fragile egos and morale of the employee staff. Censoring bad publicity doesnt make the bad publicity true. Bioware just doesnt want that out there.


Anonymous

I've worked my share of crunches (over almost 400 hours in three months during summer) on various projects. My takeaway from that is:

1) Don't be an ass about it. If you need to crunch, admit what the reason is and create a sensible plan for the crunch.

People will take their free time any way they can. Crunches that last over two weeks are too much without breaks in-between and lead to more errors than actual work being done.

2) Realize that full day's job has about average 5-6 hours of actual work that benefits the company. With crunch, you can have people in the office for 12+ hours a day, but the additional hours don't really pay off that well in comparison.

3) Pretty much every developer I know is very savvy with the industry and how it operates. Most of them put their family and life above work and wouldn't hesitate to resign in a minute if the crunch or the company seems unfair or badly managed. Again, don't be an ass about the crunch.

4) I don't want to spend years and make a shit game. It's a waste of my life and time. I'll crunch for you if you plan it with a brain in the head. If you don't, I'll do something more relevant with my life.

What anonymous posts about Bioware sounds very much like normal circumstances and I'd be willing to work in a company like that. I want to wander away from my desk to play a demo or whatever and I like people trusting me that yes, I will do my tasks by the deadline. In fact, I think that's how every game company should operate. I'm glad I haven't experienced it any other way yet during the years.


Anonymous

While those kinds of hours may be typical or expected, they are in no way excusable. Companies that ask people to work 60 hours weeks for long periods of time in the middle of a project are either a) incompetent project managers or b) guilty of taking advantage of their teams.

Just because it's the video game industry, many people think that it is just par for the course. I call shenanigans, and so did Erin Hoffman (EA Spouse). Thank you, Erin, for getting this kind of crap out of the closet.

Not every company is like that. I work at a fantastic game company right now that is creating a AAA game for the Wii. We've hit some tough times at points, but our crunch times are 50 hour weeks, and we almost never do them back-to-back.

The idea that you have to suck up and deal sometimes in games? Yes, absolutely. We're way too young as an industry with our production practices and there is a ton of money at stake. The idea that you should be forced to do long stretches of 60 or more hour weeks out of love for a project? As a responsible company owner, you need to turn around and either put some more resources onto the problem, or start cutting scope. Because at that point, you're abusing your team for the mistakes you have made.


Anonymous

WOW, you folks are lucky to think that 60 hour weeks are some kind of exception in modern corporate America. I'm a gamer, not someone who works in the games industry, but I do work in film production in Hollywood, and these hours are truly par for the course out here. I'm talking about EVERY DAY, in by 9 am and you don't leave before 7:30 pm, and you're expected to take home scripts or novels to read and write up for the following day. This is not high paid, either, average salary for a "creative executive" is in the 50K range and you're working 65 hour weeks, not including any reading or additional work that's required outside of the office.

I'm not saying this is excusable, just that this seems to be the trend in America today, and it's afflicting a lot of industries, from banking to lawyering to consulting to video games and film. And the sad truth is, most of the "work" people are doing during their 10 or 12 hour work day could be finished in 5 hours if employees weren't such believers in presentee-ism, or the idea that the amount of hours you sit at a desk = your productivity as a worker.

Anyway, I just wanted to chime in to say these abuses are not symptomatic of the games industry but American white collar work in general, so... beware! The fact is, in competitive industries like these, there are so many willing workers who will step in and suffer these kinds of abuses that we have little power to organize or protect ourselves. Good luck out there.


Anonymous

I worked at Bio for a few years and I can verify that what the OP said is true. Greg and Ray have spent years cultivating a perception in the gaming industry that they are somehow better then other companies. They aren't. I am not saying they are monsters, because in person they are both pretty cool guys. However, when it comes to business they are pretty ruthless. I think it's mostly Ray. They continually promise that things will get better and they don't. Almost every project at Bio has had extended crunch because the project directors always plan way more than they are able to deliver. So, the employees suffer and the directors get a pat on the back and bigger and more shares. The latest example is Mass Effect. There was a 9 month crunch on that game. Some people came close to nervous breakdowns. They implemented sympathetic crunch which they also promised they had abolished. That's where the whole team has to be on site just in case something goes wrong even if they don't have anything to do. What it's really about though is the politics of making sure that the programmers don't get pissed that the artist got off early or whoever. I think it's just going to get worse under EA. Eventually people will realize that BioWare is just like every other crunchy game dev out there.


Anonymous

(FYI, I posted before, but with the amount of anonymous posters I'll clarify I'm the one who spoke about a 400 hour summer crunch)

I've never understood or had to withstand a sympathetic crunch - Even though our projects had crunches, it was more about sharing workload or just realizing that yes, the coders do have more work ahead for them.

I recently talked to an U.S friend of mine who was astounded by my 4 week summer vacation and 1 week winter vacation. She couldn't really imagine it since she had never had one. Most she had was max 1 week off during a year. Add to that the 10+ hour workdays and I can't understand how you cope with that.

I work in northern europe, do 8 hour days 90% of the time and at my current place I can show the total crunch hours with 10 fingers.

When you're young, 400 hour crunches and sleeping in a sleeping bag might not be a big deal, but nearing 30, you'll get more interested about your rights and such. The 400 hours for me was a good eye opener on how not to do things. It was valuable, but never again. It took me more than a year to recover the friends and social aspects of my life to recover from that.


Anonymous

I think more people leave Bioware to get away from Deadmonton then any other reason!!

Hey! Nuts to you!

(Edmontonian here)

I'm acquainted with some people in Bioware. Their loyalty to the company is astonishing - if, as previous posters have said, there is a cultivated sense that Bioware's better than other game companies then it's absolutely taken root. They joke that it feels like almost living in a self-sustained arcology.

I remember one of them dismissing EA's Bioware takeover as no big deal, business as usual, why would anything change blah blah blah. I'm sure it was supposed to sound reassuring but it came off as naively dismissive in light of EA's infamous track record.

This is the first time I've actually seen numbers of their crunch time which is disappointing if true. It smacks of poor management and I guess I had some of that fairly-tale view of the company. I hope for Bioware's sake it doesn't get any worse.


Anonymous

One anonymous (or, from this page, I'm the anonymous [who defended Bioware earlier] to another I will respond to this comment -- They implemented sympathetic crunch which they also promised they had abolished.

My response is that they didn't implement that (??). People were on call of course but they didn't tell every one to be there because some people had to work. That's incorrect. If a manager thought their team had deliverables to make for sprints then they told their team to be there. If people were behind they had to be there. If people could help the team they had to be there. But that was almost always up to individual managers and if an employee had to be there just because then that is the fault of an individual manager and that individual manager should have been discussed with Casey or the project manager.

Not disagreeing with the rest of your post which pretty much said the same as mine. But this one point was inaccurate.


Anonymous

Anon in response to your argument about the sympathetic crunch; Of course it's the manager's fault, but do you really think anyone is going to buck Casey and not suffer for it? They never come right out and say that people have to be there for political reasons, they couch it in all kind of ways. All I know is that I spent more than a few nights at work until 2:00 or 3:00AM because something might go wrong even though I lived less than 30 minutes away and told them I could be called at any time. I also know that I was taken to task for always asking to leave when I was done because of the perception it created amongst people that were forced to stay.

show more
How does Boston compare to SV and what do MIT and Stanford have to do with it?
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2010-01-01 00:00:00 | Created: 2026-07-23 05:18:40

This is an archive of an old Google Buzz conversation on MIT vs. Stanford and Silicon Valley vs. Boston

There's no reason why the Boston area shouldn't be as much a hotbed of startups as Silicon Valley is. By contrast, there are lots of reasons why NYC is no good for startups. Nevertheless, Paul Graham gave up on the Boston area, so there must be something that hinders startup formation in the area.

Kevin: This has nothing to do with money, or talents, or what it. All it matters is "entrepreneur density".

Boston may have the money, the talent, the intelligence, but does it have an entrepreneurial spirit and enough of a density?

Marya: From http://www.xconomy.com/boston/2009/01/22/paul-graham-and-y-combinator-to-leave-cambridge-stay-in-silicon-valley-year-round/ "Graham says the reasons are mostly personal, having to do with the impending birth of his child and the desire not to try and be a bi-coastal parent" But then immediately after, we see he says: "Boston just doesn’t have the startup culture that the Valley does. It has more startup culture than anywhere else, but the gap between number 1 and number 2 is huge; nothing makes that clearer than alternating between them." Here's an interview: http://www.xconomy.com/boston/2009/03/10/paul-graham-on-why-boston-should-worry-about-its-future-as-a-tech-hub-says-region-focuses-on-ideas-not-startups-while-investors-lack-confidence/ Funny, because Graham seemed partial to the Boston area, earlier: http://www.paulgraham.com/cities.html http://www.paulgraham.com/siliconvalley.html

Rebecca: I think he's partial because he likes the intellectual side of Boston, enough to make him sad that it doesn't match SV for startup culture. I know the feeling. I guess I have seen things picking up here recently, enough to make me a little wistful that I have given my intellectual side priority over any entrepreneurial urges I might have, for the time being.

Scoble: I disagree that Boston is #2. Seattle and Tel Aviv are better and even Boulder is better, in my view.

Piaw: Seattle does have a large number of Amazon and Microsoft millionaires funding startups. They just don't get much press. I wasn't aware that Boulder is a hot-bed of startup activity.

Rebecca: On the comment "there is no reason Boston shouldn't be a hotbed of startups..." Culture matters. MIT's culture is more intellectual than entrepreneurial, and Harvard even more so. I'll tell you a story: I was hanging out in the MIT computer club in the early nineties, when the web was just starting, and someone suggested that one could claim domain names to make money reselling them. Everyone in the room agreed that was the dumbest idea they had ever heard. It was crazy. Everything was available back then, you know. And everyone in that room kindof knew they were leaving money on the ground. And yet we were part of this club that culturally needed to feel ourselves above wanting to make money that way. Or later, in the late nineties I was hanging around Philip Greenspun, who was writing a book on database backed web development. He was really getting picked on by professors for doing stuff that wasn't academic enough, that wasn't generating new ideas. He only barely graduated because he was seen as too entrepreneurial, too commercial, not original enough. Would that have happened at Stanford? I read an interview with Rajiv Motwani where he said he dug up extra disk drives whenever the Google founders asked for them, while they were still grad students. I don't think that wouldn't happen at MIT: a professor wouldn't give a grad student lots of stuff just to build something on their own that they were going to commercialize eventually. They probably would encounter complaints they weren't doing enough "real science". There was much resentment of Greenspun for the bandwidth he "stole" from MIT while starting his venture, for instance, and people weren't shy about telling him. I'm not sure I like this about MIT.

Piaw: One my friends once turned down a full time offer at Netscape (after his internship) to return to graduate school. He said at that time, "I didn't go to graduate school to get rich." Years later he said, "I succeeded... at not getting rich."

Dan: As the friend in question (I interned at Netscape in '96 and '97), I'm reasonably sure I wouldn't have gotten very rich by dropping out of grad school. Instead, by sticking with academia, I've managed to do reasonably well for myself with consulting on the side, and it's not like academics are paid peanuts, either.

Now, if I'd blown off academia altogether and joined Netscape in '93, which I have to say was a strong temptation, things would have worked out very differently.

Piaw: Well, there's always going to be another hot startup. :-) That's what Reed Hastings told me in 1995.

Rebecca: A venture capitalist with Silicon Valley habits (a very singular and strange beast around here) recently set up camp at MIT, and I tried to give him a little "Toto, you're not in Kansas anymore" speech. That is to say, I was trying to tell him that the habits one got from making money from Stanford students wouldn't work at MIT. It isn't that one couldn't make money investing in MIT students -- if one was patient enough, maybe one could make more, maybe a lot more. But it would only work if one understood how utterly different MIT culture is, and did something different out of an understanding of what one was buying. I didn't do a very good job talking to him, though; maybe I should try again by stepping back and talking more generally about the essential difference of MIT culture. You know, if I did that, maybe the Boston mayor's office might want to hear this too. Hmmm... you've given me an idea.

Marya: Apropos, Philip G just posted about his experience attending a conference on angel investing in Boston: http://blogs.law.harvard.edu/philg/2010/06/01/boston-angel-investors/ He's in cranky old man mode, as usual. I imagine him shaking his cane at the conference presenters from the rocking chair on his front porch. Fun quotes: 'Asked if it wouldn’t make more sense to apply capital in rapidly developing countries such as Brazil and China, the speakers responded that being an angel was more about having fun than getting a good return on investment. (Not sure whose idea of “fun” included sitting in board meetings with frustrated entrepreneurs, but personally I would rather be flying a helicopter or going to the beach.)... 'Nobody had thought about the question of whether Boston in fact needs more angel investors or venture capital. Nobody could point to an example of a good startup that had been unable to obtain funding. However, there were examples of startups, notably Facebook, that had moved to California because of superior access to capital and other resources out there... 'Nobody at the conference could answer a macro question: With the US private GDP shrinking, why do we need capital at all?'

Piaw: The GDP question is easily answered. Not all sectors are shrinking. For instance, Silicon Valley is growing dramatically right now. I wouldn't be able to help people negotiate 30% increases in compensation otherwise (well, more like 50% increases, depending on how you compute). The number of pre-IPO companies that are extremely profitable is also surprisingly high.

And personally, I think that investing in places like China and Brazil is asking for trouble unless you are well attuned to the local culture, so whoever answered the question with "it's fun" is being an idiot.

The fact that Facebook was asked by Accel to move to Palo Alto should definitely be something Boston area VCs should berate themselves about. But that "forced move" was very good for Facebook. They acquired Jeff Rothschild, Marc Kwiatkowski, Steve Grimm, Paul Bucheit, Sanjeev Singh, and many others by being in Palo Alto that would not have moved to Boston for Facebook no matter what. It's not clear to me that staying in Boston was an optimal move for Facebook no matter what. At least, not before things got dramatically better in Boston for startups.

Marya: The GDP question is easily answered. Not all sectors are shrinking. For instance, Silicon Valley is growing dramatically right now

I'm guessing medical technology and biotech are still growing. What else?

Someone pointed this out in the comments, and Philip addressed it; he argues that angel investors are unlikely to get a good return on their investment (partial quote): "...we definitely need some sources of capital... But every part of the U.S. financial system, from venture capital right up through investment banks, is sized for an expanding private economy. That means it is oversized for the economy that we have. Which means that the returns to additional capital should be very small...."

He doesn't provide any supporting evidence, though.

Piaw: Social networks and social gaming is growing dramatically and fast.

Rebecca: Thanks, Marya, for pointing out Philip's blog post. I think the telling quote from it is this: "What evidence is there that the Boston area has ever been a sustainable place for startups to flourish? When the skills necessary to build a computer were extremely rare, minicomputer makers were successful. As soon as the skills ... became more widespread, nearly all of the new companies started up in California, Texas, Seattle, etc. When building a functional Internet application required working at the state of the art, the Boston area was home to a lot of pioneering Internet companies, e.g., Lycos. As soon as it became possible for an average programmer to ... work effectively, Boston faded to insignificance." Philip is saying Boston can only compete when it can leverage skills that only it has. That's because its ability to handle business and commercialization are so comparatively terrible that when the technological skill becomes commoditized, other cities will do much better.

But it does often get cutting-edge technical insight and skills first -- and then completely drops the ball on developing them. I find this frustrating. Now that I think about it, it seems like Boston's leaders are frustrated by this too. But I think they're making a mistake trying to remake Boston in Silicon Valley's image. If we tried to be you, at best we would be a pathetic shadow of you. We could only be successful by being ourselves, but getting better at it.

There is a fundamental problem: the people at the cutting edge aren't interested in practical things, or they wouldn't be bothering with the cutting edge. Though it might seem strange to say now, the guy who set up the hundredth web server was quite an impractical intellectual. Who needs a web server when there are only 99 others (and no browsers yet, remember)? We were laughing at him, and he was protesting the worth of this endeavor merely out of a deep intellectual faith that this was the future, no matter how silly it seemed. Over and over I have seen the lonely obsessions of impractical intellectuals become practical in two or three years, become lucrative in five or eight, and become massive industries in seven to twelve years.

So if the nascent idea that will become a huge industry in a dozen years shows up first in Boston, why can't we take advantage of it? The problem is that the people who hone their skill at nascent ideas that won't be truly lucrative for half a decade at least, are by definition impractical, too impractical to know how to take advantage of being first. But maybe Boston could become a winner if it could figure out how to pair these people up with practical types who could take advantage of the early warning about the shape of the future, and leverage the competitive advantage of access to skills no-one else has. It would take a very particular kind of practicality, different from the standard SV thing. Maybe I'm wrong, though; maybe the market just doesn't reward being first, especially if it means being on the bleeding edge of practicality. What do you think?

Piaw: Being 5 or 10 years ahead of your time is terrible. What you want to be is just 18 months or even 12 months ahead of your time, so you have just enough time to build product before the market explodes. My book covers this part as well. :-)

Marya: Rebecca, I don't know the Boston area well enough to form an opinion. I've been here two years, but I'm certainly not in the thick of things (if there is a "thick" to speak of, I haven't seen it). My guess would be that Boston doesn't have the population to be a huge center of anything, but that's a stab in the dark.

Even so, this old survey (2004) says that Boston is #2 in biotech, close behind San Diego: http://www.forbes.com/2004/06/07/cz_kd_0607biotechclusters.html So why is Boston so successful in biotech if the people here broadly lack an interest in business, or are "impractical"? (Here's a snippet from the article: "...When the most successful San Diego biotech company, IDEC Pharmaceuticals, merged with Biogen last year to become Biogen Idec (nasdaq: BIIB - news - people ), it officially moved its headquarters to Biogen's hometown of Cambridge, Mass." Take that, San Diego!)

When you talk about a certain type of person being "impractical", I don't think that's really the issue. Such people can be very practical when it comes to pursuing their own particular kind of ambition. But their interests may not lie in the commercialization of an idea. Some extremely intelligent, highly skilled people just don't care about money and commerce, and may even despise them.

Even with all that, I find it hard to believe that the intelligentsia of New England are so much more cerebral than their cousins in Silicon Valley. There's certainly a puritan ethic in New England, but I don't think that drives the business culture.

Rebecca: Marya, thanks for pointing out to me I wasn't being clear (I'm kindof practicing explaining something on you, that I might try to say more formally later, hence the spam of your comment field. I hope you don't mind.) You question " why is Boston so successful in biotech if the people here broadly lack an interest in business?" made me realize I'm not talking about people broadly -- there are plenty of business people in Boston, as everywhere. I'm talking about a particular kind of person, or even more specifically, a particular kind of relationship. Remember I contrasted the reports of Rajeev Motwani's treatment of the Google guys with the MIT CS lab's treatment of Philip? In general, I am saying that a university town like Palo Alto or Cambridge will be a magnet for ultra-ambitious young people who look for help realizing their ambitions, and a group of adults who are looking to attract such young people and enable those ambition, and there is a characteristic relationship between them with (perhaps unspoken) terms and expectations. The idea I'm really dancing around is that these terms & expectations are very different at MIT than (I've heard) they are at Stanford. Though there may not be very many people total directly involved in this relationship, it will still determine a great deal of what the city can and can't accomplish, because it is a combination of the energy of very ambitious young people and the mentorship of experienced adults that makes big things possible.

My impression is that the most ambitious people at Stanford dream of starting the next big internet company, and if they show enough energy and talent, they will impress professors who will then open their Rolodex and tell their network of VC's "this kid will make you tons of money if you support his work." The VC's who know that this professor has been right many times before will trust this judgement. So kids with this kind of dream go to Stanford and work to impress their professors in a particular kind of way, because it puts them on a fast track to a particular kind of success.

The ambitious students most cultivated by professors in Boston have a different kind of dream: they might dream of cracking strong AI, or discovering the essential properties of programming languages that will enable fault-tolerant or parallel programming, or really understanding the calculus of lambda calculus, or revolutionizing personal genomics, or building the foundations of Bladerunner-style synthetic biology. If professors are sufficiently impressed with their student's energy and talent, they will open their Rolodex of program managers at DARPA (and NSF and NIH), and tell them "what this kid is doing isn't practical or lucrative now, nor will it be for many years to come, but nonetheless it is critical for the future economic and military competitiveness of the US that this work is supported." The program managers' who know that this professor has been right many times before will trust this judgment. In this way, the kid is put on a fast track to success -- but it is a very different kind of success than the Stanford kid was looking for, and a different kind of kid who will fight to get onto this track. The meaning of success is very different, much more intellectual and much less practical, at least in the short term.

That's what I mean when I say "Boston" is less interested in business, more impractical, less entrepreneurial. It isn't that there aren't plenty of people here who have these qualities. But the "ecosystem" that gives ultra-ambitious young people the chance to do something singular which could be done no-where else -- an ecosystem which that it does have, but in a very different kind of way -- doesn't foster skill at commercialization or an interest in the immediate practical application of technology.

Maybe there is nothing wrong with that: Boston's ecosystem just fosters a different kind of achievement. However, I can see it is frustrating to the mayor of Boston, because the young people whose ambitions are enabled by Boston's ecosystem may be doing work crucial to the economic and military competitiveness of the US in the long term, but they might not help the economy of Boston very much! What often happens in the "long term" is that the work supported by grants in Boston develops to the point it becomes practical and lucrative, and then it gets commercialized in California, Seattle, New York, etc... The program managers at DARPA who funded the work are perfectly happy with this outcome, but I can imagine that the mayor of Boston is not! The kid also might not be 100% happy with this deal, because the success which he is offered isn't much like SV success -- its a fantastic amount of work, rather hermit-like and self-abnegating, which mostly ends up making it possible for other people far away to get very, very rich using the results of his labors. At best he sees only a minuscule slice of the wealth he enabled.

What one might want instead is that the professors in Boston have two sections in their Rolodex. The first section has the names of all the relevant program managers at DARPA, and the professor flips to this section first. The second section has the names of suitable cofounders, and friendly investors, and after the student has slaved away for five to seven years making a technology practical, the professor flips to the second section and sets the student up a second time to be the chief scientist or something like that at an appropriate startup.

And its not like this doesn't happen. It does happen. But it doesn't happen as much as it could, and I think the reason why it doesn't may be that it just takes a lot of work to maintain a really good Rolodex. These professors are busy and they just don't have enough energy to be the linchpin of a really top-quality ecosystem in two different ways at the same time.

If the mayor of Boston is upset that Boston is economically getting the short end of the stick in this whole deal (which I think it is), a practical thing he could do is give these professors some help in beefing up the second section of their Rolodex, or perhaps try to build another network of mentors which was in the appropriate way Rolodex-enabled. If he took the later route, he should understand that this second network shouldn't try to be a clone of the similar thing at Stanford (because at best it would only be a pale shadow) but instead be particularly tailored to incorporating the DARPA-project graduates that are unique to Boston's ecosystem. That way he could make Boston a center of entrepreneurship in a way that was uniquely its own and not merely a wannabe version of something else -- which it would inevitably do badly. That's what I meant when I said Boston should be itself better, rather than trying to be a poor pale copy of Silicon Valley.

Piaw: I like that line of thought Rebecca. Here's the counter-example: Facebook. Facebook clearly was interested in monetizing something that was very developed, and in fact, had been tried and failed many times because the timing wasn't right. Yet Facebook had to go to Palo Alto to get funding. So the business culture has to change sufficiently that the people with money are willing to risk it on very high risk ventures like the Facebook that was around 4 years ago.

Having invested my own money in startups, I find that it's definitely something very challenging. It takes a lot to convince yourself that this risk is worth taking, even if it's a relatively small portion of your portfolio. To get enough people to build critical mass, you have to have enough success in prior ventures to gain the kind of confidence that lets you fund Facebook where it was 4 years ago. I don't think I would have been able to fund Google or Facebook at the seed stage, and I've lived in the valley and worked at startups my entire career, so if anyone would be comfortable with risk, it should be me.

Dan: Rebecca: a side note on "opening a rolodex for DARPA". It doesn't really work quite like that. It's more like "hey, kid, you should go to grad school" and you write letters of recommendation to get the kid into a top school. You, of course, steer the kid to a research group where you feel he or she will do awesome work, by whatever biased idea of awesomeness.

My own professorial take: if one of my undergrads says "I want to go to grad school", then I do as above. If he or she says "I want to go work for a cool startup", then I bust out the VC contacts in my rolodex.

Rebecca: Dan: I know. I was oversimplifying for dramatic effect, just because qualifying it would have made my story longer, and it was already pushing the limits of the reasonable length for a comment. Of course the SV version of the story isn't that simple either.

I have seen it happen that sufficiently brilliant undergraduates (and even high school students -- some amazing prodigies show up at MIT) can get direct support. But realize also I'm really talking about grad students -- after all, my comparison is with the relationship between the Google guys and Rajeev Motwani, which happened when they were graduate students. The exercise was to compare the opportunities they encountered with the opportunities similarly brilliant, energetic and ultra-ambitious students at MIT would have access to, and talk about how it would be similar and different. Maybe I shouldn't have called such people "kids," but it simplified and shortened my story, which was pushing its length limit anyway. Thanks for the feedback; I'm testing out this story on you, and its useful to know what ways of saying things work and what doesn't.

Rebecca: Piaw: I understand that investing in startups by individual is very scary. I know some Boston angels (personally more than professionally) and I hear stories about how cautious their angel groups are. I should explain some context: the Boston city government recently announced a big initiative to support startups in Boston, and renovate some land opened up by the Big Dig next to some decaying seaport buildings to create a new Innovation District. I was thinking about what they could do to make that kind of initiative a success rather than a painful embarrassment (which it could easily become). So I was thinking about the investment priorities of city governments, more than individual investors like you.

Cities invest in all sorts of crazy things, like Olympic stadiums, for instance, that lose money horrifyingly ... but when you remember that the city collects 6% hotel tax on every extra visitor, and benefits from extra publicity, and collects extra property tax when new people move to the city, it suddenly doesn't look so bad anymore. Boston is losing out because there is a gap in the funding of technology between when DARPA stops funding something, because it is developed to the point where it is commercializable, and when the cautious Boston angels will start funding something -- and other states step into the gap and get rich off of the product of Massachusetts' tax dollars. That can't make the local government too happy.

Maybe the Boston city or state government might have an incentive to do something to plug that hole. They might be more tolerant of losing money directly because even a modestly lucrative venture, or one very, very slow to generate big returns, which nonetheless successfully drew talent to the city would make them money in hotel & property tax, publicity etc. etc. -- or just not losing the huge investment they have already made in their universities! I briefly worked for someone who was funded by Boston Community Capital, an organization which, I think, divided its energies between developing low income housing and and funding selected startups that were deemed socially redeeming for Boston. When half your portfolio is low-income housing, you might have a different outlook on risk and return! I was hugely impressed by what great investors they were -- generous, helpful & patient. Patience is necessary for us because the young prodigies in Boston go into fields whose time horizon is so long -- my friends are working on synthetic biology, but it will be a long, long time before you can buy a Bladerunner-style snake!

Again, thanks for the feedback. You are helping me understand what I am not making clear.

Marya: Rebecca, you said The idea I'm really dancing around is that these terms & expectations are very different at MIT than (I've heard) they are at Stanford

I read your initial comments as being about general business conditions for startups in Boston. But now I think you're mainly talking about internet startups or at least startups that are based around work in computer science. You're saying MIT's computer science department in particular does a poor job of pointing students in an entrepreneurial direction, because they are too oriented towards academic topics.

Both MIT and Stanford have top computer science and business school rankings. Maybe the problem is that Stanford's business school is more inclined to "mine" the computer science department than MIT's?

Doug: Rebecca, your description of MIT vs. Stanford sounds right to me (though I don't know Stanford well). What's interesting is that I remember UC Berkeley as being very similar to how you describe MIT: the brightest/most ambitious students at Cal ended up working on BSD or Postgres or The Gimp or Gnutella, rather than going commercial. Well, I haven't kept up with Berkeley since the mid-90s, but have there been any significant startups there since Berkeley Softworks?

Piaw: Doug: Inktomi. It was very significant for its time.

Dan: John Ousterhout built a company around Tcl. Eric Allman built a company around sendmail. Mike Stonebreaker did Ingres, but that was old news by the time the Internet boom started. Margo Seltzer built a company around Berkeley DB. None of them were Berkeley undergrads, though Seltzer was a grad student. Insik Rhee did a bunch of Internet-ish startup companies, but none of them had the visibility of something like Google or Yahoo.

Rebecca: Dan: I was thinking more about what you said about not involving undergraduates, but instead telling them to go to grad school. Sometimes MIT is in the nice sedate academic mode which steers undergrads to the appropriate research group when they are ready to work on their PhD. But sometimes it isn't. Let me tell you more about the story of the scene in the computer club concerning installation of the first web server. It was about the 100th web server anywhere, and its maintainer accosted me with an absurd chart "proving" the exponential growth of the web -- i.e. a graph going exponentially from 0 to 100ish, which he extrapolated forward in time to over a million -- you know the standard completely bogus argument -- except this one was exceptionally audacious in its absurdity. Yet he argued for it with such intensity and conviction, as if he was saying that this graph should convince me to drop everything and work on nothing but building the Internet, because it was the only thing that mattered!

I fended him off with the biggest stick I could find: I was determined to get my money's worth for my education, do my psets, get good grades (I cared back then), and there is no way I would let that be hurt by this insane Internet obsession. But it continued like that. The Internet crowd only grew with time, and they got more insistent that they were working on the only thing that mattered and I should drop everything and join them. That I was an undergraduate did not matter a bit to anyone. Undergrads were involved, grad students were involved, everyone was involved. It wasn't just a research project; eventually so many different research projects blended together that it became a mass obsession of an entire community, a total "Be Involved or Be Square" kind of thing. I'd love to say that I did get involved. But I didn't; I simply sat in the office on the couch and did psets, proving theorems and solving the Schrodinger's equation, and fended them off with the biggest stick I could find. I was determined to get a Real Education, to get my money's worth at MIT, you know.

My point is that when the MIT ecosystem really does its thing, it is capable of tackling projects that are much bigger than ordinary research projects, because it can get a critical mass of research projects working together, involving enough grad students and also sucking in undergrads and everyone else, so that the community ends up with an emotional energy and cohesion that goes way, way beyond the normal energy of a grad student trying to finish a PhD.

There's something else too, though I cannot report on this with that much certainty, because was too young to see it all at the time. You might ask: if MIT had this kind of emotional energy focused on something in the 90's, then what is it doing in a similar way now? And the answer I'd have to say, painfully, is that it is frustrated and miserable about being an empty shell of what it once was.

Why? Because in 2000 Bush got elected and he killed the version of DARPA with which so many professors had had such a long relationship. I didn't I understand this in the 90's -- like a kid I took the things that were happening around me for granted without seeing the funding that made them possible -- but now I see that that the kind of emotional energy expended by the Internet crowd at MIT in the 90's costs a lot of money, and needs an intelligent force behind it, and that scale of money and planning can only come from the military, not from NSF.

More recently I've watched professors who clearly feel it is their birthright to be able to mobilize lots of student to do really large-scale projects, but then they try to find money for it out of NSF, and they spend all their time killing themselves writing grant proposals, never getting enough money to make themselves happy, and complaining about the cowardice of academia, and wishing they could still work with their old friends at DARPA. They aren't happy because they are merely doing big successful research projects, but a mere research project isn't enough... when MIT is really MIT it can do more. It is an empty shell of itself when it is merely a collection of merely successful but not cohesive NSF funded research projects. As I was saying, the Boston "ecosystem" has in itself the ability to do something singular, but it is singular in an entirely different way than SV's thing.

This may seem obscure, a tale of funding woes at a distant university, but perhaps it is something you should be aware of, because maybe it affects your life. The reason you should care is that when MIT was fully funded and really itself, it was building the foundations of the things that are now making you rich.

One might think of the relationship between technology and wealth like a story about potential energy: when you talk about finding a "product/market" fit, its like pushing a big stone up a hill, until you get the "fit" at the top of the hill, and then the stone rolls down and the energy you put into it spins out and generates lots of money. In SV you focus on pushing stones up short hills -- like Piaw said, no more than 12-18 months of pushing before the "fit" happens.

But MIT in its golden age could tackle much, much bigger hills -- the whole community could focus itself on ten years of nothing but pushing a really big stone up a really big hill. The potential energy that the obsessed Internet Crowd in the 90's was pushing into the system has been playing out in your life ever since. They got a really big stone over a really big hill and sent it down onto you, and then you pushed it over little bumps on the way down, and made lots of money doing it, and you thought the potential energy you were profiting from came entirely from yourselves. Some of it was, certainly, but not all. Some of it was from us. If we aren't working on pushing up another such stone, if we can't send something else over a huge hill to crash into you, then the future might not be like the past for you. Be worried.

So you might ask, how did this story end? If I'm claiming that there was intense emotional energy being poured into developing the Internet at MIT in the 90's, why didn't those same people fan out and create the Internet industry in Boston? If we were once such winners, how did we turn into such losers? What happened to this energetic, cohesive group?

I can tell you about this, because after years of fending off the emotional gravitation pull of this obsession, towards the end I began to relent. First I said "No way!" and then I said "No!" and then I said "Maybe Later," and then I said "OK, Definitely Later"... and then when I finally got around to Later, and (perhaps the standard story of my life) Later turned out to be Too Late. By 2000 I was ready to join the crowd and remake myself as an Internet Person in the MIT style. So I ended up becoming seriously involved just at the time it fell apart. Because 2000ish, almost the beginning of the Internet Era for you, was the end for us.

This weekend I was thinking of how to tell this story, and I was composing it in my head in a comic style, thinking to tell a story of myself as "Parable of Boston Loser" to talk about all my absurd mistakes as a microcosm of the difficulties of a whole city. I can pick on myself, can't I; no one will get upset at that? The short story is that in 2000ish the Internet crowd had achieved their product/market fit, DARPA popped the champagne -- you won guys! Congratulations! Now go forth and commercialize! -- and pushed us out of the nest into the big world to tackle the standard tasks of commercializing a technology -- the tasks that you guys can do in your sleep. I was there, right of the middle of things, during that transition. I thought to tell you a comic story about the absurdity of my efforts in that direction, and make you laugh at me.

But when I was trying to figure out how to explain what was making it so terribly hard for me, to my great surprise I was suddenly crying really hard. All Saturday night I was thinking about it and crying. I had repressed the memory, decided I didn't care that much -- but really it was too terrible to face. All the things you can do without thinking, for us hurt terribly. The declaration of victory, the "achievement of product/market fit", the thing you long for more than anything, I -- and I think many of the people I knew -- experienced as a massive trauma. This is maybe why I've reacted so vehemently and spammed your comment field, because I have big repressed personal trauma about all this. I realized I had a much more earnest story to tell than I had previously planned.

For instance, I was reflecting on my previous comment about what cities spend money on, and thinking that I sounded like the biggest jerk ever. Was I seriously suggesting that the city take money that they would have spent on housing for poor black babies and instead spend it on overeducated white kids with plenty of other prodigiously lucrative economic opportunities? Where do I get off suggesting something like that? If I really mean it I have a big, big burden of proof.

So I'll try to combine my more earnest story with at least a sketch of how I'd tackle this burden of proof (and try to keep it short, to keep the spam factor to a minimum. The javascript is getting slow, so I'll cut this here and continue.)

Ruchira: Interlude (hope Rebecca continues soon!): Rebecca says "that scale of money and planning can only come from the military, not from NSF." Indeed, it may be useful to check out this NY Times infographic of the federal budget: http://www.nytimes.com/interactive/2010/02/01/us/budget.html

I'll cite below some of the 2011 figures from this graphic that were proposed at that time; although these may have changed, the relative magnitudes of one sector versus another are not very different. I've mostly listed sectors in decreasing order of budget size for research, except I listed "General science & technology" sector (which includes NSF) before "Health" sector (which includes NIH) since Rebecca had contrasted the military with NSF.

The "Research, development, test, and evaluation" segment of the "National Defense" sector is $76.77B. I guess DARPA, ONR, etc. fit there.

The "General science & technology" sector is down near the lower right. The "National Science Foundation programs" segment gets $7.36B. There's also another $0.1B for "National Science Foundation and other". The "Science, exploration, and NASA supporting activities" segment gets $12.78B. (I don't know to what extent satellite technology that is relevant to the national defense is also involved here, or in the $4.89B "Space operations" segment, or in the $0.18B "NASA Inspector General, education, and other" segment.) The "Department of Energy science programs" segment gets $5.12B. The "Department of Homeland Security science and technology programs" segment gets $1.02B.

In the "Health" sector, the "National Institutes of Health" segment gets $32.09B. The "Disease control, research, and training" segment gets $6.13B (presumably this includes the CDC). There's also "Other health research and training" at $0.14B and "Diabetes research and other" at $0.095B.

In the "Natural resources and environment sector", the "National Oceanic and Atmospheric Administration" gets $5.66B. "Regulatory, enforcement, and research programs" gets $3.86B (is this the entire EPA?).

In the "Community and regional development" sector, the "National Infrastructure Innovation and Finance fund" (new this year) gets $4B.

In the "Agriculture" sector, which presumably includes USDA-funded research, "Research and education programs" gets $1.97B, "Research and statistical analysis" gets $0.25B, and "Integrated research, education, and extension programs" gets $0.025B.

In the "Transportation" sector, "Aeronautical research and technology" gets $1.15B, which by the way would be a large (130%) relative increase. (Didn't MIT find a way of increasing jet fuel efficiency by 75% recently?)

In the "Commerce and housing credit" sector, "Science and technology" gets $0.94B. I find this rather mysterious.

In the "Education, training, employment" sector, "Research and general education aids: Other" gets $1.14B. The "Institute for Education Sciences" gets $0.74B.

In the "Energy" sector, "Nuclear energy R&D" gets $0.82B and "Research and development" gets $0.024B (presumably this is the portion outside the DoE).

In the "Veterans' benefits and services" sector, "Medical and prosthetic research" gets $0.59B.

In the "Income Security" sector there's a tiny segment "Children's research and technical assistance" $0.052B. Not sure what that means.

Rebecca: I'll start with a non-sequitur which I hope to use to get at the hear of the difference between MIT and Stanford: recently I was at a Marine publicity event and I asked the recruiter what differentiates the Army from the Marines? Since they both train soldiers to fight, why don't they do it together? He answered vehemently that they must be separate because of one simple attribute in which they are utterly opposed: how they think about the effect they want to have on the life their recruits have after they retire from the service. He characterized the Army as an organization which had two goals: first, to train good soldiers, and second, to give them skills that would get them a good start in the life they would have after they left. If you want to be a Senator, you might get your start in the Army, get connections, get job skills, have "honorable service" on your resume, and generally use it to start your climb up the ladder. The Army aspires to create a legacy of winners who began their career in the Army.

By contrast the Marines, he said, have only one goal: they want to create the very best soldiers, the elite, the soldiers they can trust in the most difficult and dangerous situations to keep the Army guys behind them alive. This elite training, he said, comes with a price. The price you pay is that the training you get does not prepare you for anything at all in the civilian world. You can be the best of the best in the Marines, and then come home and discover that you have no salable civilian job skills, that you are nearly unemployable, that you have to start all over again at the bottom of the ladder. And starting over is a lot harder than starting the first time. It can be a huge trauma. It is legendary that Marines do not come back to civilian life and turn into winners: instead they often self-destruct -- the "transition to civilian life" can be violently hard for them.

He said this calmly and without apology. Did I say he was a recruiter? He said vehemently: "I will not try to recruit you! I want to you to understand everything about how painful a price you will pay to be a Marine. I will tell you straight out it probably isn't for you! The only reason you could possibly want it is because you want more than anything to be a soldier, and not just to be a soldier, but to be in the elite, the best of the best." He was saying: we don't help our alumni get started, we set them up to self-destruct, and we will not apologize for it -- it is merely the price you pay for training the elite!

This story gets to the heart of what I am trying to say is the essential difference between Stanford and MIT. Stanford is like the Army: for its best students, it has two goals -- to make them engineers, and to make them winners after they leave. And MIT is like the Marines: it has only one goal -- to make its very best student into the engineering elite, the people about whom they can truthfully tell program managers at DARPA: you can utterly trust these engineers with the future of America's economic and military competitiveness. There is a strange property to the training you get to enter into that elite, much like the strange property the non-recruiter attributed to the training of the Marines: even though it is extremely rigorous training, once you leave you can find yourself utterly without any salable skills whatever.

The skills you need to acquire to build the infrastructure ten years ahead of the market's demand for it may have zero intersection with the skills in demand in the commercial world. Not only are you not prepared to be a winner, you may not even be prepared to be basically employable. You leave and start again at the bottom. Worse than the bottom: you may have been trained with habits commercial entities find objectionable (like a visceral unwillingness to push pointers quickly, or a regrettable tendency to fight with the boss before the interview process is even over.) This can be fantastically traumatic. Much as ex-Marines suffer a difficult "transition to civilian life," the chosen children of MIT suffer a traumatic "transition to commercial life." And the leaders at MIT do not apologize for this: as the Marine said, it is just the price you pay for training the elite.

This is the general grounds which I might use to appeal to the city officials in Boston. There's more to explain, but the shape of the idea would be roughly this: much a cities often pay for programs to help ex-Marines transition to civilian life, on the principal that they represent valuable human capital that ought not to be allowed to self-destruct, it might pay off for the city to understand the peculiar predicament of graduates of MIT's intense DARPA projects, and provide them with help with the "transition to commercial life." There's something in it for them! Even though people who know nothing but how to think about the infrastructure of the next decade aren't generically commercially valuable, if you put them in proximity to normal business people, their perspective would rub off in a useful way. That's the way that Boston could have catalyzed an Internet industry of its own -- not by expecting MIT students to commercialize their work, which (with the possible exception of Philip) they were constitutionally incapable of, but by giving people who wanted to commercialize something but didn't know what a chance to learn from the accumulated (nearly ten years!) of experience and expertise of the Internet Crowd.

On that note, I wanted to say -- funny you should mention Facebook. You think of Mark Zuckerberg as the social networking visionary in Boston, and Boston could have won if they had paid to keep him. I think that strange -- Zuckerberg is fundamentally one of you, not one of us. It was right he should leave. But I'll ask you a question you've probably never thought about. Suppose the Internet had not broken into the public consciousness at the time it did; suppose the world had willfully ignored it for a few more years, so the transition from a DARPA-funded research project to a commercial proposition would have happened a few years later. There was an Internet Crowd at MIT constantly asking DARPA to let them build the "next thing," where "next" is defined as "what the market will discover it wants ten years from now." So if this crowd had gotten a few more years of government support, what would they have built?

I'm pretty sure it would have been a social networking infrastructure, not like Facebook, really, but more like the Diaspora proposal. I'm not sure, but I remember in '98/'99 that's what all the emotional energy was pointing toward. It wasn't technically possible to build yet, but the instant it was that's what people wanted. I think it strange that everyone is talking about social networking and how it should be designed now; it feels to me like deja vu all over again, and echo from a decade ago. If the city or state had picked up these people after DARPA dropped them, and given them just a little more time, a bit more government support -- say by a Mass ARPA -- they could have made Boston the home, not of the big social networking company, but of the open social networking infrastructure and and all the expertise and little industries such a thing would have thrown off. And it would have started years and years ago! That's how Boston could have become a leader by being itself better, rather than trying to be you badly.

Dan: I think you're perhaps overstating the impact of DARPA. DARPA, by and large, funds two kinds of university activities. First, it funds professors, which pays for post-docs, grad students, and sometimes full-time research staff. Second, DARPA also funds groups that have relatively little to do with academia, such as the BSD effort at Berkeley (although I don't know for a fact that they had DARPA money, they didn't do "publish or perish" academic research; they produced Berkeley Unix).

Undergrads at a place like MIT got an impressive immersion in computer science, with a rigor and verve that wasn't available most other places (although Berkeley basically cloned 6.001, and others did as well). They call it "drinking from a firehose" for a reason. MIT, Berkeley, and other big schools of the late 80's and early 90's had more CS students than they knew what to do with, so they cranked up the difficulty of the major and produced very strong students, while others left for easier pursuits.

The key inflection point is how popular culture at the university, and how the faculty, treat their "rock star" students. What are the expectations? At MIT, it's that you go to grad school, get a PhD, become a researcher. At Stanford, it's that you run off and get rich.

The decline in DARPA funding (or, more precisely, the micromanagement and short-term thinking) in recent years can perhaps be attributed to the leadership of Tony Tether. He's now gone, and the "new DARPA" is very much planning to come back in a big way. We'll see how it goes.

One last point: I don't buy the Army vs. Marines analogy. MIT vs. Stanford train students similarly, in terms of their preparation to go out and make money, and large numbers of MIT people are quite successfully out there making money. MIT had no lack of companies spin out of research there, notably including Akamai. The differences we're talking about here are not night vs. day, they're not Army vs. Marines. They're more subtle but still significant.

Rebecca: Yes, I've been hearing about the "unTethered Darpa." I should have mentioned that, but left it out to stay (vaguely) short. And yes, I am overstating to make it possible to make a simple statement of what I might be asking for that would be couched in terms a city or state government official might be able to relate to. Maybe that's irresponsible; that's why I'm testing it on you first, to give you a chance to yell at me and tell me if you think that's so.

They are casting about for a narrative of why Boston ceded its role as leaders of the Internet industry to SV, that would point them to something to do about it. So I was talking specifically about the sense in which Boston was once a leader in internet technology and the weaknesses that might have caused it to lose its lead. Paul Graham says that Boston has the weakness in developing industries that it is "too good" at other things, so I wanted to tell a dramatized story specifically about what the other things were and why that would lead to fatal weakness -- how being "too strong" in a particular way can also make you weak.

I certainly am overstating, but perhaps I am because I am trying to exert force against another prediliction I find pernicious: the tendency to be eternally vague about the internal emotional logic that makes things happen in the world. If people build a competent, cohesive, energetic community, and then it suddenly fizzles, fails to achieve its potential, and disbands, it might be important to know what weakness caused this surprising outcome so you know how to ask for the help that would keep it from happening the next time.

And to tell the truth, I'm not sure I entirely trust your objection. I've wondered why so often I hear such weak, vague narratives about the internal emotional logic that causes things to happen in the world. Vague narratives make you helpless to solve problems! I don't cling to the right to overstate things, but I do cling to the right to sleuth out the emotional logic of cause and effect that drives the world around me. I feel sometimes that I am fighting some force that wants to thwart me in that goal -- and I suspect that that force sometimes originates, not always in rationality, but in in a male tendency to not want to admit to weakness just for the sake of "seeming strong." A facade of strength can exact a high price in the currency of the real competence of the world, since often the most important action that actually makes the world better is the action of asking for help. I was really impressed with that Marine for being willing to admit to the price he paid, to the trauma he faced. That guy didn't need to fake strength! So maybe I am holding out the image of him as an example. We have government officials who are actively going out of their way to offer to help us; we have a community that accomplishes many of its greatest achievements because of government support; we shouldn't squander an opportunity to ask for what might help us. And this narrative might be wrong; that's why I'm testing it first. I'm open to criticism. But I don't want to pass by an opportunity, an opening to ask for help from someone who is offering it, merely because I'm too timid to say anything for the fear of overstatement.

Dan: Certainly, Boston's biggest strength is the huge number of universities in and around the area. Nowhere else in the country comes close. And, unsurprisingly, there are a large number of high-tech companies in and around Boston. Another MIT spin-out I forgot to mention above is iRobot, the Roomba people, which also does a variety of military robots.

To the extent that Boston "lost" the Internet revolution to Silicon Valley, consider the founding of Netscape. A few guys from Illinois and one from Kansas. They could well have gone anywhere. (Simplifying the story, but) they hooked up with a an angel investor (Jim Clark) and he draged them out to the valley where they promptly hired a bunch of ex-SGI talent and hit the road running. Could they have gone to Boston? Sure. But they didn't.

What seems to be happening is that different cities are developing their own specialties and that's where people go. Dallas, for example, has carved out a niche in telecom, and all the big players (Nortel, Alcatel, Cisco, etc.) do telecom work there. In Houston, needless to say, it's all about oilfield engineering. It's not that there's any particular Houston tax advantage or city/state funding that brings these companies here. Rather, the whole industry (or, at least the white collar part of it) is in Houston, and many of the big refineries are close nearby (but far enough away that you don't smell them).

Greater Boston, historically, was where the minicomputer companies were, notably DEC and Data General. Their whole world got nuked by workstations and PCs. DEC is now a vanishing part of HP and DG is now a vanishing part of EMC. The question is what sort of thing the greater Boston area will become a magnet for, in the future, and how you can use whatever leverage you've got to help make it happen. Certainly, there's no lack of smart talent graduating from Boston-area universities. The question is whether you can incentivize them to stay put.

I'd suggest that you could make headway, that way, by getting cheap office space in and around Cambridge (an "incubator") plus building a local pot of VC money. I don't think you can decide, in advance, what you want the city's specialty to be. You pretty much just have to hope that it evolves organically. And, once you see a trend emerging, you might want to take financial steps to reinforce it.

Thomas: BBN (which does DARPA funded research) has long been considered a halfway house between MIT and the real world.

Piaw: It looks like there's another conversation about this thread over at Hacker News: http://news.ycombinator.com/item?id=1416348 I love conversation fragmentation.

Doug: Conversation fragmentation can be annoying, but do you really want all those Hacker News readers posting on this thread?

Piaw: Why not? Then I don't have to track things in two places.

Ruchira: hga over at Hacker News says: "Self-selection by applicants is so strong (MIT survived for a dozen year without a professional as the Director), whatever gloss the Office is now putting on the Institute, it's able to change things only so much. E.g. MIT remains the a place where you don't graduate without taking (or placing out of) a year of the calculus and classical physics (taught at MIT speed), for all majors."

Well, the requirements for all majors at Caltech are: two years of calculus, two years of physics (including quantum physics), a year of chemistry, and a year of biology (the biology requirement was added after I went there); freshman chemistry lab and another introductory lab; and a total of four years of humanities and social sciences classes. The main incubator I know of near Caltech is the Idealab. Certainly JPL (the Jet Propulsion Laboratory) as well as Hollywood CGI and animation have drawn from the ranks of Caltech grads. The size of the Caltech freshman class is also much smaller than those at Stanford or MIT.

I don't know enough to gauge the relative success of Caltech grads at transitioning to local industry, versus Stanford or MIT, does anyone else?

Rebecca: The comments are teaching me what I didn't make clear, and this is one of the worst ones. When I talked about the "transition to the commercial world" I didn't mainly mean grads transitioning to industry. I was thinking more about the transition that a project goes through when it achieves product/market fit.

This might not be something that you think of as such a big deal, because when companies embark on projects, they usually start with a fairly specific plan of the market they mean to tackle and what they mean to do if and when the market does adopt their product. There is no difficult transition because they were planning for it all along. After all, that's the whole point of a company! But a ten year research project has no such plan. The web server enthusiast did not know when the market would adopt his "product" -- remember, browsers were still primitive then -- nor did he really know what it would look like when they did. Some projects are even longer term than that: a programming language professor said that the expected time from the conception of a new programming language idea to its widespread adoption is thirty years. That's a good chunk of a lifetime.

When you've spent a good bit of your life involved with something as a research project that no-one besides your small crowd cares about, when people do notice, when commercial opportunities show up, when money starts pouring out of the sky, its a huge shock! You haven't planned for it at all. Have you heard Philip's story of how he got his first contract for what became ArsDigita? I couldn't find the story exactly, but it was something like this: he had posted some of the code for his forum software online, and HP called him up and asked him to install and configure it for them. He said "No! I'm busy! Go away!" They said "we'll pay you $100,000." He's in shock: “You'll give me $100000 for 2 weeks of work?”

He wasn't exactly planning for money to start raining down out of the sky. When he started doing internet applications, he said, people had told him he was crazy, there was no future in it. I remember when I first started seeing URL's in ads on the side of buses, and I was just bowled over -- all the time my friends had been doing web stuff, I had never really believed they would ever be adopted. URL's are just so geeky, after all! I mean, seriously, if some wild-eyed nerd told you that in five years people would print "http://",on the side of a bus, what would you think? I paid attention to what they were doing because they thought it was cool, I thought it was cool, and the fact that I had no real faith anyone else ever would made no difference. So when the world actually did, it was entering a new world that none of us were prepared for, that nobody had planned for, that we had not given any thought to developing skills to be able to deal with. I guess this is a little hard to convey, because it wouldn't happen in a company. You wouldn't ever do something just because you thought it was cool, without any faith that anyone would ever agree with you, and then get completely caught by surprise, completely bowled over, when the rest of the world goes crazy about what you thought was your esoteric geeky obsession.

Piaw: I think we were all bowled over by how quickly people started exchanging e-mail addresses, and then web-sites, etc. I was stunned. But it took a really long time for real profits to show up! It took 20 or so search engine companies to start up and fail before someone succeeded!

Rebecca: Of course; you are bringing up what was in fact the big problem. The question was: in what mode is it reasonable to ask the local government for help? And if you are in the situation where $100,000 checks are raining on you out of the sky without you seeming to make the slightest effort to even solicit them, then it seems like only the biggest jerk on the planet would claim to the government that they were Needy and Deserving. Black babies without roofs on their heads are needy and deserving; rich white obnoxious nerds with money raining down on them are not. But remember though Philip doesn't seem to be expending much effort in his story, he also said in the late 90's that he had been building web apps for ten years. Who else on the planet in 1999 could show someone a ten year long resume of web app development?

As Piaw said, it isn't like picking up the potential wealth really was just a matter of holding out your hand as money rained from the sky. Quite the contrary. It wasn't easy; in fact it was singularly difficult. Sure, Philip talked like it was easy, until you think about how hard it would have been to amass the resume he had in 1999.

When the local government talks about how it wants to attract innovators to Boston, to turn the city into a Hub of Innovation, my knee-jerk reaction is -- and what are we, chopped liver? But then I realize that when they say they want to attract innovators, what they really mean is not that they want innovators, but that they want people who can innovate for a reasonable, manageable amount of time, preferably short, and then turn around, quick as quicksilver, and scoop up all the return on investment in that innovation before anyone else can get at it -- and give a big cut in taxes to the city and state! Those are the kind of innovators who are attractive! Those are the kind who properly make your Boston the kind of Hub of Innovation the Mayor of Boston wants it to be. Innovators like those in Tech Square or Stata, not so much. We definitely qualify for the Chopped Liver department.

And this hurts. It hurts to think that the Mayor of Boston might be treating us with more respect now if we had been better in ~2000 at turning around, quick as quicksilver, and remaking ourselves into people who could scoop up all, or some, or even a tiny fraction of the return on investment of the innovation at which we were then, in a technical sense, well ahead of anyone else. But remaking yourself is not easy! Especially when you realize that the state from which we were remaking ourselves was sort of like the Marines -- a somewhat ascetic state, one that gave you the nerd equivalent of military rations, a tent, maybe a shower every two weeks, and no training in any immediately salable skills whatsoever -- but also one that also gave you a community, an identity, a purpose, a sense of who you were that you never expected to change. But all of a sudden we "won," and all of a sudden there was a tremendous pressure to change. It was like being thrown in the deep end of the pool without swim lessons, and yes we sank, we sank like a stone with barely a dog paddle before making a beeline for the bottom. So we get no respect now. But was this a reasonable thing to expect? What does the mayor of Boston really want? Yes, the sense in which Boston is a Hub of Innovation (for it already is one, it is silly for it to try to become what it already is!) is problematic and not exactly what a Mayor would wish for. I understand his frustration. But I think he would do better to work with his city for what it is, in all its problematic incompetence and glory, than to try to remake it in the image of something else it is not.

Rebecca: On the subject of Problematic Innovators, I was thinking back to the scene in the computer lab where everyone agreed that hoarding domain names was the dumbest idea they had ever heard of. I'm arguing that scooping up return on the investment in innovation was hard, but registering a domain name is the easiest thing in the world. I think they were free back then, even. If I remember right, they started out free, and then Procter & Gamble registered en-mass every name that had even the vaguest entomological relation with the idea of "soap," at which point the administrators of the system said "Oops!" and instituted registration fees to discourage that kind of behavior -- which, of course, would have done little to deter P&G. They really do want to utterly own the concept of soap. (I find it amusing that P&G was the first at bat in the domain name scramble -- they are not exactly the world's image of a cutting-edge tech-savvy company -- but when it comes to the problem of marketing soap, they quietly dominate.)

How can I can explain that we were not able to expend even the utterly minimal effort in capturing the return on investment in innovation of registering a free domain name, so as to keep the resulting tax revenues in Massachusetts?

Thinking back on it, I don't think it was either incapacity, or lack of foresight, or a will to fail in our duty as Boston and Massachusetts taxpayers. It was something else: it was almost a "semper fidelis"-like group spirit that made it seem dishonorable to hoard a domain name that someone else might want, just to profit from it later. Now one might ask, why should you refrain from hoarding it sooner just so that someone else could grab it and hoard it later? That kind of honor doesn't accomplish anything for anyone!

But you have to realize, this was right at the beginning, when the domain name system was brand new and it wasn't at all clear it would be adopted. These were the people who were trying to convince the world to accept this system they had designed and whose adoption they fervently desired. In that situation, honor did make a difference. It wouldn't look good to ask the world to accept a naming system with all the good names already taken. You actually noticed back then when something (like "soap") got taken -- the question wasn't what was available, the question was what was taken, and by whom. You'd think it wouldn't hurt too much to take one cool name: recently I heard that someone got a $38 million offer for "cool.com." That's a lot of money! -- would it have hurt that much to offer the world a system with all the names available except, you know, one cool one? But there was a group spirit that was quite worried that once you started down that slope, who knew where it would lead?

There were other aspects of infrastructure, deeper down, harder to talk about, where this group ethos was even more critical. You can game an infrastructure to make it easier to personally profit from it -- but it hurts the infrastructure itself to do that. So there was a vehement group discipline that maintained a will to fight any such urge to diminish the value of the infrastructure for individual profit.

This partly explains why we were not able, when the time came, to turn around, quick as quicksilver, and scoop up the big profits. To do that would have meant changing, not only what we were good at, but what we thought was right.

When I think back, I wonder, why people weren't more scared? When we chose not to register "cool.com" or similar names, why didn't we think, life is hard, the future is uncertain, and money does really make a difference in what you can do? I think this group ethic was only possible because there was a certain confidence -- the group felt itself party to a deal: in return for being who we are, the government would take care of us, forever. Not until the time when the product achieved sufficient product/market fit that it became appropriate to expect return on investment. Forever.

This story might give a different perspective on why it hurts when the Mayor of Boston announces that he wants to make the city a Hub of Innovation. The innovators he already has are chopped liver? Well, its understandable that he isn't too pleased with the innovators in this story, because they aren't exactly a tax base. But that is the diametric opposition of the deal with the government we thought we had.

show more
Are closed social networks inevitable?
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2010-01-01 00:00:00 | Created: 2026-07-23 05:18:40

This is an archive of an old Google Buzz conversation (circa 2010?) on a variety of topics, including whether or not it's inevetible that a closed platform will dominate social.

Piaw: Social networks will be dominated primarily by network effects. That means in the long run, Facebook will dominate all the others.

Rebecca: ... which is also why no one company should dominate it. "The Social Graph" and its associated apps should be like the internet, distributed and not confined to one company's servers. I wish the narrative surrounding this battle was centered around this idea, and not around the whole Silicon Valley "who is the most genius innovator" self-aggrandizing unreality field. Thank god Tim Berners Lee wasn't from Silicon Valley, or we wouldn't have the Internet the way we know it in the first place.

I suppose I shouldn't be being so snarky, revealing how much I hate your narratives sometimes. But I think for once this isn't, as it usually is, merely harmlessly cute and endearing - you all collectively screwing up something actually important, and I'm annoyed.

Piaw: The way network effects work, one company will control it. It's inevitable.

Rebecca: No it is not inevitable! What is inevitable is either that one company controls it or that no company controls it. If you guys had been writing the narrative of the invention of the internet you would have been arguing that it was inevitable that the entire internet live on one companies servers, brokered by hidden proprietary protocols. And obviously that's just nuts.

Piaw: I see, the social graph would be collectively owned. That makes sense, but I don't see why Facebook would have an incentive to make that happen.

Rebecca: Of course not! That's why I'm biting my fingernails hoping for some other company to be the white knight and ride up and save the day, liberating the social graph (or more precisely, the APIs of the apps that live on top of them) from any hope of control by a single company. Of course, there isn't a huge incentive for any other company to do it either --- the other companies are likely to just gaze enviously at Facebook and wish they got there first. Tim Berners Lee may have done great stuff for the world, but he didn't get rich or return massive value to shareholders, so the narrative of the value he created isn't included in the standard corporate hype machine or incentives.

Google is the only company with the right position, somewhat appropriate incentives, and possibly the right internal culture to be the "Tim Berners Lee" of the new social internet. That's what I was hoping for, and I'm am more than a bit bummed they don't seem to be stepping up to the plate in an effective way in this case, especially since they are doing such a fabulous job in an analogous role with Android.

Rebecca: There is a worldview lurking behind these comments, which perhaps I should try to explain. I'm been nervous about this because it contains some strange ideas, but I'm wondering what you think.

Here's a very strange assertion: Mark Zuckerberg is not a capitalist, and therefore should not be judged by capitalist logic. Before you dismiss me as nuts, stop and think for a minute. What is the essential property that makes someone a capitalist?

For instance, when Nike goes to Indonesia and sets up sweatshops there, and if communists, unhappy with low pay & terrible conditions, threaten to rebel, they are told "this is capitalism, and however noxious it is, capitalism will make us rich, so shut up and hold your peace!" What are they really saying? Nike brings poor people sewing machines and distribution networks so they can make and sell things they could not make and sell otherwise, so they become more productive and therefore richer. The productive capacity is scarce, and Nike is bringing to Indonesia a piece of this scarce resource (and taking it away from other people, like American workers.) So Indonesia gets richer, even if sweatshop workers suffer for a while.

So is Mark Zuckerberg bringing to American workers a piece of scarce productive capacity, and therefore making American workers richer? It is true he is creating for people productive capacity they did not have before --- the possibility of writing social apps, like social games. This is "innovation" and it does make us richer.

But it is not wealth that follows the rules of capitalist logic. In particular, this kind of wealth of productive capacity, unlike the wealth created by setting up sewing machines, does not have the kind of inherent scarcity that fundamentally drives capitalist logic. Nike can set up its sewing machines for Americans, or Indonesians, but not for everyone at once. But Tim Berners Lee is not forced to make such choices -- he can design protocols that allow everyone everywhere to produce new things, and he need not restrict how they choose to do it.

But -- here's the key point -- though there is no natural scarcity, there may well be "artificial" scarcity. Microsoft can obfuscate Windows API's, and bind IE to Windows. Facebook can close the social graph, and force all apps to live on its servers. "Capitalists" like these can then extract rents from this artificial scarcity. They can use the emotional appeal of capitalist rhetoric to justify their rent-seeking. People are very conditioned to believe that when local companies get rich American workers in general will also get rich -- it works for Indonesia so why won't it work for us? And Facebook and Microsoft employees are getting richer. QED.

But they aren't getting richer in the same way that sweatshop employees are getting richer. The sweatshop employees are becoming more productive than they would otherwise be, in spite of the noxious behavior of the capitalists. But if Zuckerberg or Gates behaves noxiously, by creating a walled garden, this may make his employees richer, in the sense of giving them more money, but "more money" is not the same as "more wealth." More wealth means more productive capacity for workers, not more payout to individual employees. In a manufacturing economy those are linked, so people forget they are not the same.

And in fact, shenanigans like these reduce rather than increase the productive capacity available to everyone, by creating an artificial scarcity of a kind of productive tool that need not be scarce at all, just for the purpose of extracting rents from them. No real wealth comes from this extraction. In aggregate it makes us poorer rather than richer.

Here's where the kind of stunt that Google pulled with Android, that broke the iPhone's lock, even if it made Google no money, should be seen as the real generator of wealth, even if it is unclear whether it made any money for Google's shareholders. Wealth means I can build something I couldn't build before -- if I want I can install a Scheme or Haskell interpreter on an Android phone, which I am forbidden to put on the iPhone. It means a lot to me! Google's support of Firefox & Chrome, which sped the adoption of open web standards and HTML5, also meant a huge amount to me. I'm an American worker, and I am made richer by Google, in the sense of having more productive capacity available to me, even if Google wasn't that great for my personal wealth in the sense of directly increasing my salary.

Rebecca: (That idea turned out to be sortof shockingly long comment by itself, and on review the last two paragraphs of the original comment were a slightly different thought, so I'm breaking them into a different section.)

I'm upset that Google is getting a lot of anti-trust type flak, when I think the framework of anti-trust is just the wrong way to think. This battle isn't analogous to Roosevelt's big trust busting battles; it is much more like the much earlier battles at the beginning of the industrial revolution of the Yankee merchants against the old agricultural, aristocratic interests, which would have squelched industrialization. And Google is the company that has been most consistently on the side of really creating wealth, by not artificially limiting the productivity they make available for developers everywhere. Other companies, like Microsoft or Facebook, though they are "new economy," though they are "innovative," though they seem to generate a lot of "wealth" in the form of lots of money, really are much more like the old aristocrats rather than the scrappy new Yankees. In many ways they are slowing down the real revolution, not speeding it up.

I've been reluctant to talk too much about these ideas, because I'm anxious about being called a raving commie. But I'm upset that Google is the target of misguided anti-trust logic, and it might be sufficiently weakened that it can't continue to be the bulwark of defense against the real "new economy" abuses that it has been for the last half-decade. That defense has meant a lot to independent developers, and I would hate to see it go away.

Phil: +100, Rebecca. It is striking how little traction the rhetoric of software freedom has here in Silicon Valley relative to pretty much everywhere else in the world.

Rebecca: Thanks - I worry whether my ultra-long comments are spam and its good to hear if someone appreciates them. I have difficulty making my ideas short, but I'm using this Buzz conversation to practice.

I'm am not entirely happy with the way the "software freedom" crowd is pitching their message. I had an office down the hall from Richard Stallman for a while, and I was often harangued by him. However, I thought his message was too narrow and radicalized. But on the other hand, when I thought about it hard, I also realized that in many ways it was not radical enough...

Why are we talking about freedom? To motivate this, I sometimes tell a little story about the past. When I was young my father read to me "20,000 Leagues Under the Sea," advertising it as a great classic of futuristic science fiction. Unfortunately, I was unimpressed. It didn't seem "futuristic" at all: it seemed like an archaic fantasy. Why? Certainly it was impressive that an author in 1869 correctly predicted that people would ride in submarines under the sea. But it didn't seem like an image of the future, or even the past, at all. Why not? Because the person riding around on the submarine under the sea was a Victorian gentleman surrounded by appropriately deferential Victorian servants.

Futurists consistently get their stories wrong in a particular way: when they say that technology changes the world, they tell stories of fabulous gadgets that will enable people to do new and exciting things. They completely miss that this is not really what "change" -- serious, massive, wrenching, social change - really is. When technology truly enables dreams of change, it doesn't mean it enables aristocrats to dream about riding around under the sea. What it means is that enables the aristocrat's butler to dream of not being a butler any more --- a dream of freedom not through violence or revolution, but through economic independence. A dream of technological change -- really significant technological change -- is not a dream of spiffy gadgets, it is a dream of freedom, of social & economic liberation enabled by technology.

Lets go back to our Indonesian sweatshop worker. Even though in many ways the sweatshop job liberates her --- from backbreaking work on a farm, a garbage dump, or in brothels -- she is also quite enslaved. Why? She can sew, let us say, high-end basketball sneakers, which Nike sells for $150 apiece -- many, many times her monthly or even yearly wage. Why is she getting a small cut of the profit from her labors? Because she is dependent on the productive capacity that Nike is providing to her, so bad as the deal is, it is the best she can get.

This is where new technology comes in. People talk about the information revolution as if it is about computers, or software, but I would really say it is about society figuring out (slowly) how to automate organization. We have learned to effectively automate manufacturing, but not all of the work of a modern economy is manufacturing. What is the service Nike provides that makes this woman dependent on such a painful deal? Some part of this service is the manufacturing capacity they provide -- the sewing machine -- but sewing machines are hardly expensive or unobtainable, even for poor people. The much bigger deal is the organizational services Nike offers: all the branding, logistics, supply-chain management and retail services that go into getting a sneaker sewn in Indonesia into the hands of an eager consumer in America. One might argue that Nike is performing these services inefficiently, so even if our seamstress is effective and efficient, Nike must take an unreasonably large cut of the profits from the sale of the sneaker to support the rest of this inefficient, expensive, completely un-automated effort.

That's where technological change comes in. Slowly, it is making it possible for all these organizational services to be made more automated, streamlined and efficient. This is really the business Google is in. It is said that Google is an "advertising" business, but to call what Google does "advertising" is to paper over the true story of the profound economic shift of which they are merely providing the opening volley.

Consider the maker of custom conference tables who recently blogged in the New York Times about Adwords (http://boss.blogs.nytimes.com/2010/12/03/adwords-and-me-exploring-the-mystery/). He said he paid Google $75,124.77 last year. What does that money represent -- what need is Google filling which is worth more than seventy thousand a year to this guy? You might say that they are capturing an advertising budget of a company, until you consider that without Google this company wouldn't exist at all. Before Google, did you regularly stumble across small businesses making custom conference tables? This is a new phenomenon! The right way to see it is that this seventy thousand isn't really a normal advertising budget -- instead, think of it as a chunk of the revenue of the generic conference table manufacturer that this person no longer has to work for. Because Google is providing for him the branding, customer management services, etc, etc that this old company used to be doing much less efficiently and creatively, this blogger has a chance to go into business for himself. He is paying Google seventy thousand a year for this privilege, but this is probably much less than the cut that was skimmed off the profits of his labors by his old management (not to mention issues of control and stifled creativity he is escaping). Google isn't selling "advertising": Google is selling freedom. Google is selling to the workers of the world the chance to rid themselves of their chains -- nonviolently and without any revolutionary rhetoric -- but even without the rhetoric this service is still about economic liberation and social change.

I feel strange when I hear Eric Schmidt talk about Google's plans for the future of their advertising business, because he seems to be telling Wall Street of a grand future where Google will capture a significant portion of Nike's advertising budget (with display ads and such). This seems like both an overambitious fantasy; and also strangely not nearly ambitious enough. For I think the real development of Google's business -- not today, not tomorrow, not next year, not even next decade, but eventually and inexorably (assuming Google survives the vicissitudes of fate and cultural decay) -- isn't that Google captures Nike's advertising budget. It is that Google captures a significant portion of Nike's entire revenue, paid to them by the workers who no longer have to work for Nike anymore, because Google or a successor in Google's tradition provides them with a much more efficient and flexible alternative vendor for the services Nike's management currently provides.

Rebecca: (Once again I looked at my comment and realized it was even more horrifyingly long. My thoughts seem short in my head, but even when I try to write them down as fast and effectively as I can, they aren't short anymore! Again, I saw the comment has two parts: first, explaining the basic idea of the "freedom" we are talking about, and second, tying it back into the context of our original discussion. So to try to be vaguely reasonable I am cutting it in two.)

I suppose Eric Schmidt will never stand in front of Wall Street and say that. When it is really true that "We will bury you!" nobody ever stands up and bangs a shoe on the table while saying it. The architects of the old "new economy" didn't say such things either: the Yankee merchants never said to their aristocratic political rivals that they intended to eventually completely dismantle their social order. In 1780 there was no discussion that foretold the destructive violence of Sherman's march to the sea. I'm not sure they knew it themselves, and if they had been told that that was a possible result of their labors they might not have wanted to hear it. The new class wanted change, they wanted opportunity, they wanted freedom, but they did not want blood! That they would be cornered into seeking it anyway would have been as horrifying to them as to anyone else. Real change is not something anyone wants to crow about --- it is too terrifying.

But it is nonetheless important to face, because in the short term this transformation is hardly inevitable or necessarily smooth. If our equivalent of Sherman's march to the sea might be in our future, we might want to think about how to manage or avoid it before it is too late.

One major difficulty, as I explained in the last comment, is that while the "automation of information," if developed properly, has the potential to break fundamental laws of the scarcity of productive capacity, and thereby free "the workers of the world", nonetheless that potential can be captured, and turned into "artificial" scarcity, which doesn't set workers free, it further enslaves them. There is also a big incentive to do this, because it is the easiest way to make massive amounts of money quickly for a person in the right place at the right time.

I see Microsoft as a company that has made a definite choice of corporate strategy to make money on "artificial scarcity." I see Google as a company that has made a similar definite choice to make money "selling freedom", specifically avoiding the tricks that create artificial scarcity, even when it doesn't help or even hurts their immediate business prospects.

And Facebook? Where is Sheryl Sandburg (apparently the architect of business development at Facebook) on this crucial question? A hundred years from now, when all your "genius" and "innovation," all the gadgets you build that so delight aristocrats, and are so celebrated by "futurists", will be all but forgotten, the choices you make on this question will be remembered. This matters.

Ms. Sandburg seems to be similarly clear on her philosophy: she simply wants as much diversity of revenue streams for Facebook as she can possibly get. It is hard to imagine an more un-ideological antithesis of Richard Stallman. Freedom or scarcity, she doesn't care: if its a way to make money, she wants it. As many different ones as possible! She wants it all! Its hard for me to get too worked up about this, especially since for other reasons I am rooting for Ms. Sandburg's success. Even so, I would prefer if it were Google in control of this technological advance, because Google's preference on this question is so much more clear and unequivocal.

I don't care who is the "genius innovator" and who is the "big loser", whether this or that company has taken up the mantle of progress or not, who is going to get rich, which company will attract all the superstars, or all the other questions that seem to you such critical matters, but I do care that your work makes progress towards realizing the potential of your technology to empower the workers of the world, rather than slowing it down or blocking it. Since Google has made clear the most unequivocal preference in the right direction on this question, that means I want Google to win. This is too important to be trusted to the control of someone ambivalent about their values, no matter how much basic sympathy I have for the pragmatic attitude. Baris Baser: +100! Liberate the social graph! I wish I could share the narrative taking place here on my buzz post, but I'll just plug it in.

Rob: Google SO dropped the ball with Orkut - they let Facebook run off with the Crown Jewels. Helder Suzuki: I believe that Facebook's dominance will eventually be challenged just like credit card companies are being today. But I think it's gonna come much quickier for Facebook.

There are lots of differences, but I like this comparison because credit card companies used great network effect to dominate and shield the market from competition. If you look at them (visa, amex, mastercard), all they have today is basically brand. Today we just know that "credit card" payment (and the margins) will be so much different in the near future.

Likewise I don't think that social graph will protect Facebook's "market" in the long run. Just like today it's incredibly easier to set a POS network compared to a few years ago, social graph is gonna be something trivial in the years to come.

Rebecca: Yay! People are reading my obscenely long and intellectual comments. Thanks guys!

Piaw: I disagree with Helder, even though I agree with Rebecca that it's better for Google to own the social graph. The magic trick that Facebook pulled off was getting the typical user to provide and upload all his/her personal information. It's incredibly hard to do that: Amazon couldn't do it, and neither could Google. I don't think it's one of those things that's technically difficult, but the social engineering required to do that demands critical mass. That's why I think that Facebook is (still) under-valued.

Rob: @Piaw - it was an accident of history I think. When Facebook started, they required a student ID to join. This made a culture of "real names" that stuck, and that no one else has been able to replicate.

Piaw: @Rob: The accident of history that's difficult to replicate is what makes Facebook such a good authentication mechanism. I would be willing to not moderate my blog, for instance, if I could make all commenters disclose their true identity. The lowest qualify arguments I've seen on Quora, for instance, were those in which one party was anonymous. Elliotte Rusty Harold: This is annoying I want to reshare Rebecca's comments. not the original post, but I can't seem to do that. :-)

Rebecca: In another conversation, someone linked a particular point in a Buzz commentary to Hacker News (http://news.ycombinator.com/item?id=1416348). I'm not sure how they did it. It was a little strange, though, because then people saw it out of context. These comments were tailored for a context.

Where do you want to share it? I'm not sure I'm ready to deal with too big an audience; there is a purpose to practicing writing and getting reactions in an obscure corner of the internet. After all, I am saying things that might be offensive or objectionable in Silicon Valley, and are, in any case, awfully forward -- it is useful to me to talk to a select group of my friends to get feedback from them on how well it does or doesn't fly. Its not like I mind being public, but I also don't mind obscurity for now.

Rebecca: Speaking of which, Piaw, I was biting my fingernails a little wondering how you would react to my way of talking about "software freedom." I've sort of thought of becoming a software freedom advocate in the tradition of Stallman or ESR, but more intellectual, with more historical perspective, and (hopefully) with less of an edge of polemical insanity. However, adding in an intellectual and historical perspective also added in the difficulty of colliding with real intellectuals and historians, which makes the whole thing fraught, so for that reason among others I've been dragging my feet.

This discussion made me dredge up this whole project, partly because I really wanted to know your reactions to it. However, you only reacted to the Facebook comments, not the more general software freedom polemic. What did you think about that?

Piaw: I mostly didn't react to the free software polemic because I agree with what you're saying. I agree that something like Adwords and Google makes possible businesses that didn't exist before. Facebook, for instance, recently showed me an ad for a Suzanne Vega concert that I definitely would not have known about but would have wanted to go if not for a schedule conflict. I want to be able to "like" that ad so that I can get Facebook to show me more ads like those!

Do I agree that the world would be a better place for Facebook's social graph to be an open system? Yes and No. In the sense of Facebook having less control, I think it would be a good thing. But do I think I want anybody to have access to it? No. People are already trained to click "OK" to whatever data access any applet wants in Facebook, and I don't need to be inundated with spam in Facebook --- one of the big reasons Facebook has so much less spam is because my friends are more embarrassed about spamming me than the average marketing person, and when they do spam me it's usually with something that I'm interested in, which makes it not spam.

But yes, I do wish my Buzz comments (and yours too) all propagated to Facebook/Friendfeed/etc. and the world was one big open community with trusted/authenticated users and it would be all spam free (or at least, I get to block anonymous commenters who are unauthenticated). Am I holding my breath for that? No.

I am grateful that Facebook has made a part of the internet (albeit a walled garden part) fully authenticated and therefore much more useful. I think most people don't understand how important that is, and how powerful that is, and that this bit is what makes Facebook worth whatever valuation Wall Street puts on it.

Baris: Piaw, a more fundamental question lurks within this discussion. Ultimately, will people gravitate toward others with similar interests and wait for resources to grow there (Facebook,) or go where the resources are mature, healthy, and growing fast, and wait for everyone else to arrive (Google?)

Will people ultimately go to Google where amazing technology currently exists and will probably magnify, given the current trend (self driving cars, facial recognition, voice recognition, realtime language translation, impeccable geographic data, mobile OS with a bright future, unparalleled parallel computing, etc..) or join their friends first at the current largest social network, Facebook, and wait for the technology to arrive there?

A hypothetical way of looking at this: Will people move to a very big city and wait for it to be an amazing city, or move to an already amazing city and wait for everyone else to follow suit? Or are people ok with a bunch of amazing small cities?

Piaw: Baris, I don't think you've got the analogy fully correct. The proper analogy is this: Would you prefer to live in a small neighborhood where you sometimes have to go a long way to get what you want/are interested in but is relatively crime free, or would you like to live in a big city where it's easy to get what you want but you get tons of spam and occasionally someone comes in and breaks into your house?

The world obviously has both types of people, which is why suburbs and big cities both exist.

Baris: "tons of spam and occasionally someone comes in and breaks into your house?" I think this is a bit too draconian/general though... going with this analogy, I think becomes a bit more subjective, i.e. really depends on who you are in that city, where you live, what you own, how carefree you live your life, and so forth.

Piaw: Right. And Facebook has successfully designed a web-site around this ego-centricity. You can be the star of your tiny town by selectively picking your friends, or you can be the hub of a giant city and accept everyone as a friend. If the latter, then you gave up your privacy when your "friend" posts compromising pictures of you that gets you in trouble with your employer.

Nick: Google is the only company with the right position, somewhat appropriate incentives, and possibly the right internal culture to be the "Tim Berners Lee" of the new social internet.

I'd agree that Google hasn't done well at social, but surely are better than that!

Rebecca: Oh, you aren't impressed with Tim Berner Lee's work? Was it the original HTML standard you didn't like, or the more recent W3C stuff? I would admit there is stuff to complain about about both of them.

Nick: It seems to me that TBL got lucky. His original work on the WWW was good, but I think it is difficult to argue he was responsible for its success - certainly no more than someone like Marc Andreessen, who has a pattern of success that repeated after his initial success with Mosaic.

Rebecca: @Piaw (a little ways back) So you found my free software polemic so unobjectionable as to be barely worth comment? Wasn't it a little intellectually radical, with all that "not a capitalist" and "change in the nature of scarcity" stuff? When I told Lessig parts of basic story (not in Google context, because it was many years ago), and asked him for advice about how to talk to economists, he warned me that the words I was using contain so many warning bells of crackpot intellectual radicalism that economists would immediately write me off for using them without any further consideration.

It never ceases to amaze me how engineers will swallow shockingly strange ideas without a peep. I suppose in the company of Stallman and ESR, I am a raging intellectual conservative and pragmatist, and since engineers have accepted their style as at least a valid way to have a discussion (even if they don't agree with their politics), I seem tame by comparison. Of course talking to historians or economists is a different matter, because they don't already accept that this is a valid way to have a discussion.

Actually, it is immensely useful to me to have this discussion thread to us to show people who might think I'm a crackpot, because it is evidence for the claim that in my own world nobody bats an eyelash at this way of talking.

Incidentally, I started thinking about this subject because of Krugman. In the late nineties I was a rabid Krugman fan in a style that is now popular -- "Krugman is always right" -- but was a bit strange back then when he was just another MIT economics professor hardly anyone had ever heard of. However, when he talked about technology (http://pkarchive.org/column/61100.html), I thought he was wrong, which upset me terribly because I also was almost religiously convinced he was always right. In another essay (http://pkarchive.org/personal/howiwork.html) he said it was very important to him to "Listen to the Gentiles" i.e "Pay attention to what intelligent people are saying, even if they do not have your customs or speak your analytical language." But he also said "I have no sympathy for those people who criticize the unrealistic simplifications of model-builders, and imagine that they achieve greater sophistication by avoiding stating their assumptions clearly." So it seemed clear to me that he would be willing to hear me explain why he was wrong, as long as I would be willing to state my assumption clearly.

Before I knew exactly what I was intending to say, my plan had been to figure out my assumptions well enough to meet his standards, and then ask him to help me do the rest of the work to cast it all into a real economic model. Back then he was just an MIT professor I'd taken a class from, not a famous NYTimes columnist, Nobel-prize winning celebratory, so this plan seemed natural. Profs at MIT don't object if their students point out mistakes, as long as the students are responsible about it. It took me a while to struggle through the process of figuring out what my assumptions were (assumptions? I have assumptions?). When I did I was somewhat horrified to realize that following through with my plan meant accosting him to demand he write a new Wealth of Nations for me! (He'd also left for Princeton by then and started to become famous, so my plan was logistically more difficult than I'd planned.) I had not originally realized what it was that I would be asking for, or that the whole thing would be so daunting.

I asked Lessig for advice what to do (Lessig being the only person I knew who lived in both worlds) and Lessig read me the riot act about the rules of intellectual respectability. So it seemed it would be up to me to write the new Wealth of Nations, or at least enough of it to prove the respectability of the ideas contained therein. I was trying to be a computer science student, not an economist, so that degree of effort hardly fit into my plans. I tried to ask for help at the Lab for Computer Science (now CSAIL) by giving a talk in a Dangerous Ideas seminar series, but of the professors I talked to, only David Clark was sympathetic about the need for such a thing. However, he also said very clearly that resources to support grad students to work with economists were limited and really confined to only the kind of very specific net-neutrality stuff he was pushing in concert with his protocol work, not the kind of general worldview I was thinking about. So I was amazed to find that this kind of thing falls into the cracks between different parts of academic culture.

I'm still not sure what to do, but I am more and more inclined to ignore Lessig's (implicit) advice to be apologetic and defensive about my lack of intellectual respectability. That would entail a degree of effort I can't afford, since I am still focused on proving myself as a computer scientist, not an intellectual in the humanities. (Having this discussion thread to point to is quite useful on that score.) I could just drop it (I did for a while), but I'm getting more and more upset that technology is moving much faster than the intellectual and social progress that is required to handle it. People seem to think that powerful technology is a good thing in itself, but that is not true: it is only technology in the presence of strong institutions to control its power that provide net benefits to society -- without such controls it can be fantastically destructive. From that point of view a "new economy" is not good news -- what "new" means is that all the old institutions are now out of date and their controls are no longer working. And academic culture is culturally dislocated in ways that ensure that no one anywhere is really dealing with this problem. Pretty picture, isn't it?

Nick: @Rebecca: I don't understand your argument. Why is Google selling advertising anymore about freedom than Facebook selling advertising?

It's true that Facebook doesn't make their social graph and/or demographic data available to third parties, but Google doesn't make a person's search history available to third parties either. Why is one so much worse than the other?

Piaw: Rebecca I think that having more data be more open is ideal. However, but I view it as a purely academic discussion for the same reason I view writing "Independent Cycle Touring" in TeX to be an academic discussion. Sure it could happen, but the likelihood of it happening is so slim to none that I don't find the discussion to be of interest.

Now, I do agree that technology and its adoption does grow faster than our wisdom and controls for them. However, I don't think that information technology is the big offender. Humanity's big long term problems has more to do with fossil fuels as an energy source, and that's pretty darn old technology. You can fix all the privacy problems in the world, but if we get a runaway greenhouse planet by 2100 it is all moot. Because of that you don't find me getting worked up about privacy or the open-ness of Facebook's social graph. If Facebook does become personally objectionable to me, then I will close my account. Otherwise, I will keep leveraging the work their engineers do.

Elliotte: Rebecca, going back and rereading your comments I'm not sure your analysis is right, but I'm not sure it's wrong either. Of course, I am not an economist. From my non-economist perspective it seems worth further thought, but I also suspect that economists have already thought much of this. The first thing I'd do is chat up a few economists and see if they say something like, "Oh, that's Devereaux's Theory of Productive Capacity" or some such thing.

I guess I didn't see anything particularly radical and certainly nothing objectionable in what you wrote. You're certainly not the first to notice that software economics has a different relationship to scarcity than physical goods.Nor would I see that as incompatible with capitalism. It's only really incompatible with a particular religious view of capitalism that economists connected to the real world don't believe in anyway. The theological ideologues of the Austrian School and the nattering nabobs of television news will call you a commie (or more likely these days a socialist) but you can ignore them. Their claimed convictions are really just a bad parody of economics that bares only the slightest resemblance to the real world.

You hear a lot from these fantasy world theorists because they have been well funded over the last 40 years or so by corporations and the extremely wealthy with the explicit goal of justifying wealth. Academically this is most notable at the University of Chicago, and it's even more obvious in the pseudo-economics spouted on television news. At the extreme, these paid hucksters expouse the laissez-faire theological conviction that markets are perfectly efficient and rational and that therefore whatever the markets do must be correct; but the latest economic crises have caused most folks to realize that this emperor has no clothes. Economists doing science and not theology pay no attention to this priesthood. I wish the same could be said for the popular media.

Helder: I don't think I agree with the scarcity point that Rebecca made.

Generally, if a company is making money from something it's because their are producing some kind of wealth, otherwise they won't sustain economically. It doesn't have to be productive wealth like in factories, it could be cultural (e.g. a TV Show), or something else.

Even if you think of artificial scarcity, that's only possible for a company to make when they already have a big momentum (e.g. windows or facebook dominance). Artificial scarcity sucks when you look just at it, but it's more like a "local" optimzation based on an already dominant market position.

Perhaps Facebook, Microsoft and other co. wouldn't thrive in the first place if they weren't "allowed" to make the most of their closed system. The world is a better place with a closed Facebook and proprietary Windows API than no with no Facebook or Windows at all.

TV producers try to do their best to create the right scarcity when releasing their shows and movies to maximize profit. If they were to adopt some kind of free and open philosophy that they would release their content for download on day 1, they would simply go broke and destroy wealth in the long run.

Rebecca: Thanks guys, for the great comments! I appreciate the opportunity to answer these objections, because this is a subtle issue and I can certainly see that the reasoning behind my position is far from obvious. I won't be able to do it today because I need to be out all day, and its probably just as well that I have a little time to think of how to make the reply as clear and short as possible.

Rebecca: OK, I have about four different kinds of objections to answer, and I don't want to keep this as short as I can, so I think I will arrange it carefully so I can use the ideas in one answer to help me explain the next one. That means I'll answer in the order: Elliot, Piaw then Nick & Helder.

It actually took me much of a week to write and edit an answer I liked and believed was condensed as I could make it. And despite my efforts it is still quite long. However, your reaction to my first version has impressed on me that there are some key points I need to take the space to clarify:

  1. I shouldn't have tried to talk about a system that "isn't capitalism" in too short an essay, because that is just too easily misunderstood. I take a good bit of space in the arguments below matching Elliots' disavowal of the religious view of capitalism with an explicit disavowal of the religious view of the end of capitalism.

  2. Piaw also asked a good question "why is this important?" It isn't obvious; its only something you can see once it sinks into you how dramatically decades of exponential technological growth can change the world. Since this subject is pretty crazy-making and hard to see with any perspective, I try to use an image from the past to help us predict how people from the future will see us differently than we see ourselves. I want to impress on you why future generations are likely to make very different judgements about what is and isn't important.

  3. Finally, I said rather casually that I wanted to talk about software freedom in the standard tradition, only with more intellectual and historical perpective. As I write this, though, I'm realizing the historical perspective actually changes the substance of the position, in a way I need to make clear.

And last of all I wanted to step back again and put this all in the context of what I am trying to accomplish in general, with some commentary on your reactions to the assertion that I am being intellectually radical.

These replies are split into sections so you can choose the one you like if the whole thing is too long. But the long answer to Piaw contains the idea which is key to the rest of it.

Rebecca: so, first, @Elliot -- "I'm not the first to notice that software has a different relationship with scarcity than physical goods" But my take on the difference is not the usual: I am not repeating the infinitely-copyable thing everyone goes on about, but instead focusing on the scarcity (or increasing lack thereof) of productive capacity. That way of talking challenges more directly the fundamental assumptions of economic theory, and is therefore more intellectually radical: in a formal way, it challenges the justification for capitalism. But you didn't buy my "incompatible with capitalism" argument either, which I'm glad of, because it gives me the chance to mention that just as much as you want to disown the religious view of what capitalism is, I'd like to specifically disown the religious view of the end of capitalism.

Marx talked about an "end of capitalism" as some magic system where it becomes possible for workers to seize the means of production (the factories) and make the economy work without ownership of capital. He also was predicting that capitalism must eventually end, because after all, feudalism had ended. But if you put those two assertions together, and solved the logical syllogism, you would get the assertion that feudalism ended because the serfs seized the means of production (the farms) and made an economy work without the ownership of land. That isn't true! I grew up in Iowa. There are landowners there who own more acreage than most fabled medieval kings. Nobody challenges their ownership, and yet nobody would call that system feudalism. Why not? Because their fields are harvested by machines, not serfs. Feudalism ended not because the landowning class changed their landowning ways. It was because the land-working class, the serfs, left for better jobs in factories; and the landowners don't care anymore, because they eventually replaced the serfs with machines. The end of feudalism was not the end of the ownership of land, it was the end of a social position and set of perogatives that went along with that ownership. If your vassals are machines, you can't lord over them.

Similarly, in a non-religious view of the end of capitalism, it will come about not because the capitalist class, the class that owns factories, will ever disappear or change their ways, but because the proletariat will go away -- they will leave for better jobs doing something else, and the factory owners will replace them with machines. And in fact you can see that that is already happening. Are you proletariat? Am I? If I create an STL model and have it printed by Shapeways, I am manufacturing something, but I am not proletariat. Shapeways is certainly raising capital to buy their printers, which strictly speaking makes them "capitalists," but in a social sense they are not capitalists, because their relationship with me has a different power structure from the one Marx objected to so violently. I am not a "prole" being lorded over by them. It isn't the big dramatic revolution Marx envisioned; it is almost so subtle you can miss it entirely. What if capitalism ended and nobody noticed?

Rebecca: Next @Piaw -- Piaw said he didn't think information technology was the biggest offender in the realm of technology that grows faster than our controls of it; for instance he thought global warming was a more pressing immediate problem.

I definitely agree that the immediate problems created by information technology and the associated social change are, right now, small by comparison to global warming. It would be nice if we could tackle the most immediate and pressing problems first, and leave the others until they get big enough to worry about. But the problems of a new economy have the unique feature of being pressing not because they are necessarily immediate or large (right now), but because if they are left undealt-with they can destroy the political system's ability to effectively handle these or any other problems.

I'm a believer in understanding the present through the lens of the past: since we have so much more perspective about things that happened many, many years ago, we can interpret the present and predict our future by understanding how things that are happening to us now are analogous to things that happened long ago. Towards that end, I'd like to point out an analogy with a fictional image of people who, very early on in the previous "new economy," tried to push new ideas of freedom and met with the objection that they were making too big a deal over problems that were too unimportant. (That this image is fictional is part of my point -- bear with me.) My image comes from a dramatic schene in the musical 1776 (whose synopsis can found at http://en.wikipedia.org/wiki/1776_%28musical%29, scene seven), in which an "obnoxious and disliked" John Adams almost throws away the vote of Edward Rutledge and the rest of the southern delegation over the insistence that a condemnation of slavery be included in the Declaration of Independence. He drops this insistence only when he is persuaded to change his mind by Franklin's arguments that the fight with the British is more important than any argument on the subject -- "we must hang together or we will hang separately."

In fact, nothing like that ever happened: as the historical notes on the Wikipedia page say, everyone at the time was so totally in agreement that the issue was too unimportant to be bothered to fight about it, let alone have the big showdown depicted in the musical, with Rutledge dramatically but improbably singing a spookily beautiful song in defense of the triangle trade: "Molasses to Rum to Slaves." The scene was inserted to satisfy the sensibilities of modern audiences that whether or not such a showdown happened, it should have happened.

Why are our sensibilities so much different than reality? Why are we imposing on the past the idea that the fight ought to have been important to them, even though it wasn't, that John Adams ought to have made himself obnoxious and disliked in his intransigent insistence on America's founding values of freedom, even though he didn't and he wasn't, that Franklin ought to have argued with great reluctance that the fight with the British was more important, even though he never made that argument (because it went without saying), and that Edward Rutledge ought to have been a spooky, equally intransigent apologist for slavery, even though he wasn't either (later he freed his own slaves). We are imposing this false narrative because we are looking backwards through a lens where we know something about the future the real actors had no idea about. This is important to understand because we may be in a similar position with respect to future generations -- they will think we should have had a fight we in fact have no inclination to have, because they will know something we don't know about our own future. The central argument I want to make to Piaw hinges on an understanding of this thing that later generations are likely to know about our future that we currently have difficulty imagining.

So forgive me if I belabor this point: it is key to my answer both to Piaw's question and also to Nick & Helder's objection. Its going to take a little bit of space to set up the scenery, because it is non-trivial for me to pull my audience back into a historical mentality very different than our own. But I want to go through this exercise in order to pull out of it a general understanding of how and why political ways of thinking shift in the face of dramatic technological change -- which we can use to predict how our own future and the changing shape of our politics.

What is it that the real people behind this story didn't know that we know now? Start with John Adams: to understand why the real John Adams wouldn't have been very obnoxious about pushing his idea of freedom on slaveowners in 1776, realize that his idea of freedom, if restated in economic rather than moral terms, would have been the assertion that "it should be an absolute right of all citizens of the United States to leave the farm where they were born and seek a better job in a factory." But making a big deal about such a right in 1776 would have been absurd. There weren't very many factories, and they were sufficiently inefficient that the jobs they provided were unappealing at best. For example, at the time Jefferson wrote in total seriousness about the moral superiority of agrarian over industrial life: such a sentiment seemed reasonable in 1776, because, not to put too fine a point on it, factory life was horrible. Because of this, the politicians in 1776, like Adams or Hamilton, who were deeply enamored of industrialization, pushed their obsession with an apologetic air, as if they were only talking about their own personal predilections, which they took great pains to make clear they were not going to impose on anyone else. The real John Adams was not nearly as obnoxious as our imaginary version of him: we imagine him differently only because we wish he had been different.

We wish him different than he really was because there was one important fact that the people of 1776 may have understood intellectually, but whose full social significance they did not even begin to wrap their minds around: the factories were getting better exponentially, while the farms would always stay the same. Moore's Law-like growth rates in technology are not a new phenomenon. Improvements in the production of cotton textiles in the early nineteenth century stunned observers like the improvements in chips or memory impress us today -- and after cotton-spinning had its run, other advances stepped into the limelight each in turn, as the article at www.theatlantic.com/magazine/archive/1999/10/beyond-the-information-revolution/4658/ tries to impress on us. We forget that dramatic exponential improvements in technology are not a new phenomenon. We also forget that if exponential growth runs for decades, it changes things... and it changes things more than anybody at the beginning of such a run dares to imagine.

This brings us to the other characters in our story who made choices we now wish they had made differently (and they also later regretted). Edward Rutledge and Thomas Jefferson didn't exactly defend slavery; they were quite open about being uncomfortable with it, but they didn't consider this discomfort important enough to do much about. That position would also have made sense in 1776: landowners had owned slaves since antiquity, but slavery in ancient times was not fantastically onerous compared to the other options available to the lower classes at the time -- there are famous stories of enterprising Greek and Roman slaves who earned their freedom and rose to high positions in society. Rutledge and Jefferson probably thought they were offering their slaves a similar deal, and that all in all, it wasn't half bad.

They were wrong. American slavery turned out to be something unique, entirely different than the slavery of antiquity. My American history teacher presented this as a "paradox," that the country that was founded on an ideal of freedom also was home to the most brutal system of slavery the world has ever seen. But I think this "paradox" is quite understandable: it is two parts of the same phenomenon. Ask the question: why could ancient slaveowners afford to be relatively benign? Because they were also relatively secure in their position -- their slaves knew as well as they did that the lower classes didn't have many other better options. Sally Hemmings, Jefferson's lover, considered running away when she was in France with him, but Jefferson successfully convinced her that she would get a better deal staying with him. He didn't have to take her home in chains: she left the possibility of freedom in France and came back of her own free will (if slightly wistfully).

But as time passed and the factory jobs in the North proceeded in their Moore's Law trajectory, eventually the alternatives available to the lower classes began to look better than in any time before in human history. The slaves Harriet Tubman smuggled to Canada arrived to find options exponentially better than those Hemmings could have hoped for if she had left Jefferson. As a result, for the first time in human history, slaves had to be kept in chains.

In the more abstract terms I was using before, slavery was relatively benign when the scarcity of opportunity that bound slaves to their masters was real, but as other opportunities became available, this "real scarcity" became "artificial," something that had to be enforced with chains -- and laws. That is where the slaveowners transformed into something uniquely brutal: to preserve their way of life they needed not only to put their slaves in chains, they also needed to take over the political and legal apparatus of society to keep those chains legal. There came into existence the one-issue politician -- the politician whose motive to enter political life was not to understand or solve the problems facing the nation, to listen to other points of view or forge compromises, or any of the other natural things that a normal politician does, but merely to fight for one issue only: to write into law the "artificial scarcity" that was necessary to preserve the way of life of his constituents, and play whatever brutal political tricks were necessary to keep those laws on the books. Political violence was not off the table - a recent editorial "When Congress Was Armed And Dangerous" (www.nytimes.com/2011/01/12/opinion/12freeman.htm) reminds us that that the incitements to violence of today's politics are tame compared to the violence of the politics of the 1830's, 40's and 50's. The early 1860's were the culmination of the decades-long disaster we wish the Founding Fathers had foreseen and averted. We wish they had had the argument about slavery while there was still time for it to be a mere argument -- before the elite it supported poisoned the political system to provide for its defense.

They, in their old age, wished it too: forty-five years after Jefferson declined to make slavery an important issue in the debate over the Declaration of Independence, he was awakened by the "firebell in the night" in the form of the Missouri compromise. News of this fight caused him to wake up to the real situation, and he wrote to a friend "we have the wolf by the ears, and we can neither hold him, nor safely let him go. Justice is in one scale, and self-preservation in the other.... I regret that I am now to die in the belief that the useless sacrifice of themselves by the generation of '76, to acquire self government and happiness to their country, is to be thrown away by the unwise and unworthy passions of their sons, and that my only consolation is to be that I live not to weep over it."

So, forty-five years after he declined to engage with an "unimportant," "academic" question, he said of the consequences of that decision that his "only consolation is to be that I live not to weep over it." He had not counted on the "unwise and unworthy passions" of his sons -- for his own part, he would have been happy to let slavery lapse when economic conditions no longer gave it moral justification. However, the next generation had different ideas -- they wanted to do anything it took to preserve their prerogatives. By that point the choices he had were defined by the company he kept: since he was a Virginian, he would have had to go to war for Virginia, and fight against everything he believed in. He would have wanted to go back to the time when he could have made a choice that was his own, but that time was past and gone, and no matter how "unwise and unworthy" were the passions which were now controlling him, he had no choice but to be swept along by them.

This is my argument about why we should pay attention to "unimportant" and "academic" questions. In 1776 it was equally well "academic" to consider looking ahead through seventy five years of exponential growth to project the economic conditions of 1860, and use that projection to motivate a serious consideration of abstract principles that were faintly absurd in the conditions of the time, and would only become burning issues decades and decades later. Yet we wish they had done just that, and in their old age they also fervently wished that they had too. This seems strange: why plan for 1860 in 1776? Why plan for 2085 in 2010? Why not just cross that bridge when we come to it? Let the next generation worry about their own problems; why should we think for them? we have our own burning issues to worry about! The projected problems of 2085 are abstract, academic, and unimportant to us. Why not leave them alone and worry about our present burning concerns?

The difficulty is that if we don't leave them alone, if we don't project the battle over our values absurdly into the future and start dealing with the shape of our conflict as it will look when transformed by many decades of time and technological change, we may well lose the political freedom of action to solve these problems non-violently -- or to handle any others either. We will have "a wolf by the ears." We wish the leaders of 1776 had envisioned and taken seriously the problems of 1860, because in 1776 they were still reasonable people who could talk to each other and effectively work out a compromise. By 1860 that option was no longer available. The problem is that when these kinds of problems eventually stop being "academic," when they stop being the dreams of intellectuals and become burning issues for millions of real people, the fire burns too hot. Too many powerful people choose to "take a wolf by the ears". This wolf may well consume the entire political and legal system and make it impossible to handle that problem or any other, until the only option left to restore the body politic is civil war. Once that happens everyone will fervently wish they could go back to the time when the battles were "merely academic".

I worked out this story around 2003, because starting in 1998 I had wanted to have a name to give to a nameless anxiety (in between, I thrashed around for quite a while figuring out which historical image I believed in the most). When I was sure, I considered going to Krugman to use this story to fuel a temper tantrum about how he absolutely had to stop ignoring the geeks who tried to talk to him about "freedom." But I was inhibited: I was afraid the whole argument would come across as intellectually suspect and emotionally manipulative. Besides, the immediate danger this story predicted -- that politics would devolve into 1830's style one-issue paralysis -- seemed a bit preposterous in 2003. Krugman wasn't happy about the 2002 election, but it wasn't that bad. But now I feel some remorse in the other direction: it has gotten worse faster than I ever dreamed it would. I didn't predict what has been happening exactly. I was very focussed on tech, so I didn't expect the politicians in the service of the powerful people with "a wolf by the ears" to be funded by the oldest old economy powers imaginable -- banking and oil. That result isn't incompatible with this argument: that very traditional capitalism should gain an unprecedented brutality just when the new economy is promising new freedoms, is, this line of reasoning predicts, exactly what you should expect. I'm afraid now that Krugman will be mad at me for not bothering him in 2003, because he would have wanted the extra political freedom of action more than he would have resented the very improper intellectual argument.

Rebecca: Now that I've laid the groundwork, it is much easier for me to answer Nick and Helder. Both of you are essentially telling me that I'm being unreasonable and obnoxious. I will break dramatically with Stallman by completely conceding this objection across the board. I am being unreasonably obnoxious. However, there is a general method to this madness: as I explained in the image above, I am essentially pushing values that will make sense in a few decades, and pulling them back to the current time, admittedly somewhat inappropriately. The main reason I think it is important to do this is not because I think the values I am promoting should necessarily apply in an absolute way right now (as Stallman would say) but instead because it is a lot easier to start this fight now than to deal with it later. The reason to fight now is exactly because the opponents are still reasonable, whereas they might not be later. Unlike Stallman, I want to emphasize my respect (and gratitude) for reasonable objections to my position. My opponents are unlikely to shoot me, which is not a priviledge to be taken for granted, and one I want to take advantage of while I have it.

To address the specifics of your objections: Helder complained that companies needed the tactics I called "exploitation of artificial scarcity" to recoup their original investment -- if that wasn't allowed, the service wouldn't exist at all, which would be worse. Nick objected that 80 or 90% of Facebook's planned revenue was from essentially similar sources as Google's, so why should I complain just because of the other 10 or 20%? That was what I was complaining about -- that a portion of their revenue comes from closing their platform and taxing developers -- but that is only a small part of Ms. Sandberg's diversified revenue plans, and I admit that the rest is fairly logically indistinguishable from Google's strategy. In both cases it can easily be argued I am taking an extremely unreasonable hard line.

Let's delve into a dissection of how unreasonable I'm being. In both cases the unreasonableness comes from a problem with my general argument: I said that Mark Zuckerberg is not a capitalist, that is to say, he is not raising capital to buy physical objects that make his workers more productive -- but that is not entirely true. Facebook's data centers are expensive, and they are necessary to allow his employees to do their work.

The best story on this subject might also be the exception to prove the rule. The most "capitalist" story about a tech mogul's start is the account of how Larry & Sergey began by maxing out their credit cards to buy a terabyte of disk (http://books.google.com/books?id=UVz06fnwJvUC&pg=PA6#v=onepage&q&f=false) This story could have been written by Horatio Alger -- it so exactly follows the script of a standard capitalist's start. But for all that, L&S did not make all the standard capitalist noise. I was a fan of Google very early, and may even have pulled data from their original terabyte, and I never heard about how they needed to put restrictions on me to recoup their investment. A year or so later when I talked to Googlers at their recruiting events, I thought they were almost bizarrely chipper about their total lack of revenue strategy. Yet they got rich anyway. And now that same terabyte costs less than $100.

That last is the key point: it is not that the investments aren't significant and need to be recouped. It is that their size is shrinking exponentially. In the previous section, I emphasized the enormous transformative effect of decades of exponential improvement in technology, and the importance of extrapolating one's values forward through those decades, even if that means making assertions that currently seem absurd. The investments that need to be recouped are real and significant, but they are also shrinking, at an exponential rate. So the economic basis for the assertion of a right to restrict people's freedom of opportunity in order to recoup investment is temporary at best. And, as I described in the last part, asserting prerogatives on the basis of scarcity which is now real but will soon be artificial is ... dangerous. Even if you honestly think that you will change with changing times, you may find to your sincere horror that when the time comes to make a new choice, you no longer have the option. Your choices will be dictated by the company you keep. I didn't say it earlier, but one of the things that worries me the most about Facebook is that they seem to have gotten in bed with Goldman Sachs. The idea of soon-to-be multibillionare tech moguls taking lessons in political tactics from Lloyd Blankfein doesn't make me happy.

I am glad that you objected, and gave me licence to take the space to explain this more carefully, because actually my point is more subtle and nuanced than my original account -- which I was oversimplifying to save space -- suggested. (Lessig told me I had to write an account that fit in five pages in order to hope to be heard, to which I reacted with some despair. I can't! There is more than five pages worth of complexity to this problem! If I try to reduce beyond a certain point I burn the sauce. You are watching me struggle in public with this problem.)

There is another part of the mythology of "the end of capitalism" that I should take the time to disavow. The mythology talks as if there is one clear historical moment when an angel of annunciation appears and declares that a new social order has arrived. In reality it isn't like that. It may seem like that in history books when a few pages cover the forty-five years between the time Jefferson scratched out his denunciation of slavery in the Declaration of Independence, and when he wrote the "firebell in the night" letter. But in real, lived, life, forty-five years are most of an adult lifetime. When Jefferson wrote "justice is in one scale, self-preservation in the other," could he point to a particular moment in the previous forty-five years when the hand of justice had moved the weight to the other side of the scale? There was no one moment: just undramatic but steady exponential growth in the productivity of the industrial life whose moral inferiority seemed so obvious four decades earlier. He hadn't been paying attention, no clarion call was blown at the moment of transition (if such a moment even existed), so when he heard the alarm that woke him up to how much the world had changed, it was too late.

In a similar way, I think the only clear judgement we can make now is that we are in the middle of a transition. There was some point in time when disk drives were so expensive and the data stored on them so trivial that the right of their owners to recoup their investment clearly outweighed all other concerns. There will be some other point, decades from now, when the disk drives are so cheap and the data on them so crucial to the livelihood of their users and the health of the economy, that the right to "software freedom" will clearly outweigh the rights of the owner of the hardware. There will be some point clearly "too early" to fight for new freedoms, and some point also clearly "too late." In between these two points in time? Merely a long, slow, steady pace of change. In that time the hand of justice may refuse to put the weight on either side of the scale, no matter how much we plead with her for clarity. We may live our whole lives between two eras, where no judgement can be made black or white, where everything is grey.

But people want solid judgement: they want to know what is right and wrong. This greyness is dangerous, for it opens a vacuum of power eagerly filled by the worst sorts, causes a nameless anxiety, and induces political panic. So what can you do? I think it is impossible to make black-and-white judgements about which era's values should apply. But one can say with certainty that it is desirable to preserve the freedom of political action that will make it possible to defend the values of a new era at the point when it becomes unequivocally clear that that fight is appropriate. I'm really not very upset if companies do whatever it takes to recoup their initial investment -- as long as it's temporary. But what guarantee do I have that if Facebook realizes its 50 billion market capitalization, they won't use that money to buy politicians to justify continuing their practices indefinitely? Their association with Goldman doesn't reassure me on that score. I trust Google more to allow, and even participate in, honest political discussion. That is the issue which I'm really worried about. The speed and effectiveness with which companies are already buying the political discourse has taken me by surprise, even though I had reason to predict it. When powerful people "have a wolf by the ears" they can become truly terrifying.

Rebecca: You could have a point that this kind of argument isn't as unorthodox as it once was. After all, plenty of real economists have been talking about the "new economy" -- Peter Drucker in "Beyond the Information Revolution" (linked above), Hal Varian in "Information Rules", Larry Summers in a speech "The New Wealth of Nations", even Krugman in his short essay "The Dynamo and the Microchip", and a bunch of younger economists surveyed in David Brooks's recent editorial "The Protocol Society." But I don't share Brooks's satisfaction in the observation "it is striking how [these "new economy" theorists] are moving away from mathematical modeling and toward fields like sociology and anthropology." There is a sense that my attitude is now more orthodox than the orthodoxy -- though the argument I sketched here is not a mathematical model, it was very much designed and intended to be turned into one, and I am vehemently in agreement with Krugman's attitude that nothing less than math is good enough. Economics is supposed to be a science where mathematical discipline forces intellectual honesty and provides a bulwark of defense against corruption.

I'm in shock that this crop of "new economy talk" is so loose, sloppy and journalistic ... and because it is so intellectually sloppy, it is hard even to tell whether it is corrupt or not. For instance, though I liked the historical observations and the conclusions drawn from them in Drucker's 1999 essay as much as anything I've read, his paean to the revolutionary effects of e-commerce reads so much like dot.com advertising it is almost embarrassing. Though, to be fair, there are some hints at the essay's conclusion of a consciousness that an industrial revolution isn't just about being sprinkled with technological fairy dust magic, but also involves some aspect of painful social upheaval -- even so, his story is so strangely upbeat, especially since, given his clearly deep understanding of historical thinking, he should have known better ... one wonders whom he was trying not to offend. Similarly, can we trust Summer's purported genius or do his millions in pay from various banks, um, influence his thinking? and it goes on... Krugman is about the only one left I'm sure I can trust. People who do this for a living should be doing better than this! Even though I understand Lessig's point about my intellectual radicalism, it's hard for me to want to follow it, because some part of me just wants to challenge these guys to show enough intellectual rigor to prove that I can trust them.

To be fair, I admit part of Brooks's point that there is something "anthropological" about a new economy: it tends to drive everyone mad. Think about the new industrial rich of the nineteenth century -- dressed to the nines like pseudo-aristocrats, top hat, cane, affected accent, and maybe a bought marriage to get themselves a title too -- they were cuckoo like clocks! Deep, wrenching, technologically-driven change does that to people. But just because it is madness doesn't mean it doesn't have method to it amenable to mathematical modeling. Krugman wrote (http://pkarchive.org/personal/incidents.html) that when he was young, Azimov's "Foundation Trilogy" inspired him to dream of growing up to be a "psychohistorian who use[s his] understanding of the mathematics of society to save civilization as the Galactic Empire collapses" but he said "Unfortunately, there's no such thing (yet)." Do you think he could be tempted with the possibility of a real opportunity to be a "psychohistorian?"

I never meant for this to be a fight I pursue on my own. The whole reason I translated it into an economic and historical language is that I wanted to convince the people who do this for a living to take it up for me. I can't afford to fight alone: I don't have the time to spend caught up in political arguments, nor can I afford to make enemies of people I might want to work for. I'm making these arguments here mostly because having a record of a debate with other tech people will help convince intellectuals of my seriousness. I'm having some difficulty getting you to understand, but I think I would have a terribly uphill battle trying to convince intellectuals that I am not "crying wolf" -- they have just heard this kind of argument misused too many times before. I have to admit that I am crying wolf, but the reason I'm doing it is because this time there really is a wolf!

Piaw: Here's the thing, Rebecca: it wasn't possible to have that argument about freedom/slavery in 1776. The changes brought about later made it possible to have that argument much later. The civil war was horrifying, but I really am not sure if it was possible to change the system earlier.

Ruchira: Hi Rebecca,

I haven't yet read this long conversation. But if you're not already familiar with the concepts of rivalrous vs nonrivalrous and excludable vs nonexcludable

http://en.wikipedia.org/wiki/Rivalry_(economics)

these terms might help connect you with what others have thought about the issues you're talking about. See in particular the "Possible solutions" under Public goods:

http://en.wikipedia.org/wiki/Public_good Daniel Stoddart: I've said it before and I'll say it again: I wouldn't be so quick to count Google out of social. Oh, I know it's cool to diss Buzz like Scoble has been doing for a while now, saying that he has more followers on Quora. But that's kind of an apples and oranges comparison.

Ruchira: Rebecca: Okay, now I have read the long conversation. I do think you have an important point but I haven't digested it enough to form an opinion (which would require judging how it interconnects with other important issues). Just a couple of tangential thoughts:

1) If you fear the loss of freedom, watch out for the military-industrial complex. You've elsewhere described some of the benefits from it, but this is precisely why you shouldn't be lulled into a false sense of comfort by these benefits, just as you're thinking others should not be lulled into a false sense of comfort about the issues you're describing. Think about the long-term consequences of untouchable and unaccountable defense spending, and about the interlocking attributes of the status quo that keep it untouchable and unaccountable. They are fundamentally interconnected with information hiding and lack of transparency.

2) There exists a kind of psychohistory: cliodynamics. http://cliodynamics.info/ As far as I know it's not yet sufficiently developed to apply to the future, though.

Ruchira: Rebecca: On that note, I wonder what you think of Noam Scheiber's article "Why Wikileaks Will Kill Big Business and Big Government" http://www.tnr.com/article/politics/80481/game-changer He's certainly thinking about how technology will cause massive changes in how society is organized.

Helder: (note: I didn't read the whole thing with full attention) In the case of some closed systems. the cost of making it open (and lack business justification for that) and a general necessity to protect the business usually outweighs the need of return on investment by far. So it's not all about ROE.

Also, society's technology development and the shrinking size on capital needs for new business (e.g. terabyte cost), don't usually favors closed system business in the long run, it probably only weakens it. You can have a walled garden, but as the outside ground level goes up, the wall gets shorter and shorter. Just look at how the operating system is increasingly less relevant as most action gravitates towards the browser. Another example (perhaps to be seen?) is the credit card industry as I mentioned in my first comment.

Rebecca: Thanks for reading this long, long post and giving me feedback!

Ruchira: Helder: Facebook makes the wall shorter for its developers (I'm sure Zynga think they've grown wealth due to Facebook). This directly caused an outcry over privacy (the walled garden is not walled any more).

Rebecca: Hope you find it food for thought! You might also be interested in David Singh Grewal's Network Power http://amzn.to/h72nNJ It discusses a lot of relevant issues, and doesn't assume a lot of background (since it's targeted at multiple disciplines), so I found it very helpful, as an outsider like you. After that, you might (or might not) become interested in the coordination problem--if you do, Richard Tuck's free riding http://amzn.to/f9goyT may be of interest.

Rebecca: Thanks, Ruchira, for the links.

show more
Jonathan Shapiro's Retrospective Thoughts on BitC
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2012-03-23 00:00:00 | Created: 2026-07-23 05:18:40

This is an archive of the Jonathan Shapiro's "Retrospective Thoughts on BitC" that seems to have disappeared from the internet; at the time, BitC was aimed at the same niche as Rust

Jonathan S. Shapiro shap at eros-os.org
Fri Mar 23 15:06:41 PDT 2012

By now it will be obvious to everyone that I have stopped work on BitC. An explanation of why seems long overdue.

One answer is that work on Coyotos stopped when I joined Microsoft, and the work that I am focused on now doesn't really require (or seem to benefit from) BitC. As we all know, there is only so much time to go around in our lives. But that alone wouldn't have stopped me entirely.

A second answer is that BitC isn't going to work in its current form. I had hit a short list of issues that required a complete re-design of the language and type system followed by a ground-up new implementation. Experience with the first implementation suggested that this would take quite a while, and it was simply more than I could afford to take on without external support and funding. Programming language work is not easy to fund.

But the third answer may of greatest interest, which is that I no longer believe that type classes "work" in their current form from the standpoint of language design. That's the only important science lesson here.

In the large, there were four sticking points for the current design:

  1. The compilation model.
  2. The insufficiency of the current type system w.r.t. by-reference and reference types.
  3. The absence of some form of inheritance.
  4. The instance coherence problem.

The first two issues are in my opinion solvable, thought the second requires a nearly complete re-implementation of the compiler. The last (instance coherence) does not appear to admit any general solution, and it raises conceptual concerns about the use of type classes for method overload in my mind. It's sufficiently important that I'm going to deal with the first three topics here and take up the last as a separate note.

Inheritance is something that people on the BitC list might (and sometimes have) argue about strongly. So a few brief words on the subject may be relevant.

Prefacing Comments on Objects, Inheritance, and Purity

BitC was initially designed as an [imperative] functional language because of our focus on software verification. Specification of the typing and semantics of functional languages is an area that has a lot of people working on it. We (as a field) kind of know how to do it, and it was an area where our group at Hopkins didn't know very much when we started. Software verification is a known-hard problem, doing it over an imperative language was already a challenge, and this didn't seem like a good place for a group of novice language researchers to buck the current trends in the field. Better, it seemed, to choose our battles. We knew that there were interactions between inheritance and inference, and it appeared that type classes with clever compilation could achieve much of the same operational results. I therefore decided early not to include inheritance in the language.

To me, as a programmer, the removal of inheritance and objects was a very reluctant decision, because it sacrificed any possibility of transcoding the large body of existing C++ code into a safer language. And as it turns out, you can't really remove the underlying semantic challenges from a successful systems language. A systems language requires some mechanism for existential encapsulation. The mechanism which embodies that encapsulation isn't really the issue; once you introduce that sort of encapsulation, you bring into play most of the verification issues that objects with subtyping bring into play, and once you do that, you might as well gain the benefit of objects. The remaining issue, in essence, is the modeling of the Self type, and for a range of reasons it's fairly essential to have a Self type in a systems language once you introduce encapsulation. So you end up pushed in to an object type system at some point in any case. With the benefit of eight years of hindsight, I can now say that this is perfectly obvious!

I'm strongly of the opinion that multiple inheritance is a mess. The argument pro or con about single inheritance still seems to me to be largely a matter of religion. Inheritance and virtual methods certainly aren't the only way to do encapsulation, and they may or may not be the best primitive mechanism. I have always been more interested in getting a large body of software into a safe, high-performance language than I am in innovating in this area of language design. If transcoding current code is any sort of goal, we need something very similar to inheritance.

The last reason we left objects out of BitC initially was purity. I wanted to preserve a powerful, pure subset language - again to ease verification. The object languages that I knew about at the time were heavily stateful, and I couldn't envision how to do a non-imperative object-oriented language. Actually, I'm still not sure I can see how to do that practically for the kinds of applications that are of interest for BitC. But as our faith in the value of verification declined, my personal willingness to remain restricted by purity for the sake of verification decayed quickly.

The other argument for a pure subset language has to do with advancing concurrency, but as I really started to dig in to concurrency support in BitC, I came increasingly to the view that this approach to concurrency isn't a good match for the type of concurrent problems that people are actually trying to solve, and that the needs and uses for non-mutable state in practice are a lot more nuanced than the pure programming approach can address. Pure subprograms clearly play an important role, but they aren't enough.

And I still don't believe in monads. :-)

Compilation Model

One of the objectives for BitC was to obtain acceptable performance under a conventional, static separate compilation scheme. It may be short-sighted on my part, but complex optimizations at run-time make me very nervous from the standpoint of robustness and assurance. I understand that bytecode virtual machines today do very aggressive optimizations with considerable success, but there are a number of concerns with this:

  • For a robust language, we want to minimize the size and complexity of the code that is exempted from type checking and [eventual] verification. Run-time code is excepted in this fashion. The garbage collector taken alone is already large enough to justify assurance concerns. Adding a large and complex optimizer to the pile drags the credibility of the assurance story down immeasurably.
  • Run-time optimization has very negative consequences for startup times
  • especially in the context of transaction processing. Lots of hard data on this from IBM (in DB/2) and others. It is one of the reasons that "Java in the database" never took hold. As the frequency of process and component instantiation in a system rises, startup delays become more and more of a concern. Robust systems don't recycle subsystems.
  • Run-time optimization adds a huge amount of space overhead to the run-time environment of the application. While the code of the run-time compiler can be shared, the state of the run-time compiler cannot, and there is quite a lot of that state.
  • Run-time optimization - especially when it is done "on demand" - introduces both variance and unpredictability into performance numbers. For some of the applications that are of interest to me, I need "steady state" performance. If the code is getting optimized on the fly such that it improves by even a modest constant factor, real-time scheduling starts to be a very puzzling challenge.
  • Code that is produced by a run-time optimizer is difficult to share across address spaces, though this probably isn't solved very well by * other* compilation approaches models either.
  • If run-time optimization is present, applications will come to rely on it for performance. That is: for social reasons, "optional" run-time optimization tends to quickly become required.

To be clear, I'm not opposed to continuous compilation. I actually think it's a good idea, and I think that there are some fairly compelling use-cases. I do think that the run-time optimizer should be implemented in a strongly typed, safe language. I also think that it took an awfully long time for the hotspot technology to stabilize, and that needs to be taken as a cautionary tale. It's also likely that many of the problems/concerns that I have enumerated can be solved - but probably not * soon*. For the applications that are most important to me, the concerns about assurance are primary. So from a language design standpoint, I'm delighted to exploit continuous compilation, but I don't want to design a language that requires continuous compilation in order to achieve reasonable baseline performance.

The optimizer complexity issue, of course, can be raised just as seriously for conventional compilers. You are going to optimize somewhere. But my experience with dynamic translation tells me that it's a lot easier to do (and to reason about) one thing at a time. Once we have a high-confidence optimizer in a safe language, then it may make sense to talk about integrating it into the run-time in a high-confidence system. Until then, separation of concerns should be the watch-word of the day.

Now strictly speaking, it should be said that run-time compilation actually isn't necessary for BitC, or for any other bytecode language. Run-time compilation doesn't become necessary until you combine run-time loading with compiler-abstracted representations (see below) and allow types having abstracted representation to appear in the signatures of run-time loaded libraries. Until then it is possible to maintain a proper phase separation between code generation and execution. Read on - I'll explain some of that below.

In any case, I knew going in that strongly abstracted types would raise concerns on this issue, and I initially adopted the following view:

  • Things like kernels can be whole-program compiled. This effectively eliminates the run-time optimizer requirement.
  • Things like critical system components want to be statically linked anyway, so they can also be dealt with as whole-program compilation problems.
  • For everything else, I hoped to adopt a kind of "template expansion" approach to run-time compilation. This wouldn't undertake the full complexity of an optimizer; it would merely extend run-time linking and loading to incorporate span and offset resolution. It's still a lot of code, but it's not horribly complex code, and it's the kind of thing that lends itself to rigorous - or even formal - specification.

It took several years for me to realize that the template expansion idea wasn't going to produce acceptable baseline performance. The problem lies in the interaction between abstract types, operator overloading, and inlining.

Compiler-Abstracted Representations vs. Optimization

Types have representations. This sometimes seems to make certain members of the PL community a bit uncomfortable. A thing to be held at arms length. Very much like a zip-lock bag full of dog poo (insert cartoon here). From the perspective of a systems person, I regret to report that where the bits are placed, how big they are, and their assemblage actually does matter. If you happen to be a dog owner, you'll note that the "bits as dog poo" analogy is holding up well here. It seems to be the lot of us systems people to wade daily through the plumbing of computational systems, so perhaps that shouldn't be a surprise. Ahem.

In any case, the PL community set representation issues aside in order to study type issues first. I don't think that pragmatics was forgotten, but I think it's fair to say that representation issues are not a focus in current, mainstream PL research. There is even a school of thought that views representation as a fairly yucky matter that should be handled in the compiler "by magic", and that imperative operations should be handled that way too. For systems code that approach doesn't work, because a lot of the representations and layouts we need to deal with are dictated to us by the hardware.

In any case, types do have representations, and knowledge of those representations is utterly essential for even the simplest compiler optimizations. So we need to be a bit careful not to abstract types* too * successfully, lest we manage to break the compilation model.

In C, the "+" operator is primitive, and the compiler can always select the appropriate opcode directly. Similarly for other "core" arithmetic operations. Now try a thought experiment: suppose we take every use of such core operations in a program and replace each one with a functionally equivalent procedure call to a runtime-implemented intrinsic. You only have to do this for user operations - addition introduced by the compiler to perform things like address arithmetic is always done on concrete types, so those can still be generated efficiently. But even though it is only done for user operations, this would clearly harm the performance of the program quite a lot. You can recover that performance with a run-time optimizer, but it's complicated.

In C++, the "+" operator can be overloaded. But (1) the bindings for primitive types cannot be replaced, (2) we know, statically, what the bindings and representations are for the other types, and (3) we can control, by means of inlining, which of those operations entail a procedure call at run time. I'm not trying to suggest that we want to be forced to control that manually. The key point is that the compiler has enough visibility into the implementation of the operation that it is possible to inline the primitive operators (and many others) at static compile time.

Why is this possible in C++, but not in BitC?

In C++, the instantiation of an abstract type (a template) occurs in an environment where complete knowledge of the representations involved is visible to the compiler. That information may not all be in scope to the programmer, but the compiler can chase across the scopes, find all of the pieces, assemble them together, and understand their shapes. This is what induces the "explicit instantiation" model of C++. It also causes a lot of "internal" type declarations and implementation code to migrate into header files, which tends to constrain the use of templates and increase the number of header file lines processed for each compilation unit - we measured this at one point on a very early (pre templates) C++ product and found that we processed more than 150 header lines for each "source" line. The ratio has grown since then by at least a factor of ten, and (because of templates) quite likely 20.

It's all rather a pain in the ass, but it's what makes static-compile-time template expansion possible. From the compiler perspective, the types involved (and more importantly, the representations) aren't abstracted at all. In BitC, both of these things are abstracted at static compile time. It isn't until link time that all of the representations are in hand.

Now as I said above, we can imagine extending the linkage model to deal with this. All of that header file information is supplied to deal with * representation* issues, not type checking. Representation, in the end, comes down to sizes, alignments, and offsets. Even if we don't know the concrete values, we do know that all of those are compile-time constants, and that the results we need to compute at compile time are entirely formed by sums and multiples of these constants. We could imagine dealing with these as opaque constants at static compile time, and filling in the blanks at link time. Which is more or less what I had in mind by link-time template expansion. Conceptually: leave all the offsets and sizes "blank", and rely on the linker to fill them in, much in the way that it handles relocation.

The problem with this approach is that it removes key information that is needed for optimization and registerization, and it doesn't support inlining. In BitC, we can and do extend this kind of instantiation all the way down to the primitive operators! And perhaps more importantly, to primitive accessors and mutators. The reason is that we want to be able to write expressions like "a + b" and say "that expression is well-typed provided there is an appropriate resolution for +:('a,'a)->'a". Which is a fine way to type the operation, but it leaves the representation of 'a fully abstracted. Which means that we cannot see when they are primitive types. Which means that we are exactly (or all too often, in any case) left in the position of generating all user-originated "+" operations as procedure calls. Now surprisingly, that's actually not the end of the world. We can imagine inventing some form of "high-level assembler" that our static code generator knows how to translate into machine code. If the static code generator does this, the run-time loader can be handed responsibility for emitting procedure calls, and can substitute intrinsic calls at appropriate points. Which would cause us to lose code sharing, but that might be tolerable on non-embedded targets.

Unfortunately, this kind of high-level assembler has some fairly nasty implications for optimization: First, we no longer have any idea what the * cost* of the "+" operator is for optimization purposes. We don't know how many cycles that particular use of + will take, but more importantly, we don't know how many bytes of code it will emit. And without that information there is a very long list of optimization decisions that we can no longer make at static compile time. Second, we no longer have enough information at static code generation time to perform a long list of basic register and storage optimizations, because we don't know which procedure calls are actually going to use registers.

That creaking and groaning noise that you are hearing is the run-time code generator gaining weight and losing reliability as it grows. While the impact of this mechanism actually wouldn't be as bad as I am sketching - because a lot of user types aren't abstract - the complexity of the mechanism really is as bad as I am proposing. In effect we end up deferring code generation and optimization to link time. That's an idea that goes back (at least) to David Wall's work on link time register optimization in the mid-1980s. It's been explored in many variants since then. It's a compelling idea, but it has pros and cons.

What is going on here is that types in BitC are too successfully abstracted for static compilation. The result is a rather large bag of poo, so perhaps the PL people are on to something.:-)

Two Solutions

  • The most obvious solution - adopted by C++ - is to redesign the language so that representation issues are not hidden from the compiler. That's actually a solution that is worth considering. The problem in C++ isn't so much the number of header file lines per source line as it is the fact that the C preprocessor requires us to process those lines de novo for each compilation unit. BitC lacks (intentionally) anything comparable to the C preprocessor.
  • The other possibility is to shift to what might be labeled "install time compilation". Ship some form of byte code, and do a static compilation at install time. This gets you back all of the code sharing and optimization that you might reasonably have expected from the classical compilation approach, it opens up some interesting design point options from a systems perspective, and (with care) it can be retrofitted to existing systems. There are platforms today (notably cell phones) where we basically do this already.

The design point that you don't want to cross here is dynamic loading where the loaded interface carries a type with an abstracted representation. At that point you are effectively committing yourself to run-time code generation, though I do have some ideas on how to mitigate that.

Conclusion Concerning Compilation Model

If static, separate compilation is a requirement, it becomes necessary for the compiler to see into the source code across module boundaries whenever an abstract type is used. That is: any procedure having abstract type must have an exposed source-level implementation.

The practical alternative is a high-level intermediate form coupled with install-time or run-time code generation. That is certainly feasible, but it's more that I felt I could undertake.

That's all manageable and doable. Unfortunately, it isn't the path we had taken on, so it basically meant starting over.

Insufficiency of the Type System

At a certain point we had enough of BitC working to start building library code. It may not surprise you that the first thing we set out to do in the library was IO. We found that we couldn't handle typed input within the type system. Why not?

Even if you are prepared to do dynamic allocation within the IO library, there is a level of abstraction at which you need to implement an operation that amounts to "inputStream.read(someObject: ByRef mutable 'a)" There are a couple of variations on this, but the point is that you want the ability at some point to move the incoming bytes into previously allocated storage. So far so good.

Unfortunately, in an effort to limit creeping featurism in the type system, I had declared (unwisely, as it turned out) that the only place we needed to deal with ByRef types was at parameters. Swaroop took this statement a bit more literally than I intended. He noticed that if this is really the only place where ByRef needs to be handled, then you can internally treat "ByRef 'a" as 'a, merely keeping a marker on the parameter's identifier record to indicate that an extra dereference is required at code generation time. Which is actually quite clever, except that it doesn't extend well to signature matching between type classes and their instances. Since the argument type for read is ByRef 'a, InputStream is such a type class.

So now we were faced with a couple of issues. The first was that we needed to make ByRef 'a a first-class type within the compiler so that we could unify it, and the second was that we needed to deal with the implicit coercion issues that this would entail. That is: conversion back and forth between ByRef 'a and 'a at copy boundaries. The coercion part wasn't so bad; ByRef is never inferred, and the type coercions associated with ByRef happen in exactly the same places that const/mutable coercions happen. We already had a cleanly isolated place in the type checker to deal with that.

But even if ByRef isn't inferred, it can propagate through the code by unification. And that causes safety violations! The fact that ByRef was syntactically restricted to appear only at parameters had the (intentional) consequence of ensuring that safety restrictions associated with the lifespan of references into the stack were honored - that was why I had originally imposed the restriction that ByRef could appear only at parameters. Once the ByRef type can unify, the syntactic restriction no longer guarantees the enforcement of the lifespan restriction. To see why, consider what happens in:

  define byrefID(x:ByRef 'a) { return x; }

Something that is supposed to be a downward-only reference ends up getting returned up the stack. Swaroop's solution was clever, in part, because it silently prevented this propagation problem. In some sense, his implementation doesn't really treat ByRef as a type, so it can't propagate. But *because *he didn't treat it as a type, we also couldn't do the necessary matching check between instances and type classes.

It turns out that being able to do this is useful. The essential requirement of an abstract mutable "property" (in the C# sense) is that we have the ability within the language to construct a function that returns the location of the thing to be mutated. That location will often be on the stack, so returning the location is exactly like the example above. The "ByRef only at parameters" restriction is actually very conservative, and we knew that it was preventing certain kinds of things that we eventually wanted to do. We had a vague notion that we would come back and fix that at a later time by introducing region types.

As it turned out, "later" had to be "now", because region types are the right way to re-instate lifetime safety when ByRef types become first class. But adding region types presented two problems (which is why we had hoped to defer them):

  • Adding region types meant rewriting the type checker and re-verifying the soundness and completeness of the inference algorithm, and
  • It wasn't just a re-write. Regions introduce subtyping. Subtyping and polymorphism don't get along, so we would need to go back and do a lot of study.

Region polymorphism with region subtyping had certainly been done before, but we were looking at subtyping in another case too (below). That was pushing us toward a kinding system and a different type system.

So to fix the ByRef problem, we very nearly needed to re-design both the type system and the compiler from scratch. Given the accumulation of cruft in the compiler, that might have been a good thing in any case, but Swaroop was now full-time at Microsoft, and I didn't have the time or the resources to tackle this by myself.

Conclusion Concerning the Type System

In retrospect, it's hard to imagine a strongly typed imperative language that doesn't type locations in a first-class way. If the language simultaneously supports explicit unboxing, it is effectively forced to deal with location lifespan and escape issues, which makes memory region typing of some form almost unavoidable.

For this reason alone, even if for no other, the type system of an imperative language with unboxing must incorporate some form of subtyping. To ensure termination, this places some constraints on the use of type inference. On the bright side, once you introduce subtyping you are able to do quite a number of useful things in the language that are hard to do without it.

Inheritance and Encapsulation

Our first run-in with inheritance actually showed up in the compiler itself. In spite of our best efforts, the C++ implementation of the BitC compiler had not entirely avoided inheritance, so it didn't have a direct translation into BitC. And even if we changed the code of the compiler, there are a large number of third-party libraries that we would like to be able to transcode. A good many of those rely on [single] inheritance. Without having at least some form of interface (type) inheritance, We can't really even do a good job interfacing to those libraries as foreign objects.

The compiler aside, we also needed a mechanism for encapsulation. I had been playing with "capsules", but it soon became clear that capsules were really a degenerate form of subclassing, and that trying to duck that issue wasn't going to get me anywhere.

I could nearly imagine getting what I needed by adding "ThisType" and inherited interfaces. But the combination of those two features introduces subtyping. In fact, the combination is equivalent (from a type system perspective) to single-inheritance subclassing.

And the more I stared at interfaces, the more I started to ask myself why an interface wasn't just a type class. That brought me up against the instance coherence problem from a new direction, which was already making my head hurt. It also brought me to the realization that Interfaces work, in part, because they are always parameterized over a single type (the ThisType) - once you know that one, the bindings for all of the others are determined by type constructors or by explicit specification.

And introducing SelfType was an even bigger issue than introducing subtypes. It means moving out of System F<: entirely, and into the object type system of Cardelli et al. That wasn't just a matter of re-implementing the type checker to support a variant of the type system we already had. It meant re-formalizing the type system entirely, and learning how to think in a different model.

Doable, but time not within the framework or the compiler that we had built. At this point, I decided that I needed to start over. We had learned a lot from the various parts of the BitC effort, but sometimes you have to take a step back before you can take more steps forward.

Instance Coherence and Operator Overloading

BitC largely borrows its type classes from Haskell. Type classes aren't just a basis for type qualifiers; they provide the mechanism for *ad hoc*polymorphism. A feature which, language purists notwithstanding, real languages actually do need.

The problem is that there can be multiple type class instances for a given type class at a given type. So it is possible to end up with a function like:

define f(x : 'x) {
  ...
  a:int32 + b  // typing fully resolved at static compile time
  return x + x  // typing not resolvable until instantiation
}

Problem: we don't know which instance of "+" to use when 'x instantiates to int32. In order for "+" to be meaningful in a+b, we need a static-compile-time resolution for +:(int32, int32)->int32. And we get that from Arith(int32). So far so good. But if 'x is instantiated to int32, we will get a type class instance supplied by the caller. The problem is that there is no way to guarantee that this is the same instance of Arith(int32) that we saw before.

The solution in Haskell is to impose the ad hoc rule that you can only instantiate a type class once for each unique type tuple in a given application. This is similar to what is done in C++: you can only have one overload of a given global operator at a particular type. If there is more than one overload at that type, you get a link-time failure. This restriction is tolerable in C++ largely because operator overloading is so limited:

  1. The set of overloadable operators is small and non-extensible.
  2. Most of them can be handled satisfactorily as methods, which makes their resolution unambiguous.
  3. Most of the ones that can't be handled as methods are arithmetic operations, and there are practical limits to how much people want to extend those.
  4. The remaining highly overloaded global operators are associated with I/O. These could be methods in a suitably polymorphic language.

In languages (like BitC) that enable richer use of operator overloading, it seems unlikely that these properties would suffice.

But in Haskell and BitC, overloading is extended to type properties as well. For example, there is a type class "Ord 'a", which states whether a type 'a admits an ordering. Problem: most types that admit ordering admit more than one! The fact that we know an ordering exists really isn't enough to tell us which ordering to use. And we can't introduce two orderings for 'a in Haskell or BitC without creating an instance coherence problem. And in the end, the instance coherence problem exists because the language design performs method resolution in what amounts to a non-scoped way.

But if nothing else, you can hopefully see that the heavier use of overloading in BitC and Haskell places much higher pressure on the "single instance" rule. Enough so, in my opinion, to make that rule untenable. And coming from the capability world, I have a strong allergy to things that smell like ambient authority.

Now we can get past this issue, up to a point, by imposing an arbitrary restriction on where (which compilation unit) an instance can legally be defined. But as with the "excessively abstract types" issue, we seemed to keep tripping on type class issues. There are other problems as well when multi-variable type classes get into the picture.

At the end of the day, type classes just don't seem to work out very well as a mechanism for overload resolution without some other form of support.

A second problem with type classes is that you can't resolve operators at static compile time. And if instances are explicitly named, references to instances have a way of turning into first-class values. At that point the operator reference can no longer be statically resolved at all, and we have effectively re-invented operator methods!

Conclusion about Type Classes and Overloading:

The type class notion (more precisely: qualified types) is seductive, but absent a reasonable approach for instance coherence and lexical resolution it provides an unsatisfactory basis for operator overloading. There is a disturbingly close relationship between type class instances and object instances that needs further exploration by the PL community. The important distinction may be pragmatic rather than conceptual: type class instances are compile-time constants while object instances are run-time values. This has no major consequences for typing, but it leads to significant differences w.r.t. naming, binding, and [human] conceptualization.

There are unresolved formal issues that remain with multi-parameter type classes. Many of these appear to have natural practical solutions in a polymorphic object type system, but concerns of implementation motivate kinding distinctions between boxed and unboxed types that are fairly unsatisfactory.

Wrapping Up

The current outcome is extremely frustrating. While the blind spots here were real, we were driven by the requirements of the academic research community to spend nearly three years finding a way to do complete inference over mutability. That was an enormous effort, and it delayed our recognition that we were sitting on the wrong kind of underlying type system entirely. While I continue to think that there is some value in mutability inference, I think it's a shame that a fairly insignificant wart in the original inference mechanism managed to prevent larger-scale success in the overall project for what amount to political reasons. If not for that distraction, I think we would probably have learned enough about the I/O and the instance coherency issues to have moved to a different type system while we still had a group to do it with, and we would have a working and useful language today.

The distractions of academia aside, it is fair to ask why we weren't building small "concept test" programs as a sanity check of our design. There are a number answers, none very satisfactory:

  • Research languages can adopt simplifications on primitive types (notably integers) that systems languages cannot. That's what pushed us into type classes in the first place, we new that polymorphism over unboxed types hadn't seen a lot of attention in the literature, and we knew that mutability inference had never been done. We had limited manpower, so we chose to focus on those issues first.
  • We knew that parametric polymorphism and subtyping didn't get along, so we wanted to avoid that combination. Unfortunately, we avoided subtypes too well for too long, and they turned out to be something unavoidable.
  • For the first several years, we were very concerned with software verification, which also drove us strongly away from object-based languages and subtyping. That blinded us.
  • Coming to language design as "systems" people, working in a department that lacked deep expertise and interest in type systems, there was an enormous amount of subject matter that we needed to learn. Some of the reasons for our failure are "obvious" to people in the PL community, but others are not. Our desire for a "systems" language drove us to explore the space in a different way and with different priorities than are common in the PL community.

I think we did make some interesting contributions. We now know how to do (that is: to implement) polymorphism over unboxed types with significant code sharing, and we understand how to deal with inferred mutability. Both of those are going to be very useful down the road. We have also learned a great deal about advanced type systems.

In any case, BitC in its current form clearly needs to be set aside and re-worked. I have a fairly clear notion about how I would approach continuing this work, but that's going to have to wait until someone is willing to pay for all this.

show more
Kara Swisher interview of Jack Dorsey
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-02-12 00:00:00 | Created: 2026-07-23 05:18:40

This is a transcript of the Kara Swisher / Jack Dorsey interview from 2/12/2019, made by parsing the original Tweets because I wanted to be able to read this linearly. There's a "moment" that tries to track this, but since it doesn't distinguish between sub-threads in any way, you can't tell the difference between end of a thread and a normal reply. This linearization of the interview marks each thread break with a page break and provides some context from upthread where relevant (in grey text).

Kara: Here in my sweatiest @soulcycle outfit for my Twitterview with @jack with @Laur_Katz at the ready @voxmediainc HQ. Also @cheezit acquired. #karajack

Kara: Oh hai @jack. Let’s set me set the table. First, I am uninterested in beard amulets or weird food Mark Zuckerberg served you (though WTF with both for my personal self). Second, I would appreciate really specific answers.

Jack: Got you. Here’s my setup. I work from home Tuesdays. In my kitchen. Tweetdeck. No one here with me, and no one connected to my tweetdeck. Just me focused on your questions!

Kara: Great, let's go

Jack: Ready


Kara: As @ashleyfeinberg wrote: “press him for a clear, unambiguous example of nearly anything, and Dorsey shuts down.” That is not unfair characterization IMHO. Third, I will thread in questions from audience, but to keep this non chaotic, let’s stay in one reply thread.

Jack: Deal


Kara: To be clear with audience, there is not a new event product, a glass house, if you will, where people can see us but not comment. I will ask questions and then respond to @jack answers. So it could be CHAOS.

Jack: To be clear, we’re interested in an experience like this. Nothing built yet. This gives us a sense of what it would be like, and what we’d need to focus on. If there’s something here at all!

Kara: Well an event product WOULD BE NICE. See my why aren't you moving faster trope.


Kara: Overall here is my mood and I think a lot of people when it comes to fixing what is broke about social media and tech: Why aren’t you moving faster? Why aren’t you moving faster? Why aren’t you moving faster?

Jack: A question we ask ourselves all the time. In the past I think we were trying to do too much. We’re better at prioritizing by impact now. Believe the #1 thing we should focus on is someone’s physical safety first. That one statement leads to a lot of ramifications.

Kara: It seems twitter has been stuck in a stagnant phase of considering/thinking about the health of the conversation, which plays into safety, for about 18-24 months. How have you made actual progress? Can you point me to it SPECIFICALLY?


Kara: You know my jam these days is tech responsibility. What grade do you gave Silicon Valley? Yourself?

Jack: Myself? C. We’ve made progress, but it has been scattered and not felt enough. Changing the experience hasn’t been meaningful enough. And we’ve put most of the burden on the victims of abuse (that’s a huge fail).

Kara: Well that is like telling me I am sick and am responsible for fixing it. YOU made the product, YOU run the platform. Saying it is a huge fail is a cop out to many. It is to me

Jack: Putting the burden on victims? Yes. It’s recognizing that we have to be proactive in enforcement and promotion of healthy conversation. This is our first priority in #health. We have to change a lot of the fundamentals of product to fix.

Kara: please be specific. I see a lot of beard-stroking on this (no insult to your Lincoln jam, but it works). WHAT are you changing? SPECIFICALLY.

Jack: First and foremost we’re looking at ways to proactively enforce and promote health. So that reporting/blocking is a last resort. Problem we’re trying to solve is taking that work away.

Kara: Ok name three initiatives.


Jack: Myself? C. We’ve made progress, but it has been scattered and not felt enough. Changing the experience hasn’t been meaningful enough. And we’ve put most of the burden on the victims of abuse (that’s a huge fail).

Kara: Also my son gets a C in coding and that is NO tragedy. You getting one matters a lot.

Jack: Agree it matters a lot. And it’s the most important thing we need to address and fix. I’m stating that it’s a fail of ours to put the majority of burden on victims. That’s how the service works today.

Kara: Ok but I really want to drill down on HOW. How much downside are you willing to tolerate to balance the good that Twitter can provide? Be specific

Jack: This is exactly the balance we have to think deeply about. But in doing so, we have to look at how the product works. And where abuse happens the most: replies, mentions, search, and trends. Those are the shared spaces people take advantage of

Kara: Well, WHERE does abuse happen most

Jack: Within the service? Likely within replies. That’s why we’ve been more aggressive about proactively downranking behind interstitials, for example.

Kara: Why not just be more stringent on kicking off offenders? It seems like you tolerate a lot. If Twitter ran my house, my kids would be eating ramen, playing Red Dead Redemption 2 and wearing filthy socks

Jack: We action all we can against our policies. Most of our system today works reactively to someone reporting it. If they don’t report, we don’t see it. Doesn’t scale. Hence the need to focus on proactive

Kara: But why did you NOT see it? It seems pretty basic to run your platform with some semblance of paying mind to what people are doing on it? Can you give me some insight into why that was not done?

Jack: I think we tried to do too much in the past, and that leads to diluted answers and nothing impactful. There’s a lot we need to address globally. We have to prioritize our resources according to impact. Otherwise we won’t make much progress.

Kara: Got it. But do you think the fact that you all could not conceive of what it is to feel unsafe (women, POC, LGBTQ, other marginalized people) could be one of the issues? (new topic soon)

Jack: I think it’s fair and real. No question. Our org has to be reflective of the people we’re trying to serve. One of the reason we established the Trust and Safety council years ago, to get feedback and check ourselves.

Kara: Yes but i want THREE concrete examples.


Jack: First and foremost we’re looking at ways to proactively enforce and promote health. So that reporting/blocking is a last resort. Problem we’re trying to solve is taking that work away.

Kara: Or maybe, tell me what you think the illness is you are treating? I think you cannot solve a disease without knowing that. Or did you create the virus?

Jack: Good question. This is why we’re focused on understanding what conversational health means. We see a ton of threats to health in digital conversation. We’re focuse first on off-platform ramifications (physical safety). That clarifies priorities of policy and enforcement.

Kara: I am still confused. What the heck is "off-platform ramifications"? You are not going to have a police force, right? Are you 911?

Jack: No, not a police force. I mean we have to consider first and foremost what online activity does to impact physical safety, as a way to prioritize our efforts. I don’t think companies like ours have admitted or focused on that enough.

Kara: So you do see the link between what you do and real life danger to people? Can you say that explicitly? I could not be @finkd to even address the fact that he made something that resulted in real tragedy.

Jack: I see the link, and that’s why we need to put physical safety above all else. That’s what we’re figuring out how to do now. We don’t have all the answers just yet. But that’s the focus. I think it clarifies a lot of the work we need to do. Not all of it of course.

Kara: I grade you all an F on this and that's being kind. I'm not trying to be a jackass, but it's been a very slow roll by all of you in tech to pay attention to this. Why do you think that is? I think it is because many of the people who made Twitter never ever felt unsafe.

Jack: Likely a reason. I’m certain lack of diversity didn’t help with empathy of what people experience on Twitter every day, especially women.

Kara: And so to end this topic, I will try again. Please give me three concrete things you have done to fix this. SPECIFIC.

Jack: 1. We have evolved our polices. 2. We have prioritized proactive enforcement to remove burden from victims 3. We have given more control in product (like mute of accounts without profile pics or associated phone/emails) 4. Much more aggressive on coordinated behavior/gaming

Kara: 1. WHICH? 2. HOW? 3. OK, MUTE BUT THAT WAS A WHILE AGO 4. WHAT MORE? I think people are dying for specifics.

Jack: 1. Misgendering policy as example. 2. Using ML to downrank bad actors behind interstitials 3. Not too long ago, but most of our work going forward will have to be product features. 4. Not sure the question. We put an entire model in place to minimize gaming of system.

Kara: thx. I meant even more specifics on 4. But see the Twitter purge one.

Jack: Just resonded to that. Don’t see the twitter purge one

Kara: I wanted to get off thread with Mark added! Like he needs more of me.

Jack: Does he check this much?

Kara: No, he is busy fixing Facebook. NOT! (he makes you look good)

Kara: I am going to start a NEW thread to make it easy for people to follow (@waltmossberg just texted me that it is a "chaotic hellpit"). Stay in that one. OK?

Jack: Ok. Definitely not easy to follow the conversation. Exactly why we are doing this. Fixing stuff like this will help I believe.

Kara: Yeah, it's Chinatown, Jake.


Jack: First and foremost we’re looking at ways to proactively enforce and promote health. So that reporting/blocking is a last resort. Problem we’re trying to solve is taking that work away.

Jack: Second, we’re constantly evolving our policies to address the issues we see today. We’re rooting them in fundamental human rights (UN) and putting physical safety as our top priority. Privacy next.

Kara: When you say physical safety, I am confused. What do you mean specifically? You are not a police force. In fact, social media companies have built cities without police, fire departments, garbage pickup or street signs. IMHO What do you think of that metaphor?

Jack: I mean off platform, offline ramifications. What people do offline with what they see online. Doxxing is a good example which threatens physical safety. So does coordinate harassment campaigns.

Kara: So how do you stop THAT? I mean regular police forces cannot stop that. It seems your job is not to let it get that far in the first place.

Jack: Exactly. What can we do within the product and policy to lower probability. Again, don’t think we or others have worked against that enough.


Kara: Ok, new one @jack

What do you think about twitter breaks and purges. Why do you think that is? I can’t say I’ve heard many people say they feel “good” after not being on twitter for a while: https://twitter.com/TaylorLorenz/status/1095039347596898305

Jack: Feels terrible. I want people to walk away from Twitter feeling like they learned something and feeling empowered to some degree. It depresses me when that’s not the general vibe, and inspires me to figure it out. That’s my desire

Kara: But why do they feel that way? You made it.

Jack: We made something with one intent. The world showed us how it wanted to use it. A lot has been great. A lot has been unexpected. A lot has been negative. We weren’t fast enough to observe, learn, and improve


Kara: Ok, new one @jack

Kara: In that vein, how does it affect YOU?

Jack: I also don’t feel good about how Twitter tends to incentivize outrage, fast takes, short term thinking, echo chambers, and fragmented conversation and consideration. Are they fixable? I believe we can do a lot to address. And likely have to change more fundamentals to do so.

Kara: But you invented it. You can control it. Slowness is not really a good excuse.

Jack: It’s the reality. We tried to do too much at once and were not focused on what matters most. That contributes to slowness. As does our technology stack and how quickly we can ship things. That’s improved a lot recently


Kara: Ok trying AGAIN @jack in another new thread! This one about @realDonaldTrump:

We know a lot more about what Donald Trump thinks because of Twitter, and we all have mixed feelings about that.

Kara: Have you ever considered suspending Donald Trump? His tweets are somewhat protected because he’s a public figure, but would he have been suspended in the past if he were a “regular” user?

Jack: We hold all accounts to the same terms of service. The most controversial aspect of our TOS is the newsworthy/public interest clause, the “protection” you mention. That doesn’t extend to all public figures by default, but does speak to global leaders and seeing how they think.

Kara: That seems questionable to a lot of people. Let me try it a different way: What historic newsworthy figure would you ban? Is someone bad enough to ban. Be specific. A name.

Jack: We have to enforce based on our policy and what people do on our service. And evolve it with the current times. No way I can answer that based on people. Has to be focused on patterns of how people use the technology.

Kara: Not one name? Ok, but it is a copout imho. I have a long list.

Jack: I think it’s more durable to focus on use cases because that allows us to act broader. Likely that these aren’t isolated cases but things that spread

Kara: it would be really great to get specific examples as a lot of what you are doing appears incomprehensible to many.


Kara: Ok trying AGAIN @jack in another new thread! This one about @realDonaldTrump:

Kara: And will Twitter’s business/engagement suffer when @realDonaldTrump is no longer President?

Jack: I don’t believe our service or business is dependent on any one account or person. I will say the number of politics conversations has significantly increased because of it, but that’s just one experience on Twitter. There are multiple Twitters, all based on who you follow.

Kara: Ok new question (answer the newsworthy historical figure you MIGHT ban pls): Single biggest improvement at Twitter since 2016 that signals you’re ready for the 2020 elections?

Jack: Our work against automations and coordinated campaigns. Partnering with government agencies to improve communication around threats

Kara: Can you give a more detailed example of that that worked?

Jack: We shared a retro on 2018 within this country, and tested a lot with the Mexican elections too. Indian elections coming up. In mid-terms we were able to monitor efforts to disrupt both online and offline and able to stop those actions on Twitter.


Kara: Ok new question (answer the newsworthy historical figure you MIGHT ban pls): Single biggest improvement at Twitter since 2016 that signals you’re ready for the 2020 elections?

Kara: What confidence should we have that Russia or other state-sponsored actors won’t be able to wreak havoc on next year’s elections?

Jack: We should expect a lot more coordination between governments and platforms to address. That would give me confidence. And have some skepticism too. That’s healthy. The more we can do this work in public and share what we find, the better

Kara: I still am dying for specifics here. [meme image: Give me some specifics. I love specifics, the specifics were the best part!]


Jack: I think it’s more durable to focus on use cases because that allows us to act broader. Likely that these aren’t isolated cases but things that spread

Kara: going to shift to biz questions since it is not a lot of time and this system is CHAOTIC (as I thought it would be): What about the move to DAU instead of MAU. Why the move? And how are we to interpret the much smaller numbers?

Jack: We want to be valuable to people daily. Not monthly. It’s a higher bar for ourselves. Sure, it looks like a smaller absolute number, but the folks we have using Twitter are some of the most influential in the world. They drive conversation. We belevie we can best grow this.

Kara: Ok, then WHO is the most exciting influential on Twitter right now? BE SPECIFIC

Jack: To me personally? I like how @elonmusk uses Twitter. He’s focused on solving existential problems and sharing his thinking openly. I respect that a lot, and all the ups and downs that come with it

Kara: What about @AOC

Jack: Totally. She’s mastering the medium

Kara: She is well beyond mastering it. She speaks fluent Twitter.

Jack: True

Kara: Also are you ever going to hire someone to effectively be your number 2?

Jack: I think it’s better to spread that responsibility across multiple people. It creates less dependencies and the company gets more options around future leadership


Kara: going to shift to biz questions since it is not a lot of time and this system is CHAOTIC (as I thought it would be): What about the move to DAU instead of MAU. Why the move? And how are we to interpret the much smaller numbers?

Kara: Also: How close were you to selling Twitter in 2016? What happened?

What about giving the company to a public trust per your NYT discussion.

Jack: We ultimately decided we were better off independent. And I’m happy we did. We’ve made a lot of progress since that point. And we got a lot more focused. Definitely love the idea of opening more to 3rd parties. Not sure what that looks like yet. Twitter is close to a protocol.

Kara: Chop chop on the other answers! I have more questions! If you want to use this method, quicker!

Jack: I’m moving as fast as I can Kara

Kara: Clip clop!


Kara: going to shift to biz questions since it is not a lot of time and this system is CHAOTIC (as I thought it would be): What about the move to DAU instead of MAU. Why the move? And how are we to interpret the much smaller numbers?

Kara: also: Is twitter still considering a subscription service? Like “Twitter Premium” or something?

Jack: Always going to experiment with new models. Periscope has super hearts, which allows us to learn about direct contribution. We’d need to figure out the value exchange on subscription. Has to be really high for us to charge directly


Jack: Totally. She’s mastering the medium

Kara: Ok, last ones are about you and we need to go long because your system here it confusing says the people of Twitter:

  1. What has been Twitter’s biggest missed opportunity since you came back as CEO?

Jack: Focus on conversation earlier. We took too long to get there. Too distracted.

Kara: By what? What is the #1 thing that distracted you and others and made this obvious mess via social media?

Jack: Tried to do too much at once. Wasn’t focused on what our one core strength was: conversation. That lead to really diluted strategy and approach. And a ton of reactiveness.

Kara: Speaking of that (CONVERSATION), let's do one with sounds soon, like this

https://www.youtube.com/watch?v=oiJkANps0Qw


Kara: She is well beyond mastering it. She speaks fluent Twitter.

Jack: True

Kara: Why are you still saying you’re the CEO of two publicly traded companies? What’s the point in insisting you can do two jobs that both require maximum effort at the same time?

Jack: I’m focused on building leadership in both. Not my desire or ambition to be CEO of multiple companies just for the sake of that. I’m doing everything I can to help both. Effort doesn’t come down to one person. It’s a team


Kara: LAST Q: For the love of God, please do Recode Decode podcast with me soon, because analog talking seems to be a better way of asking questions and giving answers. I think Twitter agrees and this has shown how hard this thread is to do. That said, thx for trying. Really.

Jack: This thread was hard. But we got to learn a ton to fix it. Need to make this feel a lot more cohesive and easier to follow. Was extremely challenging. Thank you for trying it with me. Know it wasn’t easy. Will consider different formats!

Kara: Make a glass house for events and people can watch and not throw stones. Pro tip: Twitter convos are wack

Jack: Yep. And they don’t have to be wack. Need to figure this out. This whole experience is a problem statement for what we need to fix


Jack: This thread was hard. But we got to learn a ton to fix it. Need to make this feel a lot more cohesive and easier to follow. Was extremely challenging. Thank you for trying it with me. Know it wasn’t easy. Will consider different formats!

Kara: My kid is hungry and says that you should do a real interview with me even if I am mean. Just saying.

Jack: I don’t think you’re mean. Always good to experiment.

Kara: Neither does my kid. He just wants to go get dinner

Jack: Go eat! Thanks, Kara

show more
Latency mitigation strategies (by John Carmack)
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-03-05 00:00:00 | Created: 2026-07-23 05:18:40

this is an archive of an old article by John Carmack which seems to have disappeared off of the internet

Abstract

Virtual reality (VR) is one of the most demanding human-in-the-loop applications from a latency standpoint. The latency between the physical movement of a user’s head and updated photons from a head mounted display reaching their eyes is one of the most critical factors in providing a high quality experience.

Human sensory systems can detect very small relative delays in parts of the visual or, especially, audio fields, but when absolute delays are below approximately 20 milliseconds they are generally imperceptible. Interactive 3D systems today typically have latencies that are several times that figure, but alternate configurations of the same hardware components can allow that target to be reached.

A discussion of the sources of latency throughout a system follows, along with techniques for reducing the latency in the processing done on the host system.

Introduction

Updating the imagery in a head mounted display (HMD) based on a head tracking sensor is a subtly different challenge than most human / computer interactions. With a conventional mouse or game controller, the user is consciously manipulating an interface to complete a task, while the goal of virtual reality is to have the experience accepted at an unconscious level.

Users can adapt to control systems with a significant amount of latency and still perform challenging tasks or enjoy a game; many thousands of people enjoyed playing early network games, even with 400+ milliseconds of latency between pressing a key and seeing a response on screen.

If large amounts of latency are present in the VR system, users may still be able to perform tasks, but it will be by the much less rewarding means of using their head as a controller, rather than accepting that their head is naturally moving around in a stable virtual world. Perceiving latency in the response to head motion is also one of the primary causes of simulator sickness. Other technical factors that affect the quality of a VR experience, like head tracking accuracy and precision, may interact with the perception of latency, or, like display resolution and color depth, be largely orthogonal to it.

A total system latency of 50 milliseconds will feel responsive, but still subtly lagging. One of the easiest ways to see the effects of latency in a head mounted display is to roll your head side to side along the view vector while looking at a clear vertical edge. Latency will show up as an apparent tilting of the vertical line with the head motion; the view feels “dragged along” with the head motion. When the latency is low enough, the virtual world convincingly feels like you are simply rotating your view of a stable world.

Extrapolation of sensor data can be used to mitigate some system latency, but even with a sophisticated model of the motion of the human head, there will be artifacts as movements are initiated and changed. It is always better to not have a problem than to mitigate it, so true latency reduction should be aggressively pursued, leaving extrapolation to smooth out sensor jitter issues and perform only a small amount of prediction.

Data collection

It is not usually possible to introspectively measure the complete system latency of a VR system, because the sensors and display devices external to the host processor make significant contributions to the total latency. An effective technique is to record high speed video that simultaneously captures the initiating physical motion and the eventual display update. The system latency can then be determined by single stepping the video and counting the number of video frames between the two events.

In most cases there will be a significant jitter in the resulting timings due to aliasing between sensor rates, display rates, and camera rates, but conventional applications tend to display total latencies in the dozens of 240 fps video frames.

On an unloaded Windows 7 system with the compositing Aero desktop interface disabled, a gaming mouse dragging a window displayed on a 180 hz CRT monitor can show a response on screen in the same 240 fps video frame that the mouse was seen to first move, demonstrating an end to end latency below four milliseconds. Many systems need to cooperate for this to happen: The mouse updates 500 times a second, with no filtering or buffering. The operating system immediately processes the update, and immediately performs GPU accelerated rendering directly to the framebuffer without any page flipping or buffering. The display accepts the video signal with no buffering or processing, and the screen phosphors begin emitting new photons within microseconds.

In a typical VR system, many things go far less optimally, sometimes resulting in end to end latencies of over 100 milliseconds.

Sensors

Detecting a physical action can be as simple as a watching a circuit close for a button press, or as complex as analyzing a live video feed to infer position and orientation.

In the old days, executing an IO port input instruction could directly trigger an analog to digital conversion on an ISA bus adapter card, giving a latency on the order of a microsecond and no sampling jitter issues. Today, sensors are systems unto themselves, and may have internal pipelines and queues that need to be traversed before the information is even put on the USB serial bus to be transmitted to the host.

Analog sensors have an inherent tension between random noise and sensor bandwidth, and some combination of analog and digital filtering is usually done on a signal before returning it. Sometimes this filtering is excessive, which can contribute significant latency and remove subtle motions completely.

Communication bandwidth delay on older serial ports or wireless links can be significant in some cases. If the sensor messages occupy the full bandwidth of a communication channel, latency equal to the repeat time of the sensor is added simply for transferring the message. Video data streams can stress even modern wired links, which may encourage the use of data compression, which usually adds another full frame of latency if not explicitly implemented in a pipelined manner.

Filtering and communication are constant delays, but the discretely packetized nature of most sensor updates introduces a variable latency, or “jitter” as the sensor data is used for a video frame rate that differs from the sensor frame rate. This latency ranges from close to zero if the sensor packet arrived just before it was queried, up to the repeat time for sensor messages. Most USB HID devices update at 125 samples per second, giving a jitter of up to 8 milliseconds, but it is possible to receive 1000 updates a second from some USB hardware. The operating system may impose an additional random delay of up to a couple milliseconds between the arrival of a message and a user mode application getting the chance to process it, even on an unloaded system.

Displays

On old CRT displays, the voltage coming out of the video card directly modulated the voltage of the electron gun, which caused the screen phosphors to begin emitting photons a few microseconds after a pixel was read from the frame buffer memory.

Early LCDs were notorious for “ghosting” during scrolling or animation, still showing traces of old images many tens of milliseconds after the image was changed, but significant progress has been made in the last two decades. The transition times for LCD pixels vary based on the start and end values being transitioned between, but a good panel today will have a switching time around ten milliseconds, and optimized displays for active 3D and gaming can have switching times less than half that.

Modern displays are also expected to perform a wide variety of processing on the incoming signal before they change the actual display elements. A typical Full HD display today will accept 720p or interlaced composite signals and convert them to the 1920×1080 physical pixels. 24 fps movie footage will be converted to 60 fps refresh rates. Stereoscopic input may be converted from side-by-side, top-down, or other formats to frame sequential for active displays, or interlaced for passive displays. Content protection may be applied. Many consumer oriented displays have started applying motion interpolation and other sophisticated algorithms that require multiple frames of buffering.

Some of these processing tasks could be handled by only buffering a single scan line, but some of them fundamentally need one or more full frames of buffering, and display vendors have tended to implement the general case without optimizing for the cases that could be done with low or no delay. Some consumer displays wind up buffering three or more frames internally, resulting in 50 milliseconds of latency even when the input data could have been fed directly into the display matrix.

Some less common display technologies have speed advantages over LCD panels; OLED pixels can have switching times well under a millisecond, and laser displays are as instantaneous as CRTs.

A subtle latency point is that most displays present an image incrementally as it is scanned out from the computer, which has the effect that the bottom of the screen changes 16 milliseconds later than the top of the screen on a 60 fps display. This is rarely a problem on a static display, but on a head mounted display it can cause the world to appear to shear left and right, or “waggle” as the head is rotated, because the source image was generated for an instant in time, but different parts are presented at different times. This effect is usually masked by switching times on LCD HMDs, but it is obvious with fast OLED HMDs.

Host processing

The classic processing model for a game or VR application is:

Read user input -> run simulation -> issue rendering commands -> graphics drawing -> wait for vsync -> scanout

I = Input sampling and dependent calculation
S = simulation / game execution
R = rendering engine
G = GPU drawing time
V = video scanout time

All latencies are based on a frame time of roughly 16 milliseconds, a progressively scanned display, and zero sensor and pixel latency.

If the performance demands of the application are well below what the system can provide, a straightforward implementation with no parallel overlap will usually provide fairly good latency values. However, if running synchronized to the video refresh, the minimum latency will still be 16 ms even if the system is infinitely fast. This rate feels good for most eye-hand tasks, but it is still a perceptible lag that can be felt in a head mounted display, or in the responsiveness of a mouse cursor.

Ample performance, vsync:
ISRG------------|VVVVVVVVVVVVVVVV|
.................. latency 16 – 32 milliseconds

Running without vsync on a very fast system will deliver better latency, but only over a fraction of the screen, and with visible tear lines. The impact of the tear lines are related to the disparity between the two frames that are being torn between, and the amount of time that the tear lines are visible. Tear lines look worse on a continuously illuminated LCD than on a CRT or laser projector, and worse on a 60 fps display than a 120 fps display. Somewhat counteracting that, slow switching LCD panels blur the impact of the tear line relative to the faster displays.

If enough frames were rendered such that each scan line had a unique image, the effect would be of a “rolling shutter”, rather than visible tear lines, and the image would feel continuous. Unfortunately, even rendering 1000 frames a second, giving approximately 15 bands on screen separated by tear lines, is still quite objectionable on fast switching displays, and few scenes are capable of being rendered at that rate, let alone 60x higher for a true rolling shutter on a 1080P display.

Ample performance, unsynchronized:
ISRG
VVVVV
..... latency 5 – 8 milliseconds at ~200 frames per second

In most cases, performance is a constant point of concern, and a parallel pipelined architecture is adopted to allow multiple processors to work in parallel instead of sequentially. Large command buffers on GPUs can buffer an entire frame of drawing commands, which allows them to overlap the work on the CPU, which generally gives a significant frame rate boost at the expense of added latency.

CPU:ISSSSSRRRRRR----|
GPU:                |GGGGGGGGGGG----|
VID:                |               |VVVVVVVVVVVVVVVV|
    .................................. latency 32 – 48 milliseconds

When the CPU load for the simulation and rendering no longer fit in a single frame, multiple CPU cores can be used in parallel to produce more frames. It is possible to reduce frame execution time without increasing latency in some cases, but the natural split of simulation and rendering has often been used to allow effective pipeline parallel operation. Work queue approaches buffered for maximum overlap can cause an additional frame of latency if they are on the critical user responsiveness path.

CPU1:ISSSSSSSS-------|
CPU2:                |RRRRRRRRR-------|
GPU :                |                |GGGGGGGGGG------|
VID :                |                |                |VVVVVVVVVVVVVVVV|
     .................................................... latency 48 – 64 milliseconds

Even if an application is running at a perfectly smooth 60 fps, it can still have host latencies of over 50 milliseconds, and an application targeting 30 fps could have twice that. Sensor and display latencies can add significant additional amounts on top of that, so the goal of 20 milliseconds motion-to-photons latency is challenging to achieve.

Latency Reduction Strategies

Prevent GPU buffering

The drive to win frame rate benchmark wars has led driver writers to aggressively buffer drawing commands, and there have even been cases where drivers ignored explicit calls to glFinish() in the name of improved “performance”. Today’s fence primitives do appear to be reliably observed for drawing primitives, but the semantics of buffer swaps are still worryingly imprecise. A recommended sequence of commands to synchronize with the vertical retrace and idle the GPU is:

SwapBuffers();
DrawTinyPrimitive();
InsertGPUFence();
BlockUntilFenceIsReached();

While this should always prevent excessive command buffering on any conformant driver, it could conceivably fail to provide an accurate vertical sync timing point if the driver was transparently implementing triple buffering.

To minimize the performance impact of synchronizing with the GPU, it is important to have sufficient work ready to send to the GPU immediately after the synchronization is performed. The details of exactly when the GPU can begin executing commands are platform specific, but execution can be explicitly kicked off with glFlush() or equivalent calls. If the code issuing drawing commands does not proceed fast enough, the GPU may complete all the work and go idle with a “pipeline bubble”. Because the CPU time to issue a drawing command may have little relation to the GPU time required to draw it, these pipeline bubbles may cause the GPU to take noticeably longer to draw the frame than if it were completely buffered. Ordering the drawing so that larger and slower operations happen first will provide a cushion, as will pushing as much preparatory work as possible before the synchronization point.

Run GPU with minimal buffering:
CPU1:ISSSSSSSS-------|
CPU2:                |RRRRRRRRR-------|
GPU :                |-GGGGGGGGGG-----|
VID :                |                |VVVVVVVVVVVVVVVV|
     ................................... latency 32 – 48 milliseconds

Tile based renderers, as are found in most mobile devices, inherently require a full scene of command buffering before they can generate their first tile of pixels, so synchronizing before issuing any commands will destroy far more overlap. In a modern rendering engine there may be multiple scene renders for each frame to handle shadows, reflections, and other effects, but increased latency is still a fundamental drawback of the technology.

High end, multiple GPU systems today are usually configured for AFR, or Alternate Frame Rendering, where each GPU is allowed to take twice as long to render a single frame, but the overall frame rate is maintained because there are two GPUs producing frames

Alternate Frame Rendering dual GPU:
CPU1:IOSSSSSSS-------|IOSSSSSSS-------|
CPU2:                |RRRRRRRRR-------|RRRRRRRRR-------|
GPU1:                | GGGGGGGGGGGGGGGGGGGGGGGG--------|
GPU2:                |                | GGGGGGGGGGGGGGGGGGGGGGG---------|
VID :                |                |                |VVVVVVVVVVVVVVVV|
     .................................................... latency 48 – 64 milliseconds

Similarly to the case with CPU workloads, it is possible to have two or more GPUs cooperate on a single frame in a way that delivers more work in a constant amount of time, but it increases complexity and generally delivers a lower total speedup.

An attractive direction for stereoscopic rendering is to have each GPU on a dual GPU system render one eye, which would deliver maximum performance and minimum latency, at the expense of requiring the application to maintain buffers across two independent rendering contexts.

The downside to preventing GPU buffering is that throughput performance may drop, resulting in more dropped frames under heavily loaded conditions.

Late frame scheduling

Much of the work in the simulation task does not depend directly on the user input, or would be insensitive to a frame of latency in it. If the user processing is done last, and the input is sampled just before it is needed, rather than stored off at the beginning of the frame, the total latency can be reduced.

It is very difficult to predict the time required for the general simulation work on the entire world, but the work just for the player’s view response to the sensor input can be made essentially deterministic. If this is split off from the main simulation task and delayed until shortly before the end of the frame, it can remove nearly a full frame of latency.

Late frame scheduling:
CPU1:SSSSSSSSS------I|
CPU2:                |RRRRRRRRR-------|
GPU :                |-GGGGGGGGGG-----|
VID :                |                |VVVVVVVVVVVVVVVV|
                    .................... latency 18 – 34 milliseconds

Adjusting the view is the most latency sensitive task; actions resulting from other user commands, like animating a weapon or interacting with other objects in the world, are generally insensitive to an additional frame of latency, and can be handled in the general simulation task the following frame.

The drawback to late frame scheduling is that it introduces a tight scheduling requirement that usually requires busy waiting to meet, wasting power. If your frame rate is determined by the video retrace rather than an arbitrary time slice, assistance from the graphics driver in accurately determining the current scanout position is helpful.

View bypass

An alternate way of accomplishing a similar, or slightly greater latency reduction Is to allow the rendering code to modify the parameters delivered to it by the game code, based on a newer sampling of user input.

At the simplest level, the user input can be used to calculate a delta from the previous sampling to the current one, which can be used to modify the view matrix that the game submitted to the rendering code.

Delta processing in this way is minimally intrusive, but there will often be situations where the user input should not affect the rendering, such as cinematic cut scenes or when the player has died. It can be argued that a game designed from scratch for virtual reality should avoid those situations, because a non-responsive view in a HMD is disorienting and unpleasant, but conventional game design has many such cases.

A binary flag could be provided to disable the bypass calculation, but it is useful to generalize such that the game provides an object or function with embedded state that produces rendering parameters from sensor input data instead of having the game provide the view parameters themselves. In addition to handling the trivial case of ignoring sensor input, the generator function can incorporate additional information such as a head/neck positioning model that modified position based on orientation, or lists of other models to be positioned relative to the updated view.

If the game and rendering code are running in parallel, it is important that the parameter generation function does not reference any game state to avoid race conditions.

View bypass:
CPU1:ISSSSSSSSS------|
CPU2:                |IRRRRRRRRR------|
GPU :                |--GGGGGGGGGG----|
VID :                |                |VVVVVVVVVVVVVVVV|
                      .................. latency 16 – 32 milliseconds

The input is only sampled once per frame, but it is simultaneously used by both the simulation task and the rendering task. Some input processing work is now duplicated by the simulation task and the render task, but it is generally minimal.

The latency for parameters produced by the generator function is now reduced, but other interactions with the world, like muzzle flashes and physics responses, remain at the same latency as the standard model.

A modified form of view bypass could allow tile based GPUs to achieve similar view latencies to non-tiled GPUs, or allow non-tiled GPUs to achieve 100% utilization without pipeline bubbles by the following steps:

Inhibit the execution of GPU commands, forcing them to be buffered. OpenGL has only the deprecated display list functionality to approximate this, but a control extension could be formulated.

All calculations that depend on the view matrix must reference it independently from a buffer object, rather than from inline parameters or as a composite model-view-projection (MVP) matrix.

After all commands have been issued and the next frame has started, sample the user input, run it through the parameter generator, and put the resulting view matrix into the buffer object for referencing by the draw commands.

Kick off the draw command execution.

Tiler optimized view bypass:
CPU1:ISSSSSSSSS------|
CPU2:                |IRRRRRRRRRR-----|I
GPU :                |                |-GGGGGGGGGG-----|
VID :                |                |                |VVVVVVVVVVVVVVVV|
                                       .................. latency 16 – 32 milliseconds

Any view frustum culling that was performed to avoid drawing some models may be invalid if the new view matrix has changed substantially enough from what was used during the rendering task. This can be mitigated at some performance cost by using a larger frustum field of view for culling, and hardware clip planes based on the culling frustum limits can be used to guarantee a clean edge if necessary. Occlusion errors from culling, where a bright object is seen that should have been occluded by an object that was incorrectly culled, are very distracting, but a temporary clean encroaching of black at a screen edge during rapid rotation is almost unnoticeable.

Time warping

If you had perfect knowledge of how long the rendering of a frame would take, some additional amount of latency could be saved by late frame scheduling the entire rendering task, but this is not practical due to the wide variability in frame rendering times.

Late frame input sampled view bypass:
CPU1:ISSSSSSSSS------|
CPU2:                |----IRRRRRRRRR--|
GPU :                |------GGGGGGGGGG|
VID :                |                |VVVVVVVVVVVVVVVV|
                          .............. latency 12 – 28 milliseconds

However, a post processing task on the rendered image can be counted on to complete in a fairly predictable amount of time, and can be late scheduled more easily. Any pixel on the screen, along with the associated depth buffer value, can be converted back to a world space position, which can be re-transformed to a different screen space pixel location for a modified set of view parameters.

After drawing a frame with the best information at your disposal, possibly with bypassed view parameters, instead of displaying it directly, fetch the latest user input, generate updated view parameters, and calculate a transformation that warps the rendered image into a position that approximates where it would be with the updated parameters. Using that transform, warp the rendered image into an updated form on screen that reflects the new input. If there are two dimensional overlays present on the screen that need to remain fixed, they must be drawn or composited in after the warp operation, to prevent them from incorrectly moving as the view parameters change.

Late frame scheduled time warp:
CPU1:ISSSSSSSSS------|
CPU2:                |RRRRRRRRRR----IR|
GPU :                |-GGGGGGGGGG----G|
VID :                |                |VVVVVVVVVVVVVVVV|
                                    .... latency 2 – 18 milliseconds

If the difference between the view parameters at the time of the scene rendering and the time of the final warp is only a change in direction, the warped image can be almost exactly correct within the limits of the image filtering. Effects that are calculated relative to the screen, like depth based fog (versus distance based fog) and billboard sprites will be slightly different, but not in a manner that is objectionable.

If the warp involves translation as well as direction changes, geometric silhouette edges begin to introduce artifacts where internal parallax would have revealed surfaces not visible in the original rendering. A scene with no silhouette edges, like the inside of a box, can be warped significant amounts and display only changes in texture density, but translation warping realistic scenes will result in smears or gaps along edges. In many cases these are difficult to notice, and they always disappear when motion stops, but first person view hands and weapons are a prominent case. This can be mitigated by limiting the amount of translation warp, compressing or making constant the depth range of the scene being warped to limit the dynamic separation, or rendering the disconnected near field objects as a separate plane, to be composited in after the warp.

If an image is being warped to a destination with the same field of view, most warps will leave some corners or edges of the new image undefined, because none of the source pixels are warped to their locations. This can be mitigated by rendering a larger field of view than the destination requires; but simply leaving unrendered pixels black is surprisingly unobtrusive, especially in a wide field of view HMD.

A forward warp, where source pixels are deposited in their new positions, offers the best accuracy for arbitrary transformations. At the limit, the frame buffer and depth buffer could be treated as a height field, but millions of half pixel sized triangles would have a severe performance cost. Using a grid of triangles at some fraction of the depth buffer resolution can bring the cost down to a very low level, and the trivial case of treating the rendered image as a single quad avoids all silhouette artifacts at the expense of incorrect pixel positions under translation.

Reverse warping, where the pixel in the source rendering is estimated based on the position in the warped image, can be more convenient because it is implemented completely in a fragment shader. It can produce identical results for simple direction changes, but additional artifacts near geometric boundaries are introduced if per-pixel depth information is considered, unless considerable effort is expended to search a neighborhood for the best source pixel.

If desired, it is straightforward to incorporate motion blur in a reverse mapping by taking several samples along the line from the pixel being warped to the transformed position in the source image.

Reverse mapping also allows the possibility of modifying the warp through the video scanout. The view parameters can be predicted ahead in time to when the scanout will read the bottom row of pixels, which can be used to generate a second warp matrix. The warp to be applied can be interpolated between the two of them based on the pixel row being processed. This can correct for the “waggle” effect on a progressively scanned head mounted display, where the 16 millisecond difference in time between the display showing the top line and bottom line results in a perceived shearing of the world under rapid rotation on fast switching displays.

Continuously updated time warping

If the necessary feedback and scheduling mechanisms are available, instead of predicting what the warp transformation should be at the bottom of the frame and warping the entire screen at once, the warp to screen can be done incrementally while continuously updating the warp matrix as new input arrives.

Continuous time warp:
CPU1:ISSSSSSSSS------|
CPU2:                |RRRRRRRRRRR-----|
GPU :                |-GGGGGGGGGGGG---|
WARP:                |               W| W W W W W W W W|
VID :                |                |VVVVVVVVVVVVVVVV|
                                     ... latency 2 – 3 milliseconds for 500hz sensor updates

The ideal interface for doing this would be some form of “scanout shader” that would be called “just in time” for the video display. Several video game systems like the Atari 2600, Jaguar, and Nintendo DS have had buffers ranging from half a scan line to several scan lines that were filled up in this manner.

Without new hardware support, it is still possible to incrementally perform the warping directly to the front buffer being scanned for video, and not perform a swap buffers operation at all.

A CPU core could be dedicated to the task of warping scan lines at roughly the speed they are consumed by the video output, updating the time warp matrix each scan line to blend in the most recently arrived sensor information.

GPUs can perform the time warping operation much more efficiently than a conventional CPU can, but the GPU will be busy drawing the next frame during video scanout, and GPU drawing operations cannot currently be scheduled with high precision due to the difficulty of task switching the deep pipelines and extensive context state. However, modern GPUs are beginning to allow compute tasks to run in parallel with graphics operations, which may allow a fraction of a GPU to be dedicated to performing the warp operations as a shared parameter buffer is updated by the CPU.

Discussion

View bypass and time warping are complementary techniques that can be applied independently or together. Time warping can warp from a source image at an arbitrary view time / location to any other one, but artifacts from internal parallax and screen edge clamping are reduced by using the most recent source image possible, which view bypass rendering helps provide.

Actions that require simulation state changes, like flipping a switch or firing a weapon, still need to go through the full pipeline for 32 – 48 milliseconds of latency based on what scan line the result winds up displaying on the screen, and translational information may not be completely faithfully represented below the 16 – 32 milliseconds of the view bypass rendering, but the critical head orientation feedback can be provided in 2 – 18 milliseconds on a 60 hz display. In conjunction with low latency sensors and displays, this will generally be perceived as immediate. Continuous time warping opens up the possibility of latencies below 3 milliseconds, which may cross largely unexplored thresholds in human / computer interactivity.

Conventional computer interfaces are generally not as latency demanding as virtual reality, but sensitive users can tell the difference in mouse response down to the same 20 milliseconds or so, making it worthwhile to apply these techniques even in applications without a VR focus.

A particularly interesting application is in “cloud gaming”, where a simple client appliance or application forwards control information to a remote server, which streams back real time video of the game. This offers significant convenience benefits for users, but the inherent network and compression latencies makes it a lower quality experience for action oriented titles. View bypass and time warping can both be performed on the server, regaining a substantial fraction of the latency imposed by the network. If the cloud gaming client was made more sophisticated, time warping could be performed locally, which could theoretically reduce the latency to the same levels as local applications, but it would probably be prudent to restrict the total amount of time warping to perhaps 30 or 40 milliseconds to limit the distance from the source images.

Acknowledgements

Zenimax for allowing me to publish this openly.

Hillcrest Labs for inertial sensors and experimental firmware.

Emagin for access to OLED displays.

Oculus for a prototype Rift HMD.

Nvidia for an experimental driver with access to the current scan line number.

show more
About danluu.com
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-09-01 00:00:00 | Created: 2026-07-23 05:18:40

About The Blog

This started out as a way to jot down thoughts on areas that seem interesting but underappreciated. Since then, this site has grown to the point where it gets millions of hits a month and I see that it's commonly cited by professors in their courses and on stackoverflow.

That's flattering, but more than anything else, I view that as a sign there's a desperate shortage of understandable explanation of technical topics. There's nothing here that most of my co-workers don't know (with the exception of maybe three or four posts where I propose novel ideas). It's just that they don't blog and I do. I'm not going to try to convince you to start writing a blog, since that has to be something you want to do, but I will point out that there's a large gap that's waiting to be filled by your knowledge. When I started writing this blog, I figured almost no one would ever read it; sure Joel Spolsky and Steve Yegge created widely read blogs, but that was back when almost no one was blogging. Now that there are millions of blogs, there's just no way to start a new blog and get noticed. Turns out that's not true.

This site also archives a few things that have fallen off the internet, like this history of subspace, the 90s video game, the su3su2u1 introduction to physics, the su3su2u1 review of hpmor, Dan Weinreb's history of Symbolics and Lisp machines, this discussion of open vs. closed social networks, this discussion about the differences between SV and Boston, and Stanford and MIT, the comp.programming.threads FAQ, and this presentation about Microsoft culture from 2000.

P.S. If you enjoy this blog, you'd probably enjoy RC, which I've heard called "nerd camp for programmers".

show more
Verilog is weird
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-09-07 00:00:00 | Created: 2026-07-23 05:18:40

Verilog is the most commonly used language for hardware design in America (VHDL is more common in Europe). Too bad it's so baroque. If you ever browse the Verilog questions on Stack Overflow, you'll find a large number of questions, usually downvoted, asking “why doesn't my code work?”, with code that's not just a little off, but completely wrong.

6 questions, all but one with negative score

Lets look at an example: “Idea is to store value of counter at the time of reset . . . I get DRC violations and the memory, bufreadaddr, bufreadval are all optimized out.”

always @(negedge reset or posedge clk) begin
  if (reset == 0) begin
    d_out <= 16'h0000;
    d_out_mem[resetcount] <= d_out;
    laststoredvalue <= d_out;
  end else begin
    d_out <= d_out + 1'b1;
  end
end

always @(bufreadaddr)
  bufreadval = d_out_mem[bufreadaddr];

We want a counter that keeps track of how many cycles it's been since reset, and we want to store that value in an array-like structure that's indexed by resetcount. If you've read a bit on semantics of Verilog, this is a perfectly natural way to solve the problem. Our poster knows enough about Verilog to use ‘<=' in state elements, so that all of the elements are updated at the same time. Every time there's a clock edge, we'll increment d_out. When reset is 0, we'll store that value and reset d_out. What could possibly go wrong?

The problem is that Verilog was originally designed as a language to describe simulations, so it has constructs to describe arbitrary interactions between events. When X transitions from 0 to 1, do Y. Great! Sounds easy enough. But then someone had the bright idea of using Verilog to represent hardware. The vast majority of statements you could write down don't translate into any meaningful hardware. Your synthesis tool, which translates from Verilog to hardware will helpfully pattern match to the closest available thing, or produce nothing, if you write down something untranslatable. If you're lucky, you might get some warnings.

Looking at the code above, the synthesis tool will see that there's something called dout which should be a clocked element that's set to something when it shouldn't be reset, and is otherwise asynchronously reset. That's a legit hardware construct, so it will produce an N-bit flip-flop and some logic to make it a counter that gets reset to 0. BTW, this paragraph used to contain a link to http://en.wikipedia.org/wiki/Flip-flop(electronics), but ever since I switched to Hugo, my links to URLs with parens in them are broken, so maybe try copy+pasting that URL into your browser window if you want know what a flip-flop is.

Now, what about the value we're supposed to store on reset? Well, the synthesis tool will see that it's inside a block that's clocked. But it's not supposed to do anything when the clock is active; only when reset is asserted. That's pretty unusual. What's going to happen? Well, that depends on which version of which synthesis tool you're using, and how the programmers of that tool decided to implement undefined behavior.

And then there's the block that's supposed to read out the stored value. It looks like the intent is to create a 64:1 MUX. Putting aside the cycle time issues you'll get with such a wide MUX, the block isn't clocked, so the synthesis tool will have to infer some sort of combinational logic. But, the output is only supposed to change if bufreadaddr changes, and not if d_out_mem changes. It's quite easy to describe that in our simulation language, the but the synthesis tool is going to produce something that is definitely not what the user wants here. Not to mention that laststoredvalue isn't meaningfully connected to bufreadvalue.

How is it possible that a reasonable description of something in Verilog turns into something completely wrong in hardware? You can think of hardware as some state, with pure functions connecting the state elements. This makes it natural to think about modeling hardware in a functional programming language. Another natural way to think about it would be with OO. Classes describe how the hardware works. Instances of the class are actual hardware that will get put onto the chip. Yet another natural way to describe things would be declaratively, where you write down constraints the hardware must obey, and the synthesis tool outputs something that meets those constraints.

Verilog does none of these things. To write Verilog that will produce correct hardware, you have to first picture the hardware you want to produce. Then, you have to figure out how to describe that in this weird C-like simulation language. That will then get synthesized into something like what you were imaging in the first step.

As a software engineer, how would you feel if 99% of valid Java code ended up being translated to something that produced random results, even though tests pass on the untranslated Java code? And, by the way, to run tests on the translated Java code you have to go through a multi-day long compilation process, after which your tests will run 200 million times slower than code runs in production. If you're thinking of testing on some sandboxed production machines, sure, go ahead, but it costs 8 figures to push something to any number of your production machines, and it takes 3 months. But, don't worry, you can run the untranslated code only 2 million times slower than in production 1. People used to statically typed languages often complain that you get run-time errors about things that would be trivial to statically check in a language with stronger types. We hardware folks are so used to the vast majority of legal Verilog constructs producing unsynthesizable garbage that we don't find it the least bit surprising that you not only do you not get compile-time errors, you don't even get run-time errors, from writing naive Verilog code.

Old school hardware engineers will tell you that it's fine. It's fine that the language is so counter-intuitive that almost all people who initially approach Verilog write code that's not just wrong but nonsensical. "All you have to do is figure out the design and then translate it to Verilog". They'll tell you that it's totally fine that the mental model you have of what's going on is basically unrelated to the constructs the language provides, and that they never make errors now that they're experienced, much like some experienced C programmers will erronously tell you that they never have security related buffer overflows or double frees or memory leaks now that they're experienced. It reminds me of talking to assembly programmers who tell me that assembly is as productive as a high level language once you get your functions written. Programmers who haven't talked to old school assembly programmers will think I'm making that up, but I know a number of people who still maintain that assembly is as productive as any high level langauge out there. But people like that are rare and becoming rarer. With hardware, we train up a new generation of people who think that Verilog is as productive as any language could be every few years!

I won't even get into how Verilog is so inexpressive that many companies use an ad hoc tool to embed a scripting language in Verilog or generate Verilog from a scripting language.

There have been a number of attempts to do better than jamming an ad hoc scripting language into Verilog, but they've all fizzled out. As a functional language that's easy to add syntax to, Haskell is a natural choice for Verilog code generation; it spawned ForSyDe, Hydra, Lava, HHDL, and Bluespec. But adoption of ForSyDe, Hydra, Lava, and HHDL is pretty much zero, not because of deficiencies in the language, but because it's politically difficult to get people to use a Haskell based language. Bluespec has done better, but they've done it by making their language look C-like, scrapping the original Haskell syntax and introducing Bluespec SystemVerilog and Bluespec SystemC. The aversion to Haskell is so severe that when we discussed a hardware style at my new gig, one person suggested banning any Haskell based solution, even though Bluespec has been used to good effect in a couple projects within the company.

Scala based solutions look more promising, not for any technical reason, but because Scala is less scary. Scala has managed to bring the modern world (in terms of type systems) to more programmers than ML, Ocaml, Haskell, Agda, etc., combined. Perhaps the same will be true in the hardware world. Chisel is interesting. Like Bluespec, it simulates much more quickly than Verilog, and unsynthesizable representations are syntax errors. It's not as high level, but it's the only hardware description language with a modern type system that I've been able to discuss with hardware folks without people objecting that Haskell is a bad idea.

Commercial vendors are mostly moving in the other direction because C-like languages make people feel all warm and fuzzy. A number of them are pushing high-level hardware synthesis from SystemC, or even straight C or C++. These solutions are also politically difficult to sell, but this time it's the history of the industry, and not the language. Vendors pushing high-level synthesis have a decades long track record of overpromising and underdelivering. I've lost track of the number of times I've heard people dismiss modern offerings with “Why should we believe that this they're for real this time?”

What's the future? Locally, I've managed to convince a couple of people on my team that Chisel is worth looking at. At the moment, none of the Haskell based solutions are even on the table. I'm open to suggestions.

CPU internals series

P.S. Dear hardware folks, sorry for oversimplifying so much. I started writing footnotes explaining everything I was glossing over until I realized that my footnotes were longer than the post. The culled footnotes may make it into their own blog posts some day. A very long footnote that I'll briefly summarize is that semantically correct Verilog simulation is inherently slower than something like Bluespec or Chisel because of the complications involved with the event model. EDA vendors have managed to get decent performance out of Verilog, but only by hiring large teams of the best simulation people in the world to hammer at the problem, the same way JavaScript is fast not because of any property of the language, but because there are amazing people working on the VM. It should tell you something when a tiny team working on a shoestring grant-funded budget can produce a language and simulation infrastructure that smokes existing tools.

You may wonder why I didn't mention linters. They're a great idea and for reasons I don't understand, two of the three companies I've done hardware development for haven't used linters. If you ask around, everyone will agree that they're a good idea, but even though a linter will run in the thousands to tens of thousands of dollars range, and engineers run in hundreds of thousands of dollars range, it hasn't been politically possible to get a linter even on multi-person teams that have access to tools that cost tens or hundreds of thousands of dollars per license per year. Even though linters are a no-brainer, companies that spend millions to tens of millions a year on hardware development often don't use them, and good SystemVerilog linters are all out of the price range of the people who are asking StackOverflow questions that get downvoted to oblivion.


  1. Approximate numbers from the last chip I worked on. We had licenses for both major commercial simulators, and we were lucky to get 500Hz, pre-synthesis, on the faster of the two, for a chip that ran at 2GHz in silicon. Don't even get me started on open source simulators. The speed is at least 10x better for most ASIC work. Also, you can probably do synthesis much faster if you don't have timing / parasitic extraction baked into the process. [return]
show more
Writing safe Verilog
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-09-15 00:00:00 | Created: 2026-07-23 05:18:40

Troll? That's how people write Verilog1. At my old company, we had a team of formal methods PhD's who wrote a linter that typechecked our code, based on our naming convention. For our chip (which was small for a CPU), building a model (compiling) took about five minutes, running a single short test took ten to fifteen minutes, and long tests took CPU months. The value of a linter that can run in seconds should be obvious, not even considering the fact that it can take hours of tracing through waveforms to find out why a test failed2.

Lets look at some of the most commonly used naming conventions.

Pipeline stage

When you pipeline hardware, you end up with many versions of the same signal, one for each stage of the pipeline the signal traverses. Even without static checks, you'll want some simple way to differentiate between these, so you might name them foo_s1, foo_s2, and foo_s3, indicating that they originate in the first, second, and third stages, respectively. In any particular stage, a signal is most likely to interact with other signals in the same stage; it's often a mistake when logic from other stages is accessed. There are reasons to access signals from other stages, like bypass paths and control logic that looks at multiple stages, but logic that stays contained within a stage is common enough that it's not too tedious to either “cast” or add a comment that disables the check, when looking at signals from other stages.

Clock domain

Accessing a signal in a different clock domain without synchronization is like accessing a data structure from multiple threads without synchronization. Sort of. But worse. Much worse. Driving combinational logic from a metastable state (where the signal is sitting between a 0 and 1) can burn a massive amount of power3. Here, I'm not just talking about being inefficient. If you took a high-power chip from the late 90s and removed the heat sink, it would melt itself into the socket, even under normal operation. Modern chips have such a high maximum power possible power consumption that the chips would self destruct if you disabled the thermal regulation, even with the heat sink. Logic that's floating at an intermediate value not only uses a lot of power, it bypasses a chip's usual ability to reduce power by slowing down the clock4. Using cross clock domain signals without synchronization is a bad idea, unless you like random errors, high power dissipation, and the occasional literal meltdown.

Module / Region

In high speed designs, it's an error to use a signal that's sourced from another module without registering it first. This will insidiously sneak through simulation; you'll only notice when you look at the timing report. On the last chip I worked on, it took about two days to generate a timing report0. If you accidentally reference a signal from a distant module, not only will you not meet your timing budget for that path, the synthesis tool will allocate resources to try to make that path faster, which will slow down everything else5, making the entire timing report worthless6.

PL Trolling

I'd been feeling naked at my new gig, coding Verilog without any sort of static checking. I put off writing my own checker, because static analysis is one of those scary things you need a PhD to do, right? And writing a parser for SystemVerilog is a ridiculously large task7. But, it turns out that don't need much of a parser, and all the things I've talked about are simple enough that half an hour after starting, I had a tool that found seven bugs, with only two false positives. I expect we'll have 4x as much code by the time we're done, so that's 28 bugs from half an hour of work, not even considering the fact that two of the bugs were in heavily used macros.

I think I'm done for the day, but there are plenty of other easy things to check that will certainly find bugs (e.g, checking for regs/logic that are declared or assigned, but not used). Whenever I feel like tackling a self-contained challenge, there are plenty of not-so-easy things, too (e.g., checking if things aren't clock gated or power gated when they should be, which isn't hard to do statistically, but is non-trivial statically).

Huh. That wasn't so bad. I've now graduated to junior PL troll.


  1. Well, people usually use suffixes as well as prefixes. [return]
  2. You should, of course, write your own tool to script interaction with your waveform view because waveform viewers have such poor interfaces, but that's whole ‘nother blog post. [return]
  3. In static CMOS there's a network of transistors between power and output, and a dual network between ground and output. As a first-order approximation, only one of the two networks should be on at a time, except when switching, which is why switching logic gates use power than unchanging gates -- in addition to the power used to discharge the capacitance that the output is driving, there is, briefly, a direct connection from power to ground. If you get stuck into a half-on state, there's a constant connection from power to ground. [return]
  4. In theory, power gating could help, but you can't just power gate some arbitrary part of the chip that's too hot. [return]
  5. There are a number of reasons that this completely destroys the timing report. First, for any high-speed design, there's not enough fast (wide) interconnect to go around. Gates are at the bottom, and wires sit above them. Wires get wider and faster in higher layers, but there's congestion getting to and from the fast wires, and relatively few of them. There are so few of them that people pre-plan where modules should be placed in order to have enough fast interconnect to meet timing demands. If you steal some fast wires to make some slow path fast, anything relying on having a fast path through that region is hosed. Second, the synthesis tool tries to place sources near sinks, to reduce both congestion and delay. If you place a sink on a net that's very far from the rest of the sinks, the source will migrate halfway in between, to try to match the demands of all the sinks. This is recursively bad, and will pull all the second order sources away from their optimal location, and so on and so forth. [return]
  6. With some tools, you can have them avoid optimizing paths that fail timing by more than a certain margin, but there's still always some window where a bad path will destroy your entire timing report, and it's often the case that there are real critical paths that need all the resources the synthesis tool can throw at it to make it across the chip in time. [return]
  7. The SV standard is 1300 pages long, vs 800 for C++, 500 for C, 300 for Java, and 30 for Erlang. [return]
show more
Randomize HN
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-10-04 00:00:00 | Created: 2026-07-23 05:18:40

You ever notice that there's this funny threshold for getting to the front page on sites like HN? The exact threshold varies depending on how much traffic there is, but, for articles that aren't wildly popular, there's this moment when the article is at N-1 votes. There is, perhaps, a 60% chance that the vote will come and the article will get pushed to the front page, where it will receive a slew of votes. There is, maybe, a 40% chance it will never get the vote that pushes it to the front page, causing it to languish in obscurity forever.

It's non-optimal that an article that will receive 50 votes in expectation has a 60% chance of getting 100+ votes, and a 40% chance of getting 2 votes. Ideally, each article would always get its expected number of votes and stay on the front page for the expected number of time, giving readers exposure to the article in proportion to its popularity. Instead, by random happenstance, plenty of interesting content never makes it the front page, and as a result, the content that does make it gets a higher than optimal level of exposure.

You also see the same problem, with the sign bit flipped, on low traffic sites that push things to the front page the moment they're posted, like lobste.rs and the smaller sub-reddits: they displace links that most people would be interested in by putting links that almost no one cares about on the front page just so that the few things people do care about get enough exposure to be upvoted. On reddit, users "fix" this problem by heavily downvoting most submissions, pushing them off the front page, resulting in a problem that's fundamentally the same as the problem HN has.

Instead of implementing some simple and easy to optimize, sites pile on ad hoc rules. Reddit implemented the rising page, but it fails to solve the problem. On low-traffic subreddits, like r/programming the threshold is so high that it's almost always empty. On high-traffic sub-reddits, anything that's upvoted enough to make it to the rising page is already wildly successful, and whether or not an article becomes successful is heavily dependent on whether or not the first couple voters happen to be people who upvote the post instead of downvoting it, i.e., the problem of getting onto the rising page is no different than the problem of getting to the top normally.

HN tries to solve the problem by manually penalizing certain domains and keywords. That doesn't solve the problem for the 95% of posts that aren't penalized. For posts that don't make it to the front page, the obvious workaround is to delete and re-submit your post if it doesn't make the front page the first time around, but that's now a ban worthy offense. Of course, people are working around that, and HN has a workaround for the workaround, and so on. It's endless. That's the problem with "simple" ad hoc solutions.

There's an easy fix, but it's counter-intuitive. By adding a small amount of random noise to the rank of an article, we can smooth out the discontinuity between making it onto the front page and languishing in obscurity. The math is simple, but the intuition is even simpler1. Imagine a vastly oversimplified model where, for each article, every reader upvotes with a fixed probability and the front page gets many more eyeballs than the new page. The result follows. If you like, you can work through the exercise with a more realistic model, but the result is the same2.

Adding noise to smooth out a discontinuity is a common trick when you can settle for an approximate result. I recently employed it to work around the classic floating point problem, where adding a tiny number to a large number results in no change, which is problem when adding many small numbers to some large numbers3. For a simple example of applying this, consider keeping a reduced precision counter that uses loglog(n) bits to store the value. Let countVal(x) = 2^x and inc(x) = if (rand(2^x) == 0) x++4. Like understanding when to apply Taylor series, this is a simple trick that people are often impressed by if they haven't seen it before5.

Update: HN tried this! Dan Gackle tells me that it didn't work very well (it resulted in a lot of low quality junk briefly hitting the front page and then disappearing. I think that might be fixable by tweaking some parameters, but the solution that HN settled on, having a human (or multiple humans) put submissions that are deemed to be good or interesting into a "second chance queue" that boosts the submission onto the front page, works better than an a simple randomized algorithm with no direct human input could with any amount of parameter tweaking. I think this is also true of moderation, where the "new" dang/sctb moderation regime has resulted in a marked increase in comment quality, probably better than anything that could be done with an automated ML-based solution today — Google and FB have some of the most advanced automated systems in the world, and the quality of the result is much worse than what we see on HN.

Also, at the time this post was written (2013), the threshold to get onto the front page was often 2-3 votes, making the marginal impact of a random passerby who happens to like a submission checking the new page very large. Even during off peak times now (in 2019), the threshold seems to be much higher, reducing the amount of randomness. Additionally, the rise in the popularity of HN increased the sheer volume of low quality content that languishes on the new page, which would reduce the exposure that any particular "good" submisison would get if it were among the 30 items on the new page that would randomly get boosted onto the front page. That doesn't mean there aren't still problems with the current system: most people seem to upvote and comment based on the title of the article and not the content (to check this, read the comments of articles that are mistitled before someone calls this out for a partiular post — it's generally quite clear that most commenters haven't even skimmed the article, let alone read it), but that's a topic for a different post.


  1. Another way to look at it is that it's A/B testing for upvotes (though, to be pedantic it's actually closer to multi-armed bandit). Another is that the distribution of people reading the front page and the new page aren't the same, and randomizing the front page prevents the clique that reads the new page from having undue influence. [return]
  2. If you want to do the exercise yourself, pg once said the formula for HN is: (votes - 1) / (time + 2)^1.5. It's possible the power of the denominator has been tweaked, but as long as it's greater than 1.0, you'll a reasonable result. [return]
  3. Kahan summation wasn't sufficient, for the fundamental same reason it won't work for the simplified example I gave above. [return]
  4. Assume we use a rand function that returns a non-negative integer between 0 and n-1, inclusive. With x = 0, we start counting from 1, as God intended. inc(0) will definitely increment, so we'll increment and correctly count to countVal(1) = 2^1 = 2. Next, we'll increment with probability ½; we'll have to increment twice in expectation to increase x. That works out perfectly because countVal(2) = 2^2 = 4, so we want to increment twice before increasing x. Then we'll increment with probability ¼, and so on and so forth. [return]
  5. See Mitzenmacher for a good introduction to randomized algorithms that also has an explanation of all the math you need to know. If you already apply Chernoff bounds in your sleep, and want something more in-depth, Motwani & Raghavan is awesome. [return]
show more
How to discourage open source contributions
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-10-27 00:00:00 | Created: 2026-07-23 05:18:40

What's the first thing you do when you find a bug or see a missing feature in an open source project? Check out the project page and submit a patch!

Send us a pull request! (116 open pull requests)

Oh. Maybe their message is so encouraging that they get hundreds of pull requests a week, and the backlog isn't that bad.

Multiple people ask why this bug fix is being ignored. No response.

Maybe not. Giant sucker than I am, I submitted a pull request even after seeing that. All things considered, I should consider myself lucky that it's possible to submit pull requests at all. If I'm really lucky, maybe they'll get around to looking at it one day.

I don't mean to pick on this particular project. I can understand how this happens. You're a dev who can merge pull requests, but you're not in charge of triaging bugs and pull requests; you have a day job, projects that you own, and a life outside of coding. Maybe you take a look at the repo every once in a while, merge in good pull requests, and make comments on the ones that need more work, but you don't look at all 116 open pull requests; who has that kind of time?

This behavior, eminently reasonable on the part of any individual, results in a systemic failure, a tax on new open source contributors. I often get asked how to get started with open source. It's easy for me to forget that getting started can be hard because the first projects I contributed to have a response time measured in hours for issues and pull requests1. But a lot of people have experiences which aren't so nice. They contribute a few patches to a couple projects that get ignored, and have no idea where to go from there. It doesn't take egregious individual behavior to create a hostile environment.

That's kept me from contributing to some projects. At my last job, I worked on making a well-known open source project production quality, fixing hundreds of bugs over the course of a couple months. When I had some time, I looked into pushing the changes back to the open source community. But when I looked at the mailing list for the project, I saw a wasteland of good patches that were completely ignored, where the submitter would ping the list a couple times and then give up. Did it seem worth spending a week to disentangle our IP from the project in order to submit a set of patches that would, in all likelihood, get ignored? No.

If you have commit access to a project that has this problem, please own the process for incoming pull requests (or don't ask for pull requests in your repo description). It doesn't have to permanent; just until you have a system in place2. Not only will you get more contributors to your project, you'll help break down one barrier to becoming an open source contributor.

joewiz replies to an month old comment. Asks for review months later. No reply.

For an update on the repo featured in this post, check out this response to a breaking change.


  1. Props to OpenBlas, Rust, jslinux-deobfuscated, and np for being incredibly friendly to new contributors. [return]
  2. I don't mean to imply that this is trivial. It can be hard, if your project doesn't have an accepting culture, but there are popular, high traffic projects that manage to do it. If all else fails, you can always try the pull request hack. [return]
show more
Why hardware development is hard
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-11-10 00:00:00 | Created: 2026-07-23 05:18:40

In CPU design, most successful teams have a fairly long lineage and rely heavily on experienced engineers. When we look at CPU startups, teams that have a successful exist often have a core team that's been together for decades. For example, PA Semi's acquisition by Apple was a moderately successful exit, but where did that team come from? They were the SiByte team, which left after SiByte was acquired by Broadcom, and SiByte was composed of many people from DEC who had been working together for over a decade. My old company was similar: an IBM fellow collected the best people he worked with at IBM who was a very early Dell employee and then exec (back when Dell still did interesting design work), then split off to create a chip startup. There have been quite a few CPU startups that have raised tens to hundreds of millions and leaned heavily on inexperienced labor; fresh PhDs and hardware engineers with only a few years of experience. Every single such startup I know of failed1.

This is in stark contrast to software startups, where it's common to see successful startups founded by people who are just out of school (or who dropped out of school). Why should microprocessors be any different? It's unheard of for a new, young, team to succeed at making a high-performance microprocessor, although this hasn't stopped people from funding these efforts.

In software, it's common to hear about disdain for experience, such as Zuckerberg's comment, "I want to stress the importance of being young and technical, Young people are just smarter.". Even when people don't explicitly devalue experience, they often don't value it either. As of this writing, Joel Spolsky's ”Smart and gets things done” is probably the most influential piece of writing on software hiring. Note that it doesn't say "smart, experienced, and gets things done.". Just "smart and gets things done" appears to be enough, no experience required. If you lean more towards the Paul Graham camp than the Joel Spolsky camp, there will be a lot of differences in how you hire, but Paul's advice is the same in that experience doesn't rank as one of his most important criteria, except as a diss.

Let's say you wanted to hire a plumber or a carptener, what would you choose? "Smart and gets things done" or "experienced and effective"? Ceteris paribus, I'll go for "experienced and effective", doubly so if it's an emergency.

Physical work isn't the kind of thing you can derive from first principles, no matter how smart you are. Consider South Korea after WWII. Its GDP per capita was lower than Ghana, Kenya, and just barely above the Congo. For various reasons, the new regime didn't have to deal with legacy institutions; and they wanted Korea to become a first-world nation.

The story I've heard is that the government started by subsidizing concrete. After many years making concrete, they wanted to move up the chain and start more complex manufacturing. They eventually got to building ships, because shipping was a critical part of the export economy they wanted to create.

They pulled some of their best business people who had learned skills like management and operations in other manufacturing. Those people knew they didn't have the expertise to build ships themselves, so they contracted it out. They made the choice to work with Scottish firms, because Scotland has a long history of shipbuilding. Makes sense, right?

It didn't work. For historical and geographic reasons, Scotland's shipyards weren't full-sized; they built their ships in two halves and then assembled them. Worked fine for them, because they'd be doing it at scale since the 1800s, and had world renowned expertise by the 1900s. But when the unpracticed Koreans tried to build ships using Scottish plans and detailed step-by-step directions, the result was two ship halves that didn't quite fit together and sunk when assembled.

The Koreans eventually managed to start a shipbuilding industry by hiring foreign companies to come and build ships locally, showing people how it's done. And it took decades to get what we would consider basic manufacturing working smoothly, even though one might think that all of the requisite knowledge existed in books, was taught in university courses, and could be had from experts for a small fee. Now, their manufacturing industries are world class, e.g., according to Consumer Reports, Hyundai and Kia produce reliable cars. Going from producing unreliable econoboxes to reliable cars you can buy took over a decade, like it did for Toyota when they did it decades earlier. If there's a shortcut to quality other than hiring a lot of people who've done it before, no one's discovered it yet.

Today, any programmer can take Geoffrey Hinton's course on neural networks and deep learning, and start applying state of the art machine learning techniques. In software land, you can fix minor bugs in real time. If it takes a whole day to run your regression test suite, you consider yourself lucky because it means you're in one of the few environments that takes testing seriously. If the architecture is fundamentally flawed, you pull out your copy of Feathers' “Working Effectively with Legacy Code” and repeatedly apply fixes.

This isn't to say that software isn't hard, but there are a lot of valueable problems that don't need a decade of hard-won experience to attack. But if you want to build a ship, and you "only" have a decade of experience with carpentry, milling, metalworking, etc., well, good luck. You're going to need it. With a large ship, “minor” fixes can take days or weeks, and a fundamental flaw means that your ship sinks and you've lost half a year of work and tens of millions of dollars. By the time you get to something with the complexity of a modern high-performance microprocessor, a minor bug discovered in production costs three months and millions of dollars. A fundamental flaw in the architecture will cost you five years and hundreds of millions of dollars2.

Physical mistakes are costly. There's no undo and editing isn't simply a matter of pressing some keys; changes consume real, physical resources. You need enough wisdom and experience to avoid common mistakes entirely – especially the ones that can't be fixed.

Thanks to Sophia Wisdom for comments/corrections/discussion.

CPU internals series

2021 comments

In retrospect, I think that I was too optimistic about software in this post. If we're talking about product-market fit and success, I don't think the attitude in the post is wrong and people with little to no experience often do create hits. But now that I've been in the industry for a while and talked to numerous people about infra at various startups as well as large companies, I think creating high quality software infra requires no less experience than creating high quality physical items. Companies that decided this wasn't the case and hire a bunch of smart folks from top schools to build their infra have ended up with low quality, unreliable, expensive, and difficult to operate infrastructure. It just turns out that, if you have very good product-market fit, you don't need your infra to work. Your company can survive and even thrive while having infra that has 2 9s of uptime and costs an order of magnitude more than your competitor's infra or if your product's architecture means that it can't possibly work correctly. You'll make less money than you would've otherwise, but the high order bits are all on the product side. If you contrast that chip companies with inexperienced engineers that didn't produce a working product, well, you can't really sell a product that doesn't work even if you try. If you get very lucky, like if you happened to start deep learning chip company at the right time, you might get big company to acquire your non-working product. But, it's much harder to get an exit like that for a microprocessor.


  1. Comparing my old company to another x86 startup founded within the year is instructive. Both started at around the same time. Both had great teams of smart people. Our competitor even had famous software and business people on their side. But it's notable that their hardware implementers weren't a core team of multi-decade industry veterans who had worked together before. It took us about two years to get a working x86 chip, on top of $15M in funding. Our goal was to produce a low-cost chip and we nailed it. It took them five years, with over $250M in funding. Their original goal was to produce a high performance low-power processor, but they missed their performance target so badly that they were forced into the low-cost space. They ended up with worse performance than us, with a chip was 50% bigger (and hence, cost more than 50% more to produce) using team four times our size. They eventually went under, because there's no way they could survive with 4x our burn rate and weaker performance. But, not before burning through $969M in funding (including $230M from patent lawsuits). [return]
  2. A funny side effect of the importance of experience is that age discrimination doesn't affect the areas I've worked in. At 30, I'm bizarrely young for someone who's done microprocessor design. The core folks at my old place were in their 60s. They'd picked up some younger folks along the way, but 30? Freakishly young. People are much younger at the new gig: I'm surrounded by ex-supercomputer folks from Cray and SGI, who are barely pushing 50, along with a couple kids from Synplify and DESRES who, at 40, are unusually young. Not all hardware folks are that old. In another arm of the company, there are folks who grew up in the FPGA world, which is a lot more forgiving. In that group, I think I met someone who's only a few years older than me. Kidding aside, you'll see younger folks doing RTL design on complex projects at large companies that are willing to spend a decade mentoring folks. But, at startups and on small hardware teams that move fast, it's rare to hire someone into design who doesn't have a decade of experience.

    There's a crowd that's even younger than the FPGA folks, even younger than me, working on Arduinos and microcontrollers, doing hobbyist electronics and consumer products. I'm genuinely curious how many of those folks will decide to work on large-scale systems design. In one sense, it's inevitable, as the area matures, and solutions become more complex. The other sense is what I'm curious about: will the hardware renaissance spark an interest in supercomputers, microprocessors, and warehouse-scale computers?

    [return]
show more
PCA is not a panacea
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2013-12-13 00:00:00 | Created: 2026-07-23 05:18:40

Earlier this year, I interviewed with a well-known tech startup, one of the hundreds of companies that claims to have harder interviews, more challenging work, and smarter employees than Google1. My first interviewer, John, gave me the standard tour: micro-kitchen stocked with a combination of healthy snacks and candy; white male 20-somethings gathered around a foosball table; bright spaces with cutesy themes; a giant TV set up for video games; and the restroom. Finally, he showed me a closet-sized conference room and we got down to business.

After the usual data structures and algorithms song and dance, we moved on to the main question: how would you design a classification system for foo2? We had a discussion about design tradeoffs, but the key disagreement was about the algorithm. I said, if I had to code something up in an interview, I'd use a naive matrix factorization algorithm, but that I didn't expect that I would get great results because not everything can be decomposed easily. John disagreed – he was adamant that PCA was the solution for any classification problem.

We discussed the mathematical underpinnings for twenty-five minutes – half the time allocated for the interview – and it became clear that neither of us was going to convince the other with theory. I switched gears and tried the empirical approach, referring to an old result on classifying text with LSA (which can only capture pairwise correlations between words)3 vs. deep learning4. Here's what you get with LSA:

2-d LSA

Each color represents a different type of text, projected down to two dimensions; you might not want to reduce to the dimensionality that much, but it's a good way to visualize what's going on. There's some separation between the different categories; the green dots tend to be towards the bottom right, the black dots are a lot denser in the top half of the diagram, etc. But any classification based on that is simply not going to be very good when documents are similar and the differences between them are nuanced.

Here's what we get with a deep autoencoder:

2-d deep autoencoder

It's not perfect, but the results are a lot better.

Even after the example, it was clear that I wasn't going to come to an agreement with my interviewer, so I asked if we could agree to disagree and move on to the next topic. No big deal, since it was just an interview. But I see this sort of misapplication of bog standard methods outside of interviews at least once a month, usually with the conviction that all you need to do is apply this linear technique for any problem you might see.

Engineers are the first to complain when consultants with generic business knowledge come in, charge $500/hr and dispense common sense advice while making a mess of the details. But data science is new and hot enough that people get a pass when they call themselves data scientists instead of technology consultants. I don't mean to knock data science (whatever that means), or even linear methods5. They're useful. But I keep seeing people try to apply the same four linear methods to every problem in sight.

In fact, as I was writing this, my girlfriend was in the other room taking a phone interview with the data science group of a big company, where they're attempting to use multivariate regression to predict the performance of their systems and decomposing resource utilization down to the application and query level from the regression coefficient, giving you results like 4000 QPS of foobar uses 18% of the CPU. The question they posed to her, which they're currently working on, was how do you speed up the regression so that you can push their test system to web scale?

The real question is, why would you want to? There's a reason pretty much every intro grad level computer architecture course involves either writing or modifying a simulator; real system performance is full of non-linear cliffs, the sort of thing where you can't just apply a queuing theory model, let alone a linear regression model. But when all you have are linear hammers, non-linear screws look a lot like nails.

In response to this, John Myles White made the good point that linear vs. non-linear isn't really the right framing, and that there really isn't a good vocabulary for talking about this sort of thing. Sorry for being sloppy with terminology. If you want to be more precise, you can replace each mention of "linear" with "mumble mumble objective function" or maybe "simple".


  1. When I was in college, the benchmark was MS. I wonder who's going to be next. [return]
  2. I'm not disclosing the exact problem because they asked to keep the interview problems a secret, so I'm describing a similar problem where matrix decomposition has the same fundamental problems. [return]
  3. If you're familiar with PCA and not LSA, you can think of LSA as something PCA-like [return]
  4. http://www.sciencemag.org/content/313/5786/504.abstract, http://www.cs.toronto.edu/~amnih/cifar/talks/salakhut_talk.pdf. In a strict sense, this work was obsoleted by a slew of papers from 2011 which showed that you can achieve similar results to this 2006 result with "simple" algorithms, but it's still true that current deep learning methods are better than the best "simple" feature learning schemes, and this paper was the first example that came to mind. [return]
  5. It's funny that I'm writing this blog post because I'm a huge fan of using the simplest thing possible for the job. That's often a linear method. Heck, one of my most common tricks is to replace a complex function with a first order Taylor expansion. [return]
show more
Data alignment and caches
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-01-02 00:00:00 | Created: 2026-07-23 05:18:40

Here's the graph of a toy benchmark1 of page-aligned vs. mis-aligned accesses; it shows a ratio of performance between the two at different working set sizes. If this benchmark seems contrived, it actually comes from a real world example of the disastrous performance implications of using nice power of 2 alignment, or page alignment in an actual system2.

Graph of Sandy Bridge Performance Graph of Westmere Performance

Except for very small working sets (1-8), the unaligned version is noticeably faster than the page-aligned version, and there's a large region up to a working set size of 512 where the ratio in performance is somewhat stable, but more so on our Sandy Bridge chip than our Westmere chip.

To understand what's going on here, we have to look at how caches organize data. By way of analogy, consider a 1,000 car parking garage that has 10,000 permits. With a direct mapped scheme (which you could call 1-way associative3), each of the ten permits that has the same 3 least significant digits would be assigned the same spot, i.e., permits 0618, 1618, 2618, and so on, are only allowed to park in spot 618. If you show up at your spot and someone else is in it, you kick them out and they have to drive back home. The next time they get called in to work, they have to drive all the way back to the parking garage.

Instead, if each car's permit allows it to park in a set that has ten possible spaces, we'll call that a 10-way set associative scheme, which gives us 100 sets of ten spots. Each set is now defined by the last 2 significant digits instead of the last 3. For example, with permit 2618, you can park in any spot from the set {018, 118, 218, …, 918}. If all of them are full, you kick out one unlucky occupant and take their spot, as before.

Let's move out of analogy land and back to our benchmark. The main differences are that there isn't just one garage-cache, but a hierarchy of them, from the L14, which is the smallest (and hence, fastest) to the L2 and L3. Each seat in a car corresponds to an address. On x86, each addresses points to a particular byte. In the Sandy Bridge chip we're running on, we've got a 32kB L1 cache with 64-byte line size and, 64 sets, with 8-way set associativity. In our analogy, a line size of 64 would correspond to a car with 64 seats. We always transfer things in 64-byte chunks and the bottom log₂(64) = 6 bits of an address refer to a particular byte offset in a cache line. The next log₂(64) = 6 bits determine which set an address falls into5. Each of those sets can contain 8 different things, so we have 64 sets * 8 lines/set * 64 bytes/line = 32kB. If we use the cache optimally, we can store 32,768 items. But, since we're accessing things that are page (4k) aligned, we effectively lose the bottom log₂(4k) = 12 bits, which means that every access falls into the same set, and we can only loop through 8 things before our working set is too large to fit in the L1! But if we'd misaligned our data to different cache lines, we'd be able to use 8 * 64 = 512 locations effectively.

Similarly, our chip has a 512 set L2 cache, of which 8 sets are useful for our page aligned accesses, and a 12288 set L3 cache, of which 192 sets are useful for page aligned accesses, giving us 8 sets * 8 lines / set = 64 and 192 sets * 8 lines / set = 1536 useful cache lines, respectively. For data that's misaligned by a cache line, we have an extra 6 bits of useful address, which means that our L2 cache now has 32,768 useful locations.

In the Sandy Bridge graph above, there's a region of stable relative performance between 64 and 512, as the page-aligned version version is running out of the L3 cache and the unaligned version is running out of the L1. When we pass a working set of 512, the relative ratio gets better for the aligned version because it's now an L2 access vs. an L3 access. Our graph for Westmere looks a bit different because its L3 is only 3072 sets, which means that the aligned version can only stay in the L3 up to a working set size of 384. After that, we can see the terrible performance we get from spilling into main memory, which explains why the two graphs differ in shape above 384.

For a visualization of this, you can think of a 32 bit pointer looking like this to our L1 and L2 caches:

TTTT TTTT TTTT TTTT TTTT SSSS SSXX XXXX

TTTT TTTT TTTT TTTT TSSS SSSS SSXX XXXX

The bottom 6 bits are ignored, the next bits determine which set we fall into, and the top bits are a tag that let us know what's actually in that set. Note that page aligning things, i.e., setting the address to

???? ???? ???? ???? ???? 0000 0000 0000

was just done for convenience in our benchmark. Not only will aligning to any large power of 2 cause a problem, generating addresses with a power of 2 offset from each other will cause the same problem.

Nowadays, the importance of caches is well understood enough that, when I'm asked to look at a cache related performance bug, it's usually due to the kind of thing we just talked about: conflict misses that prevent us from using our full cache effectively6. This isn't the only way for that to happen -- bank conflicts and and false dependencies are also common problems, but I'll leave those for another blog post.

Resources

For more on caches on memory, see What Every Programmer Should Know About Memory. For something with more breadth, see this blog post for something "short", or Modern Processor Design for something book length. For even more breadth (those two links above focus on CPUs and memory), see Computer Architecture: A Quantitative Approach, which talks about the whole system up to the datacenter level.


  1. The Sandy Bridge is an i7 3930K and the Westmere is a mobile i3 330M [return]
  2. Or anyone who aligned their data too nicely on a calculation with two source arrays and one destination when running on a chip with a 2-way associative or direct mapped cache. This is surprisingly common when you set up your arrays in some nice way in order to do cache blocking, if you're not careful. [return]
  3. Don't call it that. People will you look at you funny the same way they would if you pronounced SQL as squeal or squll. [return]
  4. In this post, L1 refers to the l1d. Since we're only concerned with data, the l1i isn't relevant. Apologies for the sloppy use of terminology. [return]
  5. If it seems odd that the least significant available address bits are used for the set index, that's because of the cardinal rule of computer architecture, make the common case fast -- Google Instant completes “make the common” to “make the common case fast”, “make the common case fast mips”, and “make the common case fast computer architecture”. The vast majority of accesses are close together, so moving the set index bits upwards would cause more conflict misses. You might be able to get away with a hash function that isn't simply the least significant bits, but most proposed schemes hurt about as much as they help while adding extra complexity. [return]
  6. Cache misses are often described using the 3C model: conflict misses, which are caused by the type of aliasing we just talked about; compulsory misses, which are caused by the first access to a memory location; and capacity misses, which are caused by having a working set that's too large for a cache, even without conflict misses. Page-aligned accesses like these also make compulsory misses worse, because prefetchers won't prefetch beyond a page boundary. But if you have enough data that you're aligning things to page boundaries, you probably can't do much about that anyway. [return]
show more
Do programmers need math?
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-01-09 00:00:00 | Created: 2026-07-23 05:18:40

Dear David,

I'm afraid my off the cuff response the other day wasn't too well thought out; when you talked about taking calc III and linear algebra, and getting resistance from one of your friends because "wolfram alpha can do all of that now," my first reaction was horror-- which is why I replied that while I've often regretted not taking a class seriously because I've later found myself in a situation where I could have put the skills to good use, I've never said to myself "what a waste of time it was to learn that fundamental mathematical concept and use it enough to that I truly understand it."

But could this be selection bias? It's easier to recall the math that I use than the math I don't. To check, let's look at the nine math classes I took as an undergrad. If I exclude the jobs I've had that are obviously math oriented (pure math and CS theory, plus femtosecond optics), and consider only whether I've used math skills in non-math-oriented work, here's what I find: three classes whose material I've used daily for months or years on end (Calc I/II, Linear Algebra, and Calc III); three classes that have been invaluable for short bursts (Combinatorics, Error Correcting Codes, and Computational Learning Theory); one course I would have had use for had I retained any of the relevant information when I needed it (Graduate Level Matrix Analysis); one class whose material I've only relied on once (Mathematical Economics); and only one class I can't recall directly applying to any non-math-y work (Real Analysis). Here's how I ended up using these:

Calculus I/II1: critical for dealing with real physical things as well as physically inspired algorithms. Moreover, one of my most effective tricks is substituting a Taylor or Remez series (or some other approximation function) for a complicated function, where the error bounds aren't too high and great speed is required.

Linear Algebra: although I've gone years without, it's hard to imagine being able to dodge linear algebra for the rest of my career because of how general matrices are.

Calculus III: same as Calc I/II.

Combinatorics: useful for impressing people in interviews, if nothing else. Most of my non-interview use of combinatorics comes from seeing simplifications of seemingly complicated problems; combines well with probability and randomized algorithms.

Error Correcting Codes: there's no substitute when you need ECC. More generally, information theory is invaluable.

Graduate Level Matrix Analysis: had a decade long gap between learning this and working on something where the knowledge would be applicable. Still worthwhile, though, for the same reason Linear Algebra is important.

Real Analysis: can't recall any direct applications, although this material is useful for understanding topology and measure theory.

Computational Learning Theory: useful for making the parts of machine learning people think are scary quite easy, and for providing an intuition for areas of ML that are more alchemy than engineering.

Mathematical Economics: Lagrange multipliers have come in handy sometimes, but more for engineering than programming.

Seven out of nine. Not bad. So I'm not sure how to reconcile my experience with the common sentiment that, outside of a handful of esoteric areas like computer graphics and machine learning, there is no need to understand textbook algorithms, let alone more abstract concepts like math.

Part of it is selection bias in the jobs I've landed; companies that do math-y work are more likely to talk to me. A couple weeks ago, I had a long discussion with a group of our old Hacker School friends, who now do a lot of recruiting at career fairs; a couple of them, whose companies don't operate at the intersection of research and engineering, mentioned that they politely try to end the discussion when they run into someone like me because they know that I won't take a job with them2.

But it can't all be selection bias. I've gotten a lot of mileage out of math even in jobs that are not at all mathematical in nature. Even in low-level systems work that's as far removed from math as you can get, it's not uncommon to be find a simple combinatorial proof to show that a solution that seems too stupid to be correct is actually optimal, or correct with high probability; even when doing work that's far outside the realm of numerical methods, it sometimes happens that the bottleneck is a function that can be more quickly computed using some freshman level approximation technique like a Taylor expansion or Newton's method.

Looking back at my career, I've gotten more bang for the buck from understanding algorithms and computer architecture than from understanding math, but I really enjoy math and I'm glad that knowing a bit of it has biased my career towards more mathematical jobs, and handed me some mathematical interludes in profoundly non-mathematical jobs.

All things considered, my real position is a bit more relaxed than I thought: if you enjoy math, taking more classes for the pure joy of solving problems is worthwhile, but math classes aren't the best use of your time if your main goal is to transition from an academic career to programming.



Cheers,
Dan

Russian translation available here


  1. A brilliant but mad lecturer crammed both semesters of the theorem/proof-oriented Apostol text into two months and then started lecturing about complex analysis when we ran out of book. I didn't realize that math is fun until I took this class. This footnote really ought to be on the class name, but rdiscount doesn't let you put a footnote on or in bolded text. [return]
  2. This is totally untrue, by the way. It would be super neat to see what a product oriented role is like. As it is now, I'm five teams removed from any actual customer. Oh well. I'm one step closer than I was in my last job. [return]
show more
Why don't schools teach debugging?
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-02-08 00:00:00 | Created: 2026-07-23 05:18:40

In the fall of 2000, I took my first engineering class: ECE 352, an entry-level digital design class for first-year computer engineers. It was standing room only, filled with waitlisted students who would find seats later in the semester as people dropped out. We had been warned in orientation that half of us wouldn't survive the year. In class, We were warned again that half of us were doomed to fail, and that ECE 352 was the weed-out class that would be responsible for much of the damage.

The class moved briskly. The first lecture wasted little time on matters of the syllabus, quickly diving into the real course material. Subsequent lectures built on previous lectures; anyone who couldn't grasp one had no chance at the next. Projects began after two weeks, and also built upon their predecessors; anyone who didn't finish one had no hope of doing the next.

A friend of mine and I couldn't understand why some people were having so much trouble; the material seemed like common sense. The Feynman Method was the only tool we needed.

  1. Write down the problem
  2. Think real hard
  3. Write down the solution

The Feynman Method failed us on the last project: the design of a divider, a real-world-scale project an order of magnitude more complex than anything we'd been asked to tackle before. On the day he assigned the project, the professor exhorted us to begin early. Over the next few weeks, we heard rumors that some of our classmates worked day and night without making progress.

But until 6pm the night before the project was due, my friend and I ignored all this evidence. It didn't surprise us that people were struggling because half the class had trouble with all of the assignments. We were in the half that breezed through everything. We thought we'd start the evening before the deadline and finish up in time for dinner.

We were wrong.

An hour after we thought we'd be done, we'd barely started; neither of us had a working design. Our failures were different enough that we couldn't productively compare notes. The lab, packed with people who had been laboring for weeks alongside those of us who waited until the last minute, was full of bad news: a handful of people had managed to produce a working division unit on the first try, but no one had figured how to convert an incorrect design into something that could do third-grade arithmetic.

I proceeded to apply the only tool I had: thinking really hard. That method, previously infallible, now yielded nothing but confusion because the project was too complex to visualize in its entirety. I tried thinking about the parts of the design separately, but that only revealed that the problem was in some interaction between the parts; I could see nothing wrong with each individual component. Thinking about the relationship between pieces was an exercise in frustration, a continual feeling that the solution was just out of reach, as concentrating on one part would push some other critical piece of knowledge out of my head. The following semester I would acquire enough experience in managing complexity and thinking about collections of components as black-box abstractions that I could reason about a design another order of magnitude more complicated without problems — but that was three long winter months of practice away, and this night I was at a loss for how to proceed.

By 10pm, I was starving and out of ideas. I rounded up people for dinner, hoping to get a break from thinking about the project, but all we could talk about was how hopeless it was. How were we supposed to finish when the only approach was to flawlessly assemble thousands of parts without a single misstep? It was a tedious version of a deranged Atari game with no lives and no continues. Any mistake was fatal.

A number of people resolved to restart from scratch; they decided to work in pairs to check each other's work. I was too stubborn to start over and too inexperienced to know what else to try. After getting back to the lab, now half empty because so many people had given up, I resumed staring at my design, as if thinking about it for a third hour would reveal some additional insight.

It didn't. Nor did the fourth hour.

And then, just after midnight, a number of our newfound buddies from dinner reported successes. Half of those who started from scratch had working designs. Others were despondent, because their design was still broken in some subtle, non-obvious way. As I talked with one of those students, I began poring over his design. And after a few minutes, I realized that the Feynman method wasn't the only way forward: it should be possible to systematically apply a mechanical technique repeatedly to find the source of our problems. Beneath all the abstractions, our projects consisted purely of NAND gates (woe to those who dug around our toolbox enough to uncover dynamic logic), which outputs a 0 only when both inputs are 1. If the correct output is 0, both inputs should be 1. If the output is, incorrectly, 1, then at least one of the inputs must incorrectly be 0. The same logic can then be applied with the opposite polarity. We did this recursively, finding the source of all the problems in both our designs in under half an hour.

We excitedly explained our newly discovered technique to those around us, walking them through a couple steps. No one had trouble; not even people who'd struggled with every previous assignment. Within an hour, the group of folks within earshot of us had finished, and we went home.

I understand now why half the class struggled with the earlier assignments. Without an explanation of how to systematically approach problems, anyone who didn't intuitively grasp the correct solution was in for a semester of frustration. People who were, like me, above average but not great, skated through most of the class and either got lucky or wasted a huge chunk of time on the final project. I've even seen people talented enough to breeze through the entire degree without ever running into a problem too big to intuitively understand; those people have a very bad time when they run into a 10 million line codebase in the real world. The more talented the engineer, the more likely they are to hit a debugging wall outside of school.

What I don't understand is why schools don't teach systematic debugging. It's one of the most fundamental skills in engineering: start at the symptom of a problem and trace backwards to find the source. It takes, at most, half an hour to teach the absolute basics – and even that little bit would be enough to save a significant fraction of those who wash out and switch to non-STEM majors. Using the standard engineering class sequence of progressively more complex problems, a focus on debugging could expand to fill up to a semester, which would be enough to cover an obnoxious real-world bug: perhaps there's a system that crashes once a day when a Blu-ray DVD is repeatedly played using hardware acceleration with a specific video card while two webcams and record something with significant motion, as long as an obscure benchmark from 1994 is running1.

This dynamic isn't unique to ECE 352, or even Wisconsin – I saw the same thing when TA'ed EE 202, a second year class on signals and systems at Purdue. The problems were FFTs and Laplace transforms instead of dividers and Boolean2, but the avoidance of teaching fundamental skills was the same. It was clear, from the questions students asked me in office hours, that those who were underperforming weren't struggling with the fundamental concepts in the class, but with algebra: the problems were caused by not having an intuitive understanding of, for example, the difference between f(x+a) and f(x)+a.

When I suggested to the professor3 that he spend half an hour reviewing algebra for those students who never had the material covered cogently in high school, I was told in no uncertain terms that it would be a waste of time because some people just can't hack it in engineering. I was told that I wouldn't be so naive once the semester was done, because some people just can't hack it in engineering. I was told that helping students with remedial material was doing them no favors; they wouldn't be able to handle advanced courses anyway because some students just can't hack it in engineering. I was told that Purdue has a loose admissions policy and that I should expect a high failure rate, because some students just can't hack it in engineering.

I agreed that a few students might take an inordinately large amount of help, but it would be strange if people who were capable of the staggering amount of memorization required to pass first year engineering classes plus calculus without deeply understanding algebra couldn't then learn to understand the algebra they had memorized. I'm no great teacher, but I was able to get all but one of the office hour regulars up to speed over the course of the semester. An experienced teacher, even one who doesn't care much for teaching, could have easily taught the material to everyone.

Why do we leave material out of classes and then fail students who can't figure out that material for themselves? Why do we make the first couple years of an engineering major some kind of hazing ritual, instead of simply teaching people what they need to know to be good engineers? For all the high-level talk about how we need to plug the leaks in our STEM education pipeline, not only are we not plugging the holes, we're proud of how fast the pipeline is leaking.

Thanks to Kelley Eskridge, @brcpo9, and others for comments/corrections.

Elsewhere


  1. This is an actual CPU bug I saw that took about a month to track down. And this is the easy form of the bug, with a set of ingredients that causes the fail to be reproduced about once a day - the original form of the bug only failed once every few days. I'm not picking this example because it's particularly hard, either: I can think of plenty of bugs that took longer to track down and had stranger symptoms, including a disastrous bug that took six months for our best debugger to understand.

    For ASIC post-silicon debug folks out there, this chip didn't have anything close to full scan, and our only method of dumping state out of the chip perturbed the state of the chip enough to make some bugs disappear. Good times. On the bright side, after dealing with non-deterministic hardware bugs with poor state visibility, software bugs seem easy. At worst, they're boring and tedious because debugging them is a matter of tracing things backwards to the source of the issue.

    [return]
  2. A co-worker of mine told me about a time at Cray when a high-level PM referred to the lack of engineering resources by saying that the project “needed more Boolean.” Ever since, I've thought of digital designers as people who consume caffeine and produce Boolean. I'm still not sure what analog magicians produce. [return]
  3. When I TA'd EE 202, there were two separate sections taught be two different professors. The professor who told me that students who fail just can't hack it was the professor who was more liked by students. He's affable and charismatic and people like him. Grades in his section were also lower than grades under the professor who people didn't like because he was thought to be mean. TA'ing this class taught me quite a bit, that people have no idea who's doing a good job and who's helping them, and also basic signals and systems (I took signals and systems I as an undergrad to fulfill a requirement and showed up to exams and passed them without learning any of the material, so to walk students through signals and systems II, I had to actually learn the material from both signals and systems I and II; before TA'ing the course, I told the department I hadn't taken the class and should probably TA a different class, but they didn't care, which taught another good life lesson). [return]
show more
That time Oracle tried to have a professor fired for benchmarking their database
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-03-05 00:00:00 | Created: 2026-07-23 05:18:40

In 1983, at the University of Wisconsin, Dina Bitton, David DeWitt, and Carolyn Turbyfill created a database benchmarking framework. Some of their results included (lower is better):

Join without indices

system joinAselB joinABprime joinCselAselB
U-INGRES 10.2 9.6 9.4
C-INGRES 1.8 2.6 2.1
ORACLE > 300 > 300 > 300
IDMnodac > 300 > 300 > 300
IDMdac > 300 > 300 > 300
DIRECT 10.2 9.5 5.6
SQL/DS 2.2 2.2 2.1

Join with indices, primary (clustered) index

system joinAselB joinABprime joinCselAselB
U-INGRES 2.11 1.66 9.07
C-INGRES 0.9 1.71 1.07
ORACLE 7.94 7.22 13.78
IDMnodac 0.52 0.59 0.74
IDMdac 0.39 0.46 0.58
DIRECT 10.21 9.47 5.62
SQL/DS 0.92 1.08 1.33

Join with indicies, secondary (non-clustered) index

system joinAselB joinABprime joinCselAselB
U-INGRES 4.49 3.24 10.55
C-INGRES 1.97 1.80 2.41
ORACLE 8.52 9.39 18.85
IDMnodac 1.41 0.81 1.81
IDMdac 1.19 0.59 1.47
DIRECT 10.21 9.47 5.62
SQL/DS 1.62 1.4 2.66

Projection (duplicate tuples removed)

system 100/10000 1000/10000
U-INGRES 64.6 236.8
C-INGRES 26.4 132.0
ORACLE 828.5 199.8
IDMnodac 29.3 122.2
IDMdac 22.3 68.1
DIRECT 2068.0 58.0
SQL/DS 28.8 28.0

Aggregate without indicies

system MIN scalar MIN agg fn 100 parts SUM agg fun 100 parts
U-INGRES 40.2 176.7 174.2
C-INGRES 34.0 495.0 484.4
ORACLE 145.8 1449.2 1487.5
IDMnodac 32.0 65.0 67.5
IDMdac 21.2 38.2 38.2
DIRECT 41.0 227.0 229.5
SQL/DS 19.8 22.5 23.5

Aggregate with indicies

system MIN scalar MIN agg fn 100 parts SUM agg fun 100 parts
U-INGRES 41.2 186.5 182.2
C-INGRES 37.2 242.2 254.0
ORACLE 160.5 1470.2 1446.5
IDMnodac 27.0 65.0 66.8
IDMdac 21.2 38.0 38.0
DIRECT 41.0 227.0 229.5
SQL/DS 8.5 22.8 23.8

Selection without indicies

system 100/10000 1000/10000
U-INGRES 53.2 64.4
C-INGRES 38.4 53.9
ORACLE 194.2 230.6
IDMnodac 31.7 33.4
IDMdac 21.6 23.6
DIRECT 43.0 46.0
SQL/DS 15.1 38.1

Selection with indicies

system 100/10000 clustered 100/10000 clustered 100/10000 1000/10000
U-INGRES 7.7 27.8 59.2 78.9
C-INGRES 3.9 18.9 11.4 54.3
ORACLE 16.3 130.0 17.3 129.2
IDMnodac 2.0 9.9 3.8 27.6
IDMdac 1.5 8.7 3.3 23.7
DIRECT 43.0 46.0 43.0 46.0
SQL/DS 3.2 27.5 12.3 39.2

In case you're familiar with the database universe as of 1983, at the time, INGRES was a research project by Stonebreaker and Wong at Berkeley that had been commercialized. C-INGRES is the commercial versionn and U-INGRES is the university version. IDM* are the IDM/500 database machine, the first widely used commercial database machine; dac is with a "database accelerator" and nodac is without. DIRECT was a research project in database machines that was started by DeWitt in 1977.

In Bitton et al.'s work, Oracle's performance stood out as unusually poor.

Larry Ellison wasn't happy with the results and it's said that he tried to have DeWitt fired. Given how difficult it is to fire professors when there's actual misconduct, the probability of Ellison sucessfully getting someone fired for doing legitimate research in their field was pretty much zero. It's also said that, after DeWitt's non-firing, Larry banned Oracle from hiring Wisconsin grads and Oracle added a term to their EULA forbidding the publication of benchmarks. Over the years, many major commercial database vendors added a license clause that made benchmarking their database illegal.

Today, Oracle hires from Wisconsin, but Oracle still forbids benchmarking of their database. Oracle's shockingly poor performance and Larry Ellison's response have gone down in history; anti-benchmarking clauses are now often known as "DeWitt Clauses", and they've spread from databases to all software, from compilers to cloud offerings1.

Meanwhile, Bitcoin users have created anonymous markets for assassinations -- users can put money into a pot that gets paid out to the assassin who kills a particular target.

Anonymous assassination markets appear to be a joke, but how about anonymous markets for benchmarks? People who want to know what kind of performance a database offers under a certain workload puts money into a pot that gets paid out to whoever runs the benchmark.

With things as they are now, you often see comments and blog posts about how someone was using postgres until management made them switch to "some commercial database" which had much worse performance and it's hard to tell if the terrible database was Oracle, MS SQL server, or perhaps another database.

If we look at major commercial databases today, two out of the three big names in commericial databases forbid publishing benchmarks. Microsoft's SQL server eula says:

You may not disclose the results of any benchmark test ... without Microsoft’s prior written approval

Oracle says:

You may not disclose results of any Program benchmark tests without Oracle’s prior consent

IBM is notable for actually allowing benchmarks:

Licensee may disclose the results of any benchmark test of the Program or its subcomponents to any third party provided that Licensee (A) publicly discloses the complete methodology used in the benchmark test (for example, hardware and software setup, installation procedure and configuration files), (B) performs Licensee's benchmark testing running the Program in its Specified Operating Environment using the latest applicable updates, patches and fixes available for the Program from IBM or third parties that provide IBM products ("Third Parties"), and (C) follows any and all performance tuning and "best practices" guidance available in the Program's documentation and on IBM's support web sites for the Program...

This gives people ammunition for a meta-argument that IBM probably delivers better performance than either Oracle or Microsoft, since they're the only company that's not scared of people publishing benchmark results, but it would be nice if we had actual numbers.

Thanks to Leah Hanson and Nathan Wailes for comments/corrections/discussion.


  1. There's at least one cloud service that disallows not only publishing benchmarks, but even "competitive benchmarking", running benchmarks to see how well the competition does. As a result, there's a product I'm told I shouldn't use to avoid even the appearance of impropriety because I work in an office with people who work on cloud related infrastructure.

    An example of a clause like this is the following term in the Salesforce agreement:

    You may not access the Services for purposes of monitoring their availability, performance or functionality, or for any other benchmarking or competitive purposes.

    If you ever wondered why uptime "benchmarking" services like cloudharmony don't include Salesforce, this is probably why. You will sometimes see speculation that Salesforce and other companies with these terms know that their service is so poor that it would be worse to have public benchmarks than to have it be known that they're afraid of public benchmarks.

    [return]
show more
That bogus gender gap article
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-03-09 00:00:00 | Created: 2026-07-23 05:18:40

Last week, Quartz published an article titled “There is no gender gap in tech salaries”. That resulted in linkbait copycat posts all over the internet, from obscure livejournals to Smithsonian.com. The claims are awfully strong, considering that the main study cited only looked at people who graduated with a B.S. exactly one year ago, not to mention the fact that the study makes literally the opposite claim.

Let's look at the evidence from the AAUW study that all these posts cite.

Looks like women make 88% of what men do in “engineering and engineering technology” and 77% of what men do in “computer and information sciences”.

The study controls for a number of factors to try to find the source of the pay gap. It finds that after controlling for self-reported hours worked, type of employment, and quality of school, “over one-third of the pay gap cannot be explained by any of these factors and appears to be attributable to gender alone”. One-third is not zero, nor is one-third of 12% or 23%. If that sounds small, consider an average raise in the post-2008 economy and how many years of experience that one-third of 23% turns into.

The Quartz article claims that, since the entire gap can be explained by some variables, the gap is by choice. In fact, the study explicitly calls out that view as being false, citing Stender v. Lucky Stores and a related study1, saying that “The case illustrates how discrimination can play a role in the explained portion of the pay gap when employers mistakenly assume that female employees prefer lower-paid positions traditionally held by women and --intentionally or not--place men and women into different jobs, ensuring higher pay for men and lower pay for women”. Women do not, in fact, just want lower paying jobs; this is, once again, diametrically opposed to the claims in the Quartz article.

Note that the study selectively controls for factors that reduce the pay gap, but not for factors that increase it. For instance, the study notes that “Women earn higher grades in college, on average, than men do, so academic achievement does not help us understand the gender pay gap”. Adjusting for grades would increase the pay gap; adjusting for all possible confounding factors, not only the factors that reduce the gap, would only make the adjusted pay gap larger.

The AAUW study isn't the only evidence the Quartz post cites. To support the conclusion that “Despite strong evidence suggesting gender pay equality, there is still a general perception that women earn less than men do”, the Quartz author cites three additional pieces of evidence. First, the BLS figure that, “when measured hourly, not annually, the pay gap between men and women is 14% not 23%”; 14% is not 0%. Second, a BLS report that indicates that men make more than women, cherry picking a single figure where women do better than men (“women who work between 30 and 39 hours a week … see table 4”); this claim is incorrect2. Third, a study from the 80s which is directly contradicted by the AAUW report from 2012; the older study indicates that cohort effects are responsible for the gender gap, but the AAUW report shows a gender gap despite studying only a single cohort.

The Smithsonian Mag published a correction in response to criticism about their article, but most of the mis-informed articles remain uncorrected.

It's clear that the author of the Quartz piece had an agenda in mind, picked out evidence that supported that agenda, and wrote a blog post. A number of bloggers picked up the post and used its thesis as link bait to drive hits to their sites, without reading any of the cited evidence. If this is how “digitally native news” works, I'm opting out.

If you liked reading this, you might also enjoy this post on the interaction of markets with discrimination, and this post, which has a very partial explanation of why so many people drop out of science and engineering.

Updates

Update: A correction! I avoided explicitly linking to the author of the original article, because I find the sort of twitter insults and witch hunts that often pop up to be unconstructive, and this is really about what's right and not who's right. The author obviously disagrees because I saw no end of insults until I blocked the author.

Charlie Clarke was kind enough to wade through the invective and decode the author's one specific claim that about my illiteracy was that footnote 3 was not rounded from 110.3 to 111. It turns out that instead of rounding from 110.3 to 111, the author of the article cited the wrong source entirely and the other source just happened to have a number that was similar to 111.


  1. There's plenty of good news in this study. The gender gap has gotten much smaller over the past forty years. There's room for a nuanced article that explores why things improved, and why certain aspects have improved while others have remained stubbornly stuck in the 70s. I would love to read that article. [return]
  2. The Quartz article claims that “women who work 30 to 39 hours per week make 111% of what men make (see table 4)”. Table 4 is a breakdown of part-time workers. There is no 111% anywhere in the table, unless 110.3% is rounded to 111%; perhaps the author is referring to the racial breakdown in the table, which indicates that among Asian part-time workers, women earn 110.3% of what men do per hour. Note that Table 3, showing a breakdown of full-time workers (who are the vast majority of workers) indicates that women earn much less than men when working full time. To find a figure that supports the author's agenda, the author had to not only look at part time workers, but only look at part-time Asian women, and then round .3% up to 1%. [return]
show more
Editing binaries
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-03-23 00:00:00 | Created: 2026-07-23 05:18:40

Editing binaries is a trick that comes in handy a few times a year. You don't often need to, but when you do, there's no alternative. When I mention patching binaries, I get one of two reactions: complete shock or no reaction at all. As far as I can tell, this is because most people have one of these two models of the world:

  1. There exists source code. Compilers do something to source code to make it runnable. If you change the source code, different things happen.

  2. There exists a processor. The processor takes some bits and decodes them to make things happen. If you change the bits, different things happen.

If you have the first view, breaking out a hex editor to modify a program is the action of a deranged lunatic. If you have the second view, editing binaries is the most natural thing in the world. Why wouldn't you just edit the binary? It's often the easiest way to get what you need.

For instance, you're forced to do this all the time if you use a non-Intel non-AMD x86 processor. Instead of checking CPUID feature flags, programs will check the CPUID family, model, and stepping to determine features, which results in incorrect behavior on non-standard CPUs. Sometimes you have to do an edit to get the program to use the latest SSE instructions and sometimes you have to do an edit to get the program to run at all. You can try filing a bug, but it's much easier to just edit your binaries.

Even if you're running on a mainstream Intel CPU, these tricks are useful when you run into bugs in closed sourced software. And then there are emergencies.

The other day, a DevOps friend of mine at a mid-sized startup told me about the time they released an internal alpha build externally, which caused their auto-update mechanism to replace everyone's working binary with a buggy experimental version. It only took a minute to figure out what happened. Updates gradually roll out to all users over a couple days, which meant that the bad version had only spread to 1 / (60*24*2) = 0.03% of all users. But they couldn't push the old version into the auto-updater because the client only accepts updates from higher numbered versions. They had to go through the entire build and release process (an hour long endeavor) just to release a version that was identical to their last good version. If it had occurred to anyone to edit the binary to increment the version number, they could have pushed out a good update in a minute instead of an hour, which would have kept the issue from spreading to more than 0.06% of their users, instead of sending 2% of their users a broken update1.

This isn't nearly as hard as it sounds. Let's try an example. If you're going to do this sort of thing regularly, you probably want to use a real disassembler like IDA2. But, you can get by with simple tools if you only need to do this every once in a while. I happen to be on a Mac that I don't use for development, so I'm going to use lldb for disassembly and HexFiend to edit this example. Gdb, otool, and objdump also work fine for quick and dirty disassembly.

Here's a toy code snippet, wat-arg.c, that should be easy to binary edit:

#include <stdio.h>

int main(int argc, char **argv) {
  if (argc > 1) {
    printf("got an arg\n");
  } else {
    printf("no args\n");
  }
}

If we compile this and then launch lldb on the binary and step into main, we can see the following machine code:

$ lldb wat-arg
(lldb) breakpoint set -n main
Breakpoint 1: where = original`main, address = 0x0000000100000ee0
(lldb) run
(lldb) disas -b -p -c 20
;  address       hex opcode            disassembly
-> 0x100000ee0:  55                    pushq  %rbp
   0x100000ee1:  48 89 e5              movq   %rsp, %rbp
   0x100000ee4:  48 83 ec 20           subq   $32, %rsp
   0x100000ee8:  c7 45 fc 00 00 00 00  movl   $0, -4(%rbp)
   0x100000eef:  89 7d f8              movl   %edi, -8(%rbp)
   0x100000ef2:  48 89 75 f0           movq   %rsi, -16(%rbp)
   0x100000ef6:  81 7d f8 01 00 00 00  cmpl   $1, -8(%rbp)
   0x100000efd:  0f 8e 16 00 00 00     jle    0x100000f19               ; main + 57
   0x100000f03:  48 8d 3d 4c 00 00 00  leaq   76(%rip), %rdi            ; "got an arg\n"
   0x100000f0a:  b0 00                 movb   $0, %al
   0x100000f0c:  e8 23 00 00 00        callq  0x100000f34               ; symbol stub for: printf
   0x100000f11:  89 45 ec              movl   %eax, -20(%rbp)
   0x100000f14:  e9 11 00 00 00        jmpq   0x100000f2a               ; main + 74
   0x100000f19:  48 8d 3d 42 00 00 00  leaq   66(%rip), %rdi            ; "no args\n"
   0x100000f20:  b0 00                 movb   $0, %al
   0x100000f22:  e8 0d 00 00 00        callq  0x100000f34               ; symbol stub for: printf

As expected, we load a value, compare it to 1 with cmpl $1, -8(%rbp), and then print got an arg or no args depending on which way we jump as a result of the compare.

$ ./wat-arg
no args
$ ./wat-arg 1
got an arg

If we open up a hex editor and change 81 7d f8 01 00 00 00; cmpl 1, -8(%rbp) to 81 7d f8 06 00 00 00; cmpl 6, -8(%rbp), that should cause the program to check for 6 args instead of 1

Replace cmpl with cmpl 6

$ ./wat-arg
no args
$ ./wat-arg 1
no args
$ ./wat-arg 1 2
no args
$ ./wat-arg 1 2 3 4 5 6 7 8
got an arg

Simple! If you do this a bit more, you'll soon get in the habit of patching in 903 to overwrite things with NOPs. For example, if we replace 0f 8e 16 00 00 00; jle and e9 11 00 00 00; jmpq with 90, we get the following:

   0x100000ee1:  48 89 e5              movq   %rsp, %rbp
   0x100000ee4:  48 83 ec 20           subq   $32, %rsp
   0x100000ee8:  c7 45 fc 00 00 00 00  movl   $0, -4(%rbp)
   0x100000eef:  89 7d f8              movl   %edi, -8(%rbp)
   0x100000ef2:  48 89 75 f0           movq   %rsi, -16(%rbp)
   0x100000ef6:  81 7d f8 01 00 00 00  cmpl   $1, -8(%rbp)
   0x100000efd:  90                    nop
   0x100000efe:  90                    nop
   0x100000eff:  90                    nop
   0x100000f00:  90                    nop
   0x100000f01:  90                    nop
   0x100000f02:  90                    nop
   0x100000f03:  48 8d 3d 4c 00 00 00  leaq   76(%rip), %rdi            ; "got an arg\n"
   0x100000f0a:  b0 00                 movb   $0, %al
   0x100000f0c:  e8 23 00 00 00        callq  0x100000f34               ; symbol stub for: printf
   0x100000f11:  89 45 ec              movl   %eax, -20(%rbp)
   0x100000f14:  90                    nop
   0x100000f15:  90                    nop
   0x100000f16:  90                    nop
   0x100000f17:  90                    nop
   0x100000f18:  90                    nop
   0x100000f19:  48 8d 3d 42 00 00 00  leaq   66(%rip), %rdi            ; "no args\n"
   0x100000f20:  b0 00                 movb   $0, %al
   0x100000f22:  e8 0d 00 00 00        callq  0x100000f34               ; symbol stub for: printf

Note that since we replaced a couple of multi-byte instructions with single byte instructions, the program now has more total instructions.

$ ./wat-arg
got an arg
no args

Other common tricks include patching in cc to redirect to an interrupt handler, db to cause a debug breakpoint, knowing which bit to change to flip the polarity of a compare or jump, etc. These things are all detailed in the Intel architecture manuals, but the easiest way to learn these is to develop the muscle memory for them one at a time.

Have fun!


  1. I don't actually recommend doing this in an emergency if you haven't done it before. Pushing out a known broken binary that leaks details from future releases is bad, but pushing out an update that breaks your updater is worse. You'll want, at a minimum, a few people who create binary patches in their sleep to code review the change to make sure it looks good, even after running it on a test client.

    Another solution, not quite as "good", but much less dangerous, would have been to disable the update server until the new release was ready.

    [return]
  2. If you don't have $1000 to spare, r2 is a nice, free, tool with IDA-like functionality. [return]
  3. on x86 [return]
show more
Data-driven bug finding
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-04-06 00:00:00 | Created: 2026-07-23 05:18:40

I can't remember the last time I went a whole day without running into a software bug. For weeks, I couldn't invite anyone to Facebook events due to a bug that caused the invite button to not display on the invite screen. Google Maps has been giving me illegal and sometimes impossible directions ever since I moved to a small city. And Google Docs regularly hangs when I paste an image in, giving me a busy icon until I delete the image.

It's understandable that bugs escape testing. Testing is hard. Integration testing is harder. End to end testing is even harder. But there's an easier way. A third of bugs like this – bugs I run into daily – could be found automatically using analytics.

If you think finding bugs with analytics sounds odd, ask a hardware person about performance counters. Whether or not they're user accessible, every ASIC has analytics to allow designers to figure out what changes need to be made for the next generation chip. Because people look at perf counters anyway, they notice when a forwarding path never gets used, when way prediction has a strange distribution, or when the prefetch buffer never fills up. Unexpected distributions in analytics are a sign of a misunderstanding, which is often a sign of a bug1.

Facebook logs all user actions. That can be used to determine user dead ends. Google Maps reroutes after “wrong” turns. That can be used to determine when the wrong turns are the result of bad directions. Google Docs could track all undos2. That could be used to determine when users run into misfeatures or bugs3.

I understand why it might feel weird to borrow hardware practices for software development. For the most part, hardware tools are decades behind software tools. As examples: current hardware tools include simulators on Linux that are only half ported from windows, resulting in some text boxes requiring forward slashes while others require backslashes; libraries that fail to compile with `default_nettype none4; and components that come with support engineers because they're expected to be too buggy to work without full-time people supporting any particular use.

But when it comes to testing, hardware is way ahead of software. When I write software, fuzzing is considered a state of the art technique. But in hardware land, fuzzing doesn't have a special name. It's just testing, and why should there be a special name for "testing that uses randomness"? That's like having a name for "testing by running code". Well over a decade ago, I did hardware testing via a tool that used constrained randomness on inputs, symbolic execution, with state reduction via structural analysis. For small units, the tool was able to generate a formal proof of correctness. For larger units, the tool automatically generated and used coverage statistics and used them to exhaustively search over as diverse a state space as possible. In the case of a bug, a short, easy to debug, counter example would be produced. And hardware testing tools have gotten a lot better since then.

But in software land, I'm lucky if a random project I want to contribute to has tests at all. When tests exist, they're usually handwritten, with all the limitations that implies. Once in a blue moon, I'm pleasantly surprised to find that a software project uses a test framework which has 1% of the functionality that was standard a decade ago in chip designs.

Considering the relative cost of hardware bugs vs. software bugs, it's not too surprising that a lot more effort goes into hardware testing. But here's a case where there's almost no extra effort. You've already got analytics measuring the conversion rate through all sorts of user funnels. The only new idea here is that clicking on an ad or making a purchase isn't the only type of conversion you should measure. Following directions at an intersection is a conversion, not deleting an image immediately after pasting it is a conversion, and using a modal dialogue box after opening it up is a conversion.

Of course, whether it's ad click conversion rates or cache hit rates, blindly optimizing a single number will get you into a local optima that will hurt you in the long run, and setting thresholds for conversion rates that should send you an alert is nontrivial. There's a combinatorially large space of user actions, so it takes judicious use of machine learning to figure out reasonable thresholds. That's going to cost time and effort. But think of all the effort you put into optimizing clicks. You probably figured out, years ago, that replacing boring text with giant pancake buttons gives you 3x the clickthrough rate; you're now down to optimizing 1% here and 2% there. That's great, and it's a sign that you've captured all the low hanging fruit. But what do you think the future clickthrough rate is when a user encounters a show-stopping bug that prevents any forward progress on a modal dialogue box?

If this sounds like an awful lot of work, find a known bug that you've fixed, and grep your logs data for users who ran into that bug. Alienating those users by providing a profoundly broken product is doing a lot more to your clickthrough rate than having a hard to find checkout button, and the exact same process that led you to that gigantic checkout button can solve your other problem, too. Everyone knows that adding 200ms of load time can cause 20% of users to close the window. What do you think the effect of exposing them to a bug that takes 5,000ms of user interaction to fix is?

If that's worth fixing, pull out scalding, dremel, cascalog, or whatever your favorite data processing tool is. Start looking for user actions that don't make sense. Start looking for bugs.

Thanks to Pablo Torres for catching a typo in this post


  1. It's not that all chip design teams do this systematically (although they should), but that people are looking at the numbers anyway, and will see anomalies. [return]
  2. Undos aren't just literal undos; pasting an image in and then deleting it afterwards because it shows a busy icon forever counts, too. [return]
  3. This is worse than it sounds. In addition to producing a busy icon forever in the doc, it disconnects that session from the server, which is another thing that could be detected: it's awfully suspicious if a certain user action is always followed by a disconnection.

    Moreover, both of these failure modes could have been found with fuzzing, since they should never happen. Bugs are hard enough to find that defense in depth is the only reasonable solution.

    [return]
  4. if you talk to a hardware person, call this verification instead of testing, or they'll think you're talking about DFT, testing silicon for manufacturing defects, or some other weird thing with no software analogue. [return]
show more
Verilog Won & VHDL Lost? — You Be The Judge!
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-08-14 00:00:00 | Created: 2026-07-23 05:18:40

This is an archived USENET post from John Cooley on a competitive comparison between VHDL and Verilog that was done in 1997.

I knew I hit a nerve. Usually when I publish a candid review of a particular conference or EDA product I typically see around 85 replies in my e-mail "in" box. Buried in my review of the recent Synopsys Users Group meeting, I very tersely reported that 8 out of the 9 Verilog designers managed to complete the conference's design contest yet none of the 5 VHDL designers could. I apologized for the terseness and promised to do a detailed report on the design contest at a later date. Since publishing this, my e-mail "in" box has become a veritable Verilog/VHDL Beirut filling up with 169 replies! Once word leaked that the detailed contest write-up was going to be published in the DAC issue of "Integrated System Design" (formerly "ASIC & EDA" magazine), I started getting phone calls from the chairman of VHDL International, Mahendra Jain, and from the president of Open Verilog International, Bill Fuchs. A small army of hired gun spin doctors (otherwise know as PR agents) followed with more phone calls. I went ballistic when VHDL columnist Larry Saunders had approached the Editor-in-Chief of ISD for an advanced copy of my design contest report. He felt I was "going to do a hatchet job on VHDL" and wanted to write a rebuttal that would follow my article... and all this was happening before I had even written one damned word of the article!

Because I'm an independent consultant who makes his living training and working both HDL's, I'd rather not go through a VHDL Salem witch trial where I'm publically accused of being secretly in league with the Devil to promote Verilog, thank you. Instead I'm going present everything that happened at the Design Contest, warts and all, and let you judge! At the end of court evidence, I'll ask you, the jury, to write an e-mail reply which I can publish in my column in the follow-up "Integrated System Design".

The Unexpected Results

Contestants were given 90 minutes using either Verilog or VHDL to create a gate netlist for the fastest fully synchronous loadable 9-bit increment-by-3 decrement-by-5 up/down counter that generated even parity, carry and borrow.

Of the 9 Verilog designers in the contest, only 1 didn't get to a final gate level netlist because he tried to code a look-ahead parity generator. Of the 8 remaining, 3 had netlists that missed on functional test vectors. The 5 Verilog designers who got fully functional gate-level designs were:

   Larry Fiedler     NVidea               3.90 nsec     1147 gates
   Steve Golson      Trilobyte Systems    4.30 nsec     1909 gates
   Howard Landman    HaL Computer         5.49 nsec     1495 gates
   Mark Papamarcos   EDA Associates       5.97 nsec     1180 gates
   Ed Paluch         Paluch & Assoc.      7.85 nsec     1514 gates

The surprize was that, during the same time, none of 5 VHDL designers in the contest managed to produce any gate level designs.

Not VHDL Newbies vs. Verilog Pros

The first reaction I get from the VHDL bigots (who weren't at the competition) is: "Well, this is obviously a case where Verilog veterans whipped some VHDL newbies. Big deal." Well, they're partially right. Many of those Verilog designers are damned good at what they do — but so are the VHDL designers!

I've known Prasad Paranjpe of LSI Logic for years. He has taught and still teaches VHDL with synthesis classes at U.C. Santa Cruz University Extention in the heart of Silicon Valley. He was VP of the Silicon Valley VHDL Local Users Group. He's been a full time ASIC designer since 1987 and has designed real ASIC's since 1990 using VHDL & Synopsys since rev 1.3c. Prasad's home e-mail address is "vhdl@ix.netcom.com" and his home phone is (XXX) XXX-VHDL. ASIC designer Jan Decaluwe has a history of contributing insightful VHDL and synthesis posts to ESNUG while at Alcatel and later as a founder of Easics, a European ASIC design house. (Their company motto: "Easics - The VHDL Design Company".) Another LSI Logic/VHDL contestant, Vikram Shrivastava, has used the VHDL/Synopsys design approach since 1992. These guys aren't newbies!

Creating The Contest

I followed a double blind approach to putting together this design contest. That is, not only did I have Larry Saunders (a well known VHDL columnist) and Yatin Trivedi (a well known Verilog columnist), both of Seva Technologies comment on the design contest — unknown to them I had Ken Nelsen (a VHDL oriented Methodology Manager from Synopsys) and Jeff Flieder (a Verilog based designer from Ford Microelectronics) also help check the design contest for any conceptual or implementation flaws.

My initial concern in creating the contest was to not have a situation where the Synopsys Design Compiler could quickly complete the design by just placing down a DesignWare part. Yet, I didn't want to have contestants trying (and failing) to design some fruity, off-the-wall thingy that no one truely understood. Hence, I was restricted to "standard" designs that all engineers knew — but with odd parameters thrown in to keep DesignWare out of the picture. Instead of a simple up/down counter, I asked for an up-by-3 and down-by-5 counter. Instead of 8 bits, everything was 9 bits.

                                  recycled COUNT_OUT [8:0]
                     o---------------<---------------<-------------------o
                     |                                                   |
                     V                                                   |
               -------------                     --------                |
  DATA_IN -->-|   up-by-3   |->-----carry----->-| D    Q |->- CARRY_OUT  |
   [8:0]      |  down-by-5  |->-----borrow---->-| D    Q |->- BORROW_OUT |
              |             |                   |        |               |
       UP -->-|    logic    |                   |        |               |
     DOWN -->-|             |-o------->---------| D[8:0] |               |
               -------------  | new_count [8:0] | Q[8:0] |->-o---->------o
                              |                 |        |   |
                 o------<-----o        CLOCK ---|>       |   o->- COUNT_OUT
                 |                               --------           [8:0]
 new_count [8:0] |     -----------
                 |    |   even    |              --------
                 o-->-|  parity   |->-parity-->-| D    Q |->- PARITY_OUT
                      | generator |   (1 bit)   |        |
                       -----------           o--|>       |
                                             |   --------
                                   CLOCK ----o


Fig.1) Basic block diagram outlining design's functionality

The even PARITY, CARRY and BORROW requirements were thrown in to give the contestants some space to make significant architectural trade-offs that could mean the difference between winning and losing.

The counter loaded when the UP and DOWN were both "low", and held its state when UP and DOWN were "high" — exactly opposite to what 99% of the world's loadable counters traditionally do.

                  UP  DOWN   DATA_IN    |    COUNT_OUT    
                 -----------------------------------------
                   0    0     valid     |   load DATA_IN
                   0    1   don't care  |     (Q - 5)
                   1    0   don't care  |     (Q + 3)
                   1    1   don't care  |   Q unchanged


Fig. 2) Loading and up/down counting specifications.  All I/O events
happen on the rising edge of CLOCK.

To spice things up a bit further, I chose to use the LSI Logic 300K ASIC library because wire loading & wire delay is a significant factor in this technology. Having the "home library" advantage, one saavy VHDL designer, Prasad Paranjpe of LSI Logic, cleverly asked if the default wire loading model was required (he wanted to use a zero wire load model to save in timing!) I replied: "Nice try. Yes, the default wire model is required."

To let the focus be on design and not verification, contestants were given equivalent Verilog and VHDL testbenches provided by Yatin Trivedi & Larry Saunder's Seva Technologies. These testbenches threw the same 18 vectors at the Verilog/VHDL source code the contestants were creating and if it passed, for contest purposes, their design was judged "functionally correct."

For VHDL, contestants had their choice of Synopsys VSS 3.2b and/or Cadence Leapfrog VHDL 2.1.4; for Verilog, contestants had their choice of Cadence Verilog-XL 2.1.2 or Chronologic VCS 2.3.2 plus their respective Verilog/VHDL design environments. (The CEO of Model Technology Inc., Bob Hunter, was too paranoid about the possiblity of Synopsys employees seeing his VHDL to allow it in the contest.) LCB 300K rev 3.1A.1.1.101 was the LSI Logic library.

I had a concern that some designers might not know that an XOR reduction tree is how one generates parity — but Larry, Yatin, Ken & Jeff all agreed that any engineer not knowing this shouldn't be helped to win a design contest. As a last minute hint, I put in every contestant's directory an "xor.readme" file that named the two XOR gates available in LSI 300K library (EO and EO3) plus their drive strengths and port lists.

To be friendly synthesis-wise, I let the designers keep the unrealistic Synopsys default setting of all inputs having infinite input drive strength and all outputs were driving zero loads.

The contest took place in three sessions over the same day. To keep things equal, my guiding philosophy throughout these sessions was to conscientiously not fix/improve anything between sessions — no matter how frustrating!

After all that was said & done, Larry & Yatin thought that the design contest would be too easy while Ken & Jeff thought it would have just about the right amount of complexity. I asked all four if they saw any Verilog or VHDL specific "gotchas" with the contest; all four categorically said "no."

Murphy's Law

Once the contest began, Murphy's Law — "that which can go wrong, will go wrong" — prevailed. Because we couldn't get the SUN and HP workstations until a terrifying 3 days before the contest, I lived through a nightmare domino effect on getting all the Verilog, VHDL, Synopsys and LSI libraries in and installed. Nobody could cut keys for the software until the machine ID's were known — and this wasn't until 2 days before the contest! (As it was, I had to drop the HP machines because most of the EDA vendors couldn't cut software keys for HP machines as fast as they could for SUN workstations.)

The LSI 300K Libraries didn't arrive until an hour before the contest began. The Seva guys found and fixed a bug in the Verilog testbench (that didn't exist in the VHDL testbench) some 15 minutes before the constest began.

Some 50 minutes into the first design session, one engineer's machine crashed — which also happened to be the licence server for all the Verilog simulation software! (Luckily, by this time all the Verilog designers were deep into the synthesis stage.) Unfortunately, the poor designer who had his machine crash couldn't be allowed to redo the contest in a following session because of his prior knowlege of the design problem. This machine was rebooted and used solely as a licence server for the rest of the contest.

The logistics nightmare once again reared its ugly head when two designers innocently asked: "John, where are your Synopsys manuals?" Inside I screamed to myself: "OhMyGod! OhMyGod! OhMyGod!"; outside I calmly replied: "There are no manuals for any software here. You have to use the online docs available."

More little gremlins danced in my head when I realized that six of the eight data books that the LSI lib person brought weren't for the exact LCB 300K library we were using — these data books would be critical for anyone trying to hand build an XOR reduction tree — and one Verilog contestant had just spent ten precious minutes reading a misleading data book! (There were two LCB 300K, one LCA 300K and five LEA 300K databooks.) Verilog designer Howard Landman of HaL Computer noted: "I probably wasted 15 minutes trying to work through this before giving up and just coding functional parity — although I used parentheses in hopes of Synopsys using 3-input XOR gates."

Then, just as things couldn't get worst, everyone got to discover that when Synopsys's Design Compiler runs for the first time in a new account — it takes a good 10 to 15 minutes to build your very own personal DesignWare cache. Verilog contestant Ed Paluch, a consultant, noted: "I thought that first synthesis run building [expletive deleted] DesignWare caches would never end! It felt like days!"

Although, in my opinion, none of these headaches compromised the integrity of the contest, at the time I had to continually remind myself: "To keep things equal, I can not fix nor improve anything no matter how frustrating."

Judging The Results

Because I didn't want to be in the business of judging source code intent, all judging was based solely on whether the gate level passed the previously described 18 test vectors. Once done, the design was read into the Synopsys Design Compiler and all constraints were removed. Then I applied the command "clocks_at 0, 6, 12 clock" and then took the longest path as determined by "report_timing -path full -delay max -max_paths 12" as the final basis for comparing designs — determining that Verilog designer Larry Fiedler of NVidia won with a 1147 gate design timed at 3.90 nsec.

      reg [9:0] cnt_up, cnt_dn;   reg [8:0] count_nxt;

      always @(posedge clock)
      begin
        cnt_dn = count_out - 3'b 101;  // synopsys label add_dn
        cnt_up = count_out + 2'b 11;   // synopsys label add_up

        case ({up,down})
           2'b 00 : count_nxt = data_in;
           2'b 01 : count_nxt = cnt_dn;
           2'b 10 : count_nxt = cnt_up;
           2'b 11 : count_nxt = 9'bX;  // SPEC NOT MET HERE!!!
          default : count_nxt = 9'bX;  // avoiding ambiguity traps
        endcase

        parity_out  <= ^count_nxt;
        carry_out   <= up & cnt_up[9];
        borrow_out  <= down & cnt_dn[9];
        count_out   <= count_nxt;
      end


Fig. 3) The winning Verilog source code.  (Note that it failed to meet
the spec of holding its state when UP and DOWN were both high.)

Since judging was open to any and all who wanted to be there, Kurt Baty, a Verilog contestant and well respected design consultant, registered a vocal double surprize because he knew his design was of comparable speed but had failed to pass the 18 test vectors. (Kurt's a good friend — I really enjoyed harassing him over this discovery — especially since he had bragged to so many people on how he was going to win this contest!) An on the spot investigation yielded that Kurt had accidently saved the wrong design in the final minute of the contest. Even further investigation then also yielded that the 18 test vectors didn't cover exactly all the counter's specified conditions. Larry's "winning" gate level Verilog based design had failed to meet the spec of holding its state when UP and DOWN were high — even though his design had successfully passed the 18 test vectors!

If human visual inspection of the Verilog/VHDL source code to subjectively check for places where the test vectors might have missed was part of the judging criteria, Verilog designer Steve Golson would have won. Once again, I had to reiterate that all designs which passed the testbench vectors were considered "functionally correct" by definition.

What The Contestants Thought

Despite NASA VHDL designer Jeff Solomon's "I didn't like the idea of taking the traditional concept of counters and warping it to make a contest design problem", the remaining twelve contestants really liked the architectural flexiblity of the up-by-3/down-by-5, 9 bit, loadable, synchronous counter with even party, carry and borrow. Verilog designer Mark Papamarcos summed up the majority opinion with: "I think that the problem was pretty well devised. There was a potential resource sharing problem, some opportunities to schedule some logic to evaluate concurrently with other logic, etc. When I first saw it, I thought it would be very easy to implement and I would have lots of time to tune. I also noticed the 2 and 3-input XOR's in the top-level directory, figured that it might be somehow relevant, but quickly dismissed any clever ideas when I ran into problems getting the vectors to match."

Eleven of contestants were tempted by the apparent correlation between known parity and the adding/subtracting of odd numbers. Only one Verilog designer, Oren Rubinstein of Hewlett-Packard Canada, committed to this strategy but ran way out of time. Once home, Kurt Baty helped Oren conceptually finish his design while Prasad Paranjpe helped with the final synthesis. It took about 7 hours brain time and 8 hours coding/sim/synth time (15 hours total) to get a final design of 3.05 nsec & 1988 gates. Observing it took 10x the original estimated 1.5 hours to get a 22% improvement in speed, Oren commented: "Like real life, it's impossible to create accurate engineering design schedules."

Two of the VHDL designers, Prasad Paranjpe of LSI Logic and Jan Decaluwe of Easics, both complained of having to deal with type conversions in VHDL. Prasad confessed: "I can't believe I got caught on a simple typing error. I used IEEE std_logic_arith, which requires use of unsigned & signed subtypes, instead of std_logic_unsigned." Jan agreed and added: "I ran into a problem with VHDL or VSS (I'm still not sure.) This case statement doesn't analyze: "subtype two_bits is unsigned(1 downto 0); case two_bits'(up & down)..." But what worked was: "case two_bits'(up, down)..." Finally I solved this problem by assigning the concatenation first to a auxiliary variable."

Verilog competitor Steve Golson outlined the first-get-a-working-design-and- then-tweak-it-in-synthesis strategy that most of the Verilog contestants pursued with: "As I recall I had some stupid typos which held me up; also I had difficulty with parity and carry/borrow. Once I had a correctly functioning baseline design, I began modifying it for optimal synthesis. My basic idea was to split the design into four separate modules: the adder, the 4:1 MUXes, the XOR logic (parity and carry/borrow), and the top counter module which contains only the flops and instances of the other three modules. My strategy was to first compile the three (purely combinational) submodules individually. I used a simple "max_delay 0 all_outputs()" constraint on each of them. The top-level module got the proper clock constraint. Then "dont_touch" these designs, and compile the top counter module (this just builds the flops). Then to clean up I did an "ungroup -all" followed by a "compile -incremental" (which shaved almost 1 nsec off my critical path.)"

Typos and panic hurt the performance of a lot of contestants. Verilog designer Daryoosh Khalilollahi of National Semiconductor said: "I thought I would not be able to finish it on time, but I just made it. I lost some time because I would get a Verilog syntax error that turned up because I had one extra file in my Verilog "include" file (verilog -f include) which was not needed." Also, Verilog designer Howard Landman of Hal Computers never realized he had put both a complete behavioral and a complete hand instanced parity tree in his source Verilog. (Synopsys Design Compiler just optimized one of Howard's dual parity trees away!)

On average, each Verilog designer managed to get two to five synthesis runs completed before running out of time. Only two VHDL designers, Jeff Solomon and Jan Decaluwe, managed to start (but not complete) one synthesis run. In both cases I disqualified them from the contest for not making the deadline but let their synthesis runs attempt to finish. Jan arrived a little late so we gave Jan's run some added time before disqualifying him. His unfinished run had to be killed after 21 minutes because another group of contestants were arriving. (Incidently, I had accidently given the third session an extra 6 design minutes because of a goof on my part. No Verilog designers were in this session but VHDL designers Jeff Solomon, Prasad Paranjpe, Vikram Shrivastava plus Ravi Srinivasan of Texus Instruments all benefited from this mistake.) Since Jeff was in the last session, I gave him all the time needed for his run to complete. After an additional 17 minutes (total) he produced a gate level design that timed out to 15.52 nsec. After a total of 28 more minutes he got the timing down to 4.46 nsec but his design didn't pass functional vectors. He had an error somewhere in his VHDL source code.

Failed Verilog designer Kurt Baty closed with: "John, I look forward to next year's design contest in whatever form or flavor it takes, and a chance to redeem my honor."

Closing Arguments To The Jury

Closing aurguments the VHDL bigots may make in this trial might be: "What 14 engineers do isn't statistically significant. Even the guy who ran this design contest admitted all sorts of last minute goofs with it. You had a workstation crash, no manuals & misleading LSI databooks. The test vectors were incomplete. One key VHDL designer ran into a Synopsys VHDL simulator bug after arriving late to his session. The Verilog design which won this contest didn't even meet the spec completely! In addition, this contest wasn't put together to be a referendum on whether Verilog or VHDL is the better language to design in — hence it may miss some major issues."

The Verilog bigots might close with: "No engineers work under the contrived conditions one may want for an ideal comparision of Verilog & VHDL. Fourteen engineers may or may not be statistally significant, but where there's smoke, there's fire. I saw all the classical problems engineers encounter in day to day designing here. We've all dealt with workstation crashes, bad revision control, bugs in tools, poor planning and incomplete testing. It's because of these realities I think this design contest was perfect to determine how each HDL measures up in real life. And Verilog won hands down!"

The jury's veridict will be seen in the next "Integrated System Design".

You The Jury...

You the jury are now asked to please take ten minutes to think about what you have just read and, in 150 words or less, send your thoughts to me at "jcooley@world.std.com". Please don't send me "VHDL sucks." or "Verilog must die!!!" — but personal experiences and/or observations that add to the discussion. It's OK to have strong/violent opinions, just back them with something more than hot air. (Since I don't want to be in the business of chasing down permissions, my default setting is whatever you send me is completely publishable. If you wish to send me letters with a mix of publishable and non-publishable material CLEARLY indicate which is which.) I will not only be reprinting replied letters, I'll also be publishing stats on how many people had reported each type of specific opinion/experience.

John Cooley
Part Time EDA Consumer Advocate
Full Time ASIC, FPGA & EDA Design Consultant

P.S. In replying, please indicate your job, your company, whether you use Verilog or VHDL, why, and for how long. Also, please DO NOT copy this article back to me — I know why you're replying! :^)

show more
Google wage fixing, 11-CV-02509-LHK, ORDER DENYING PLAINTIFFS' MOTION FOR PRELIMINARY APPROVAL OF SETTLEMENTS WITH ADOBE, APPLE, GOOGLE, AND INTEL
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-08-14 00:00:00 | Created: 2026-07-23 05:18:40
UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF CALIFORNIA
SAN JOSE DIVISION

IN RE: HIGH-TECH EMPLOYEE
ANTITRUST LITIGATION

THIS DOCUMENT RELATES TO:
ALL ACTIONS

Case No.: 11-CV-02509-LHK

ORDER DENYING PLAINTIFFS' MOTION FOR PRELIMINARY APPROVAL OF SETTLEMENTS WITH ADOBE, APPLE, GOOGLE, AND INTEL

Before the Court is a Motion for Preliminary Approval of Class Action Settlement with Defendants Adobe Systems Inc. ("Adobe"), Apple Inc. ("Apple"), Google Inc. ("Google"), and Intel Corp. ("Intel") (hereafter, "Remaining Defendants") brought by three class representatives, Mark Fichtner, Siddharth Hariharan, and Daniel Stover (hereafter, "Plaintiffs"). See ECF No. 920. The Settlement provides for $324.5 million in recovery for the class in exchange for release of antitrust claims. A fourth class representative, Michael Devine ("Devine"), has filed an Opposition contending that the settlement amount is inadequate. See ECF No. 934. Plaintiffs have filed a Reply. See ECF No. 938. Plaintiffs, Remaining Defendants, and Devine appeared at a hearing on June 19, 2014. See ECF No. 940. In addition, a number of Class members have submitted letters in support of and in opposition to the proposed settlement. ECF Nos. 914, 949-51. The Court, having considered the briefing, the letters, the arguments presented at the hearing, and the record in this case, DENIES the Motion for Preliminary Approval for the reasons stated below.

I. BACKGROUND AND PROCEDURAL HISTORY

Michael Devine, Mark Fichtner, Siddharth Hariharan, and Daniel Stover, individually and on behalf of a class of all those similarly situated, allege antitrust claims against their former employers, Adobe, Apple, Google, Intel, Intuit Inc. ("Intuit"), Lucasfilm Ltd. ("Lucasfilm"), and Pixar (collectively, "Defendants"). Plaintiffs allege that Defendants entered into an overarching conspiracy through a series of bilateral agreements not to solicit each other's employees in violation of Section 1 of the Sherman Antitrust Act, 15 U.S.C. § 1, and Section 4 of the Clayton Antitrust Act, 15 U.S.C. § 15. Plaintiffs contend that the overarching conspiracy, made up of a series of six bilateral agreements (Pixar-Lucasfilm, Apple-Adobe, Apple-Google, Apple-Pixar, Google-Intuit, and Google-Intel) suppressed wages of Defendants' employees.

The five cases underlying this consolidated action were initially filed in California Superior Court and removed to federal court. See ECF No. 532 at 5. The cases were related by Judge Saundra Brown Armstrong, who also granted a motion to transfer the related actions to the San Jose Division. See ECF Nos. 52, 58. After being assigned to the undersigned judge, the cases were consolidated pursuant to the parties' stipulation. See ECF No. 64. Plaintiffs filed a consolidated complaint on September 23, 2011, see ECF No. 65, which Defendants jointly moved to dismiss, see ECF No. 79. In addition, Lucasfilm filed a separate motion to dismiss on October 17, 2011. See ECF No. 83. The Court granted in part and denied in part the joint motion to dismiss and denied Lucasfilm's separate motion to dismiss. See ECF No. 119.

On October 1, 2012, Plaintiffs filed a motion for class certification. See ECF No. 187. The motion sought certification of a class of all of the seven Defendants' employees or, in the alternative, a narrower class of just technical employees of the seven Defendants. After full briefing and a hearing, the Court denied class certification on April 5, 2013. See ECF No. 382. The Court was concerned that Plaintiffs' documentary evidence and empirical analysis were insufficient to determine that common questions predominated over individual questions with respect to the issue of antitrust impact. See id. at 33. Moreover, the Court expressed concern that there was insufficient analysis in the class certification motion regarding the class of technical employees. Id. at 29. The Court afforded Plaintiffs leave to amend to address the Court's concerns. See id. at 52.

On May 10, 2013, Plaintiffs filed their amended class certification motion, seeking to certify only the narrower class of technical employees. See ECF No. 418. Defendants filed their opposition on June 21, 2013, ECF No. 439, and Plaintiffs filed their reply on July 12, 2013, ECF No. 455. The hearing on the amended motion was set for August 5, 2013.

On July 12 and 30, 2013, after class certification had been initially denied and while an amended motion was pending, Plaintiffs settled with Pixar, Lucasfilm, and Intuit (hereafter, "Settled Defendants"). See ECF Nos. 453, 489. Plaintiffs filed a motion for preliminary approval of the settlements with Settled Defendants on September 21, 2013. See ECF No. 501. No opposition to the motion was filed, and the Court granted the motion on October 30, 2013, following a hearing on October 21, 2013. See ECF No. 540. The Court held a fairness hearing on May 1, 2014, ECF No. 913, and granted final approval of the settlements and accompanying requests for attorneys' fees, costs, and incentive awards over five objections on May 16, 2014, ECF Nos. 915-16. Judgment was entered as to the Settled Defendants on June 20, 2014. ECF No. 947.

After the Settled Defendants settled, this Court certified a class of technical employees of the seven Defendants (hereafter, "the Class") on October 25, 2013 in an 86-page order granting Plaintiffs' amended class certification motion. See ECF No. 532. The Remaining Defendants petitioned the Ninth Circuit to review that order under Federal Rule of Civil Procedure 23(f). After full briefing, including the filing of an amicus brief by the National and California Chambers of Commerce and the National Association of Manufacturing urging the Ninth Circuit to grant review, the Ninth Circuit denied review on January 15, 2014. See ECF No. 594.

Meanwhile, in this Court, the Remaining Defendants filed a total of five motions for summary judgment and filed motions to strike and to exclude the testimony of Plaintiffs' principal expert on antitrust impact and damages, Dr. Edward Leamer, who opined that the total damages to the Class exceeded $3 billion in wages Class members would have earned in the absence of the anti-solicitation agreements.1 The Court denied the motions for summary judgment on March 28, 2014, and on April 4, 2014, denied the motion to exclude Dr. Leamer and denied in large part the motion to strike Dr. Leamer's testimony. ECF Nos. 777, 788.

On April 24, 2014, counsel for Plaintiffs and counsel for Remaining Defendants sent a joint letter to the Court indicating that they had reached a settlement. See ECF No. 900. This settlement was reached two weeks before the Final Pretrial Conference and one month before the trial was set to commence.2: Upon receipt of the joint letter, the Court vacated the trial date and pretrial deadlines and set a schedule for preliminary approval. See ECF No. 904. Shortly after counsel sent the letter, the media disclosed the total amount of the settlement, and this Court received three letters from individuals, not including Devine, objecting to the proposed settlement in response to media reports of the settlement amount.3 See ECF No. 914. On May 22, 2014, in accordance with this Court's schedule, Plaintiffs filed their Motion for Preliminary Approval. See ECF No. 920. Devine filed an Opposition on June 5, 2014.4 See ECF No. 934. Plaintiffs filed a Reply on June 12, 2014. See ECF No. 938. The Court held a hearing on June 19, 2014. See ECF No. 948. After the hearing, the Court received a letter from a Class member in opposition to the proposed settlement and two letters from Class members in support of the proposed settlement. See ECF Nos. 949-51.

The Court must review the fairness of class action settlements under Federal Rule of Civil Procedure 23(e). The Rule states that "[t]he claims, issues, or defenses of a certified class may be settled, voluntarily dismissed, or compromised only with the court's approval." The Rule requires the Court to "direct notice in a reasonable manner to all class members who would be bound by the proposal" and further states that if a settlement "would bind class members, the court may approve it only after a hearing and on finding that it is fair, reasonable, and adequate." Fed. R. Civ. P. 23(e)(1)-(2). The principal purpose of the Court's supervision of class action settlements is to ensure "the agreement is not the product of fraud or overreaching by, or collusion between, the negotiating parties." Officers for Justice v. Civil Serv. Comm'n of City & Cnty. of S.F., 688 F.2d 615, 625 (9th Cir. 1982).

District courts have interpreted Rule 23(e) to require a two-step process for the approval of class action settlements: "the Court first determines whether a proposed class action settlement deserves preliminary approval and then, after notice is given to class members, whether final approval is warranted." Nat'l Rural Telecomms. Coop. v. DIRECTV, Inc., 221 F.R.D. 523, 525 (C.D. Cal. 2004). At the final approval stage, the Ninth Circuit has stated that "[a]ssessing a settlement proposal requires the district court to balance a number of factors: the strength of the plaintiffs' case; the risk, expense, complexity, and likely duration of further litigation; the risk of maintaining class action status throughout the trial; the amount offered in settlement; the extent of discovery completed and the stage of the proceedings; the experience and views of counsel; the presence of a governmental participant; and the reaction of the class members to the proposed settlement." Hanlon v. Chrysler Corp., 150 F.3d 1011, 1026 (9th Cir. 1998).

In contrast to these well-established, non-exhaustive factors for final approval, there is relatively scant appellate authority regarding the standard that a district court must apply in reviewing a settlement at the preliminary approval stage. Some district courts, echoing commentators, have stated that the relevant inquiry is whether the settlement "falls within the range of possible approval" or "within the range of reasonableness." In re Tableware Antitrust Litig., 484 F. Supp. 2d 1078, 1079 (N.D. Cal. 2007); see also Cordy v. USS-Posco Indus., No. 12-553, 2013 WL 4028627, at *3 (N.D. Cal. Aug. 1, 2013) ("Preliminary approval of a settlement and notice to the proposed class is appropriate if the proposed settlement appears to be the product of serious, informed, non-collusive negotiations, has no obvious deficiencies, does not improperly grant preferential treatment to class representatives or segments of the class, and falls with the range of possible approval." (internal quotation marks omitted)). To undertake this analysis, the Court "must consider plaintiffs' expected recovery balanced against the value of the settlement offer." In re Nat'l Football League Players' Concussion Injury Litig., 961 F. Supp. 2d 708, 714 (E.D. Pa. 2014) (internal quotation marks omitted).

III. DISCUSSION

Pursuant to the terms of the instant settlement, Class members who have not already opted out and who do not opt out will relinquish their rights to file suit against the Remaining Defendants for the claims at issue in this case. In exchange, Remaining Defendants will pay a total of $324.5 million, of which Plaintiffs' counsel may seek up to 25% (approximately $81 million) in attorneys' fees, $1.2 million in costs, and $80,000 per class representative in incentive payments. In addition, the settlement allows Remaining Defendants a pro rata reduction in the total amount they must pay if more than 4% of Class members opt out after receiving notice.5 Class members would receive an average of approximately $3,7506 from the instant settlement if the Court were to grant all requested deductions and there were no further opt-outs.7

The Court finds the total settlement amount falls below the range of reasonableness. The Court is concerned that Class members recover less on a proportional basis from the instant settlement with Remaining Defendants than from the settlement with the Settled Defendants a year ago, despite the fact that the case has progressed consistently in the Class's favor since then. Counsel's sole explanation for this reduced figure is that there are weaknesses in Plaintiffs' case such that the Class faces a substantial risk of non-recovery. However, that risk existed and was even greater when Plaintiffs settled with the Settled Defendants a year ago, when class certification had been denied.

The Court begins by comparing the instant settlement with Remaining Defendants to the settlements with the Settled Defendants, in light of the facts that existed at the time each settlement was reached. The Court then discusses the relative strengths and weaknesses of Plaintiffs' case to assess the reasonableness of the instant settlement.

A. Comparison to the Initial Settlements

1. Comparing the Settlement Amounts

The Court finds that the settlements with the Settled Defendants provide a useful benchmark against which to analyze the reasonableness of the instant settlement. The settlements with the Settled Defendants led to a fund totaling $20 million. See ECF No. 915 at 3. In approving the settlements, the Court relied upon the fact that the Settled Defendants employed 8% of Class members and paid out 5% of the total Class compensation during the Class period. See ECF No. 539 at 16:20-22 (Plaintiffs' counsel's explanation at the preliminary approval hearing with the Settled Defendants that the 5% figure "giv[es] you a sense of how big a slice of the case this settlement is relative to the rest of the case"). If Remaining Defendants were to settle at the same (or higher) rate as the Settled Defendants, Remaining Defendants' settlement fund would need to total at least $380 million. This number results from the fact that Remaining Defendants paid out 95% of the Class compensation during the Class period, while Settled Defendants paid only 5% of the Class compensation during the Class period.8

At the hearing on the instant Motion, counsel for Remaining Defendants suggested that the relevant benchmark is not total Class compensation, but rather is total Class membership. This would result in a benchmark figure for the Remaining Defendants of $230 million0. At a minimum, counsel suggested, the Court should compare the settlement amount to a range of $230 million to $380 million, within which the instant settlement falls. The Court rejects counsel's suggestion, which is contrary to the record. Counsel has provided no basis for why the number of Class members employed by each Defendant is a relevant metric. To the contrary, the relevant inquiry has always been total Class compensation. For example, in both of the settlements with the Settled Defendants and in the instant settlement, the Plans of Allocation call for determining each individual Class member's pay out by dividing the Class member's compensation during the Class period by the total Class compensation during the Class period. ECF No. 809 at 6 (noting that the denominator in the plan of allocation in the settlements with the Settled Defendants is the "total of base salaries paid to all approved Claimants in class positions during the Class period"); ECF No. 920 at 22 (same in the instant settlement); see also ECF No. 539 at 16:20-22 (Plaintiffs' counsel's statement that percent of the total Class compensation was relevant for benchmarking the settlements with the Settled Defendants to the rest of the case). At no point in the record has the percentage of Class membership employed by each Defendant ever been the relevant factor for determining damages exposure. Accordingly, the Court rejects the metric proposed by counsel for Remaining Defendants. Using the Settled Defendants' settlements as a yardstick, the appropriate benchmark settlement for the Remaining Defendants would be at least $380 million, more than $50 million greater than what the instant settlement provides.

Counsel for Remaining Defendants also suggested that benchmarking against the initial settlements would be inappropriate because the magnitude of the settlement numbers for Remaining Defendants dwarfs the numbers at issue in the Settled Defendants' settlements. This argument is premised on the idea that Defendants who caused more damage to the Class and who benefited more by suppressing a greater portion of class compensation should have to pay less than Defendants who caused less damage and who benefited less from the allegedly wrongful conduct. This argument is unpersuasive. Remaining Defendants are alleged to have received 95% of the benefit of the anti-solicitation agreements and to have caused 95% of the harm suffered by the Class in terms of lost compensation. Therefore, Remaining Defendants should have to pay at least 95% of the damages, which, under the instant settlement, they would not.

The Court also notes that had Plaintiffs prevailed at trial on their more than $3 billion damages claim, antitrust law provides for automatic trebling, see 15 U.S.C. § 15(a), so the total damages award could potentially have exceeded $9 billion. While the Ninth Circuit has not determined whether settlement amounts in antitrust cases must be compared to the single damages award requested by Plaintiffs or the automatically trebled damages amount, see Rodriguez v. W. Publ'g Corp., 563 F.3d 948, 964-65 (9th Cir. 2009), the instant settlement would lead to a total recovery of 11.29% of the single damages proposed by Plaintiffs' expert or 3.76% of the treble damages. Specifically, Dr. Leamer has calculated the total damages to the Class resulting from Defendants' allegedly unlawful conduct as $3.05 billion. See ECF No. 856-10. If the Court approves the instant settlements, the total settlements with all Defendants would be $344.5 million. This total would amount to 11.29% of the single damages that Dr. Leamer opines the Class suffered or 3.76% if Dr. Leamer's damages figure had been trebled.

2. Relative Procedural Posture

The discount that Remaining Defendants have received vis-a-vis the Settled Defendants is particularly troubling in light of the changes in the procedural posture of the case between the two settlements, changes that the Court would expect to have increased, rather than decreased, Plaintiffs' bargaining power. Specifically, at the time the Settled Defendants settled, Plaintiffs were at a particularly weak point in their case. Though Plaintiffs had survived Defendants' motion to dismiss, Plaintiffs' motion for class certification had been denied, albeit without prejudice. Plaintiffs had re-briefed the class certification motion, but had no class certification ruling in their favor at the time they settled with the Settled Defendants. If the Court ultimately granted certification, Plaintiffs also did not know whether the Ninth Circuit would grant Federal Rule of Civil Procedure 23(f) review and reverse the certification. Accordingly, at that point, Defendants had significant leverage.

In contrast, the procedural posture of the case swung dramatically in Plaintiffs' favor after the initial settlements were reached. Specifically, the Court certified the Class over the vigorous objections of Defendants. In the 86-page order granting class certification, the Court repeatedly referred to Plaintiffs' evidence as "substantial" and "extensive," and the Court stated that it "could not identify a case at the class certification stage with the level of documentary evidence Plaintiffs have presented in the instant case." ECF No. 531 at 69. Thereafter, the Ninth Circuit denied Defendants' request to review the class certification order under Federal Rule of Civil Procedure 23(f). This Court also denied Defendants' five motions for summary judgment and denied Defendants' motion to exclude Plaintiffs' principal expert on antitrust impact and damages. The instant settlement was reached a mere two weeks before the final pretrial conference and one month before a trial at which damaging evidence regarding Defendants would have been presented.

In sum, Plaintiffs were in a much stronger position at the time of the instant settlement—after the Class had been certified, appellate review of class certification had been denied, and Defendants' dispositive motions and motion to exclude Dr. Leamer's testimony had been denied—than they were at the time of the settlements with the Settled Defendants, when class certification had been denied. This shift in the procedural posture, which the Court would expect to have increased Plaintiffs' bargaining power, makes the more recent settlements for a proportionally lower amount even more troubling.

B. Strength of Plaintiffs' Case

The Court now turns to the strength of Plaintiffs' case against the Remaining Defendants to evaluate the reasonableness of the settlement.

At the hearing on the instant Motion, Plaintiffs' counsel contended that one of the reasons the instant settlement was proportionally lower than the previous settlements is that the documentary evidence against the Settled Defendants (particularly, Lucasfilm and Pixar) is more compelling than the documentary evidence against the Remaining Defendants. As an initial matter, the Court notes that relevant evidence regarding the Settled Defendants would be admissible at a trial against Remaining Defendants because Plaintiffs allege an overarching conspiracy that included all Defendants. Accordingly, evidence regarding the role of Lucasfilm and Pixar in the creation of and the intended effect of the overarching conspiracy would be admissible.

Nonetheless, the Court notes that Plaintiffs are correct that there are particularly clear statements from Lucasfilm and Pixar executives regarding the nature and goals of the alleged conspiracy. Specifically, Edward Catmull (Pixar President) conceded in his deposition that anti-solicitation agreements were in place because solicitation "messes up the pay structure." ECF No. 431-9 at 81. Similarly, George Lucas (former Lucasfilm Chairman of the Board and CEO) stated, "we cannot get into a bidding war with other companies because we don't have the margins for that sort of thing." ECF No. 749-23 at 9.

However, there is equally compelling evidence that comes from the documents of the Remaining Defendants. This is particularly true for Google and Apple, the executives of which extensively discussed and enforced the anti-solicitation agreements. Specifically, as discussed in extensive detail in this Court's previous orders, Steve Jobs (Co-Founder, Former Chairman, and Former CEO of Apple, Former CEO of Pixar), Eric Schmidt (Google Executive Chairman, Member of the Board of Directors, and former CEO), and Bill Campbell (Chairman of Intuit Board of Directors, Co-Lead Director of Apple, and advisor to Google) were key players in creating and enforcing the anti-solicitation agreements. The Court now turns to the evidence against the Remaining Defendants that the finder of fact is likely to find compelling.

There is substantial and compelling evidence that Steve Jobs (Co-Founder, Former Chairman, and Former CEO of Apple, Former CEO of Pixar) was a, if not the, central figure in the alleged conspiracy. Several witnesses, in their depositions, testified to Mr. Jobs' role in the anti-solicitation agreements. For example, Eric Schmidt (Google Executive Chairman, Member of the Board of Directors, and former CEO) stated that Mr. Jobs "believed that you should not be hiring each others', you know, technical people" and that "it was inappropriate in [Mr. Jobs'] view for us to be calling in and hiring people." ECF No. 819-12 at 77. Edward Catmull (Pixar President) stated that Mr. Jobs "was very adamant about protecting his employee force." ECF No. 431-9 at 97. Sergey Brin (Google Co-Founder) testified that "I think Mr. Jobs' view was that people shouldn't piss him off. And I think that things that pissed him off were—would be hiring, you know—whatever." ECF No. 639-1 at 112. There would thus be ample evidence Mr. Jobs was involved in expanding the original anti-solicitation agreement between Lucasfilm and Pixar to the other Defendants in this case. After the agreements were extended, Mr. Jobs played a central role in enforcing these agreements. Four particular sets of evidence are likely to be compelling to the fact-finder.

First, after hearing that Google was trying to recruit employees from Apple's Safari team, Mr. Jobs threatened Mr. Brin, stating, as Mr. Brin recounted, "if you hire a single one of these people that means war." ECF No. 833-15.9 In an email to Google's Executive Management Team as well as Bill Campbell (Chairman of Intuit Board of Directors, Co-Lead Director of Apple, and advisor to Google), Mr. Brin advised: "lets [sic] not make any new offers or contact new people at Apple until we have had a chance to discuss." Id. Mr. Campbell then wrote to Mr. Jobs: "Eric [Schmidt] told me that he got directly involved and firmly stopped all efforts to recruit anyone from Apple." ECF No. 746-5. As Mr. Brin testified in his deposition, "Eric made a—you know, a—you know, at least some kind of—had a conversation with Bill to relate to Steve to calm him down." ECF No. 639-1 at 61. As Mr. Schmidt put it, "Steve was unhappy, and Steve's unhappiness absolutely influenced the change we made in recruiting practice." ECF No. 819-12 at 21. Danielle Lambert (Apple's head of Human Resources) reciprocated to maintain Apple's end of the anti-solicitation agreements, instructing Apple recruiters: "Please add Google to your 'hands-off list. We recently agreed not to recruit from one another so if you hear of any recruiting they are doing against us, please be sure to let me know." ECF No. 746-15.

Second, other Defendants' CEOs maintained the anti-solicitation agreements out of fear of and deference to Mr. Jobs. For example, in 2005, when considering whether to enter into an anti-solicitation agreement with Apple, Bruce Chizen (former Adobe CEO), expressed concerns about the loss of "top talent" if Adobe did not enter into an anti-solicitation agreement with Apple, stating, "if I tell Steve it's open season (other than senior managers), he will deliberately poach Adobe just to prove a point. Knowing Steve, he will go after some of our top Mac talent like Chris Cox and he will do it in a way in which they will be enticed to come (extraordinary packages and Steve wooing)."10 ECF No. 297-15.

This was the genesis of the Apple-Adobe agreement. Specifically, after Mr. Jobs complained to Mr. Chizen on May 26, 2005 that Adobe was recruiting Apple employees, ECF No. 291-17, Mr. Chizen responded by saying, "I thought we agreed not to recruit any senior level employees . . . . I would propose we keep it that way. Open to discuss. It would be good to agree." Id. Mr. Jobs was not satisfied, and replied by threatening to send Apple recruiters after Adobe's employees: "OK, I'll tell our recruiters that they are free to approach any Adobe employee who is not a Sr. Director or VP. Am I understanding your position correctly?" Id. Mr. Chizen immediately gave in: "I'd rather agree NOT to actively solicit any employee from either company . . . . If you are in agreement I will let my folks know." Id. (emphasis in original). The next day, Theresa Townsley (Adobe Vice President Human Resources) announced to her recruiting team, "Bruce and Steve Jobs have an agreement that we are not to solicit ANY Apple employees, and vice versa." ECF No. 291-18 (emphasis in original). Adobe then placed Apple on its "[c]ompanies that are off limits" list, which instructed Adobe employees not to cold call Apple employees. ECF No. 291-11.

Google took even more drastic actions in response to Mr. Jobs. For example, when a recruiter from Google's engineering team contacted an Apple employee in 2007, Mr. Jobs forwarded the message to Mr. Schmidt and stated, "I would be very pleased if your recruiting department would stop doing this." ECF No. 291-23. Google responded by making a "public example" out of the recruiter and "terminat[ing] [the recruiter] within the hour." Id. The aim of this public spectacle was to "(hopefully) prevent future occurrences." Id. Once the recruiter was terminated, Mr. Schmidt emailed Mr. Jobs, apologizing and informing Mr. Jobs that the recruiter had been terminated. Mr. Jobs forwarded Mr. Schmidt's email to an Apple human resources official and stated merely, ":)." ECF No. 746-9.

A year prior to this termination, Google similarly took seriously Mr. Jobs' concerns. Specifically, in 2006, Mr. Jobs emailed Mr. Schmidt and said, "I am told that Googles [sic] new cell phone software group is relentlessly recruiting in our iPod group. If this is indeed true, can you put a stop to it?" ECF No. 291-24 at 3. After Mr. Schmidt forwarded this to Human Resources professionals at Google, Arnnon Geshuri (Google Recruiting Director) prepared a detailed report stating that an extensive investigation did not find a breach of the anti-solicitation agreement.

Similarly, in 2006, Google scrapped plans to open a Google engineering center in Paris after a Google executive emailed Mr. Jobs to ask whether Google could hire three former Apple engineers to work at the prospective facility, and Mr. Jobs responded "[w]e'd strongly prefer that you not hire these guys." ECF No. 814-2. The whole interaction began with Google's request to Steve Jobs for permission to hire Jean-Marie Hullot, an Apple engineer. The record is not clear whether Mr. Hullot was a current or former Apple employee. A Google executive contacted Steve Jobs to ask whether Google could make an offer to Mr. Hullot, and Mr. Jobs did not timely respond to the Google executive's request. At this point, the Google executive turned to Intuit's Board Chairman Bill Campbell as a potential ambassador from Google to Mr. Jobs. Specifically, the Google executive noted that Mr. Campbell "is on the board at Apple and Google, so Steve will probably return his call." ECF No. 428-6. The same day that Mr. Campbell reached out to Mr. Jobs, Mr. Jobs responded to the Google executive, seeking more information on what exactly the Apple engineer would be working. ECF No. 428-9. Once Mr. Jobs was satisfied, he stated that the hire "would be fine with me." Id. However, two weeks later, when Mr. Hullot and a Google executive sought Mr. Jobs' permission to hire four of Mr. Hullot's former Apple colleagues (three were former Apple employees and one had given notice of impending departure from Apple), Mr. Jobs promptly responded, indicating that the hires would not be acceptable. ECF No. 428-9. Google promptly scrapped the plan, and the Google executive responded deferentially to Mr. Jobs, stating, "Steve, Based on your strong preference that we not hire the ex-Apple engineers, Jean-Marie and I decided not to open a Google Paris engineering center." Id. The Google executive also forwarded the email thread to Mr. Brin, Larry Page (Google Co-Founder), and Mr. Campbell. Id.

Third, Mr. Jobs attempted (unsuccessfully) to expand the anti-solicitation agreements to Palm, even threatening litigation. Specifically, Mr. Jobs called Edward Colligan (former President and CEO of Palm) to ask Mr. Colligan to enter into an anti-solicitation agreement and threatened patent litigation against Palm if Palm refused to do so. ECF No. 293 ¶¶ 6-8. Mr. Colligan responded via email, and told Mr. Jobs that Mr. Jobs' "proposal that we agree that neither company will hire the other's employees, regardless of the individual's desires, is not only wrong, it is likely illegal." Id. at 4-5. Mr. Colligan went on to say that, "We can't dictate where someone will work, nor should we try. I can't deny people who elect to pursue their livelihood at Palm the right to do so simply because they now work for Apple, and I wouldn't want you to do that to current Palm employees." Id. at 5. Finally, Mr. Colligan wrote that "[t]hreatening Palm with a patent lawsuit in response to a decision by one employee to leave Apple is just out of line. A lawsuit would not serve either of our interests, and will not stop employees from migrating between our companies . . . . We will both just end up paying a lot of lawyers a lot of money." Id. at 5-6. Mr. Jobs wrote the following back to Mr. Colligan: "This is not satisfactory to Apple." Id. at 8. Mr. Jobs went on to write that "I'm sure you realize the asymmetry in the financial resources of our respective companies when you say: 'we will both just end up paying a lot of lawyers a lot of money.'" Id. Mr. Jobs concluded: "My advice is to take a look at our patent portfolio before you make a final decision here." Id.

Fourth, Apple's documents provide strong support for Plaintiffs' theory of impact, namely that rigid wage structures and internal equity concerns would have led Defendants to engage in structural changes to compensation structures to mitigate the competitive threat that solicitation would have posed. Apple's compensation data shows that, for each year in the Class period, Apple had a "job structure system," which included categorizing and compensating its workforce according to a discrete set of company-wide job levels assigned to all salaried employees and four associated sets of base salary ranges applicable to "Top," "Major," "National," and "Small" geographic markets. ECF No. 745-7 at 14-15, 52-53; ECF No.517-16 ¶¶ 6, 10 & Ex. B. Every salary range had a "min," "mid," and "max" figure. See id. Apple also created a Human Resources and recruiting tool called "Merlin," which was an internal system for tracking employee records and performance, and required managers to grade employees at one of four pre-set levels. See ECF No. 749-6 at 142-43, 145-46; ECF No. 749-11 at 52-53; ECF No. 749-12 at 33. As explained by Tony Fadell (former Apple Senior Vice President, iPod Division, and advisor to Steve Jobs), Merlin "would say, this is the employee, this is the level, here are the salary ranges, and through that tool we were then—we understood what the boundaries were." ECF No. 749-11 at 53. Going outside these prescribed "guidelines" also required extra approval. ECF No. 749-7 at 217; ECF No. 749-11 at 53 ("And if we were to go outside of that, then we would have to pull in a bunch of people to then approve anything outside of that range.").

Concerns about internal equity also permeated Apple's compensation program. Steven Burmeister (Apple Senior Director of Compensation) testified that internal equity—which Mr. Burmeister defined as the notion of whether an employee's compensation is "fair based on the individual's contribution relative to the other employees in your group, or across your organization"—inheres in some, "if not all," of the guidelines that managers consider in determining starting salaries. ECF No. 745-7 at 61-64; ECF No. 753-12. In fact, as explained by Patrick Burke (former Apple Technical Recruiter and Staffing Manager), when hiring a new employee at Apple, "compar[ing] the candidate" to the other people on the team they would join "was the biggest determining factor on what salary we gave." ECF No. 745-6 at 279.

The evidence against Google is equally compelling. Email evidence reveals that Eric Schmidt (Google Executive Chairman, Member of the Board of Directors, and former CEO) terminated at least two recruiters for violations of anti-solicitation agreements, and threatened to terminate more. As discussed above, there is direct evidence that Mr. Schmidt terminated a recruiter at Steve Jobs' behest after the recruiter attempted to solicit an Apple employee. Moreover, in an email to Bill Campbell (Chairman of Intuit Board of Directors, Co-Lead Director of Apple, and advisor to Google), Mr. Schmidt indicated that he directed a for-cause termination of another Google recruiter, who had attempted to recruit an executive of eBay, which was on Google's do-not-cold-call list. ECF No. 814-14. Finally, as discussed in more detail below, Mr. Schmidt informed Paul Otellini (CEO of Intel and Member of the Google Board of Directors) that Mr. Schmidt would terminate any recruiter who recruited Intel employees.

Furthermore, Google maintained a formal "Do Not Call" list, which grouped together Apple, Intel, and Intuit and was approved by top executives. ECF No. 291-28. The list also included other companies, such as Genentech, Paypal, and eBay. Id. A draft of the "Do Not Call" list was presented to Google's Executive Management Group, a committee consisting of Google's senior executives, including Mr. Schmidt, Larry Page (Google Co-Founder), Sergey Brin (Google Co-Founder), and Shona Brown (former Google Senior Vice President of Business Operations). ECF No. 291-26. Mr. Schmidt approved the list. See id.; see also ECF No. 291-27 (email from Mr. Schmidt stating: "This looks very good."). Moreover, there is evidence that Google executives knew that the anti-solicitation agreements could lead to legal troubles, but nevertheless proceeded with the agreements. When Ms. Brown asked Mr. Schmidt whether he had any concerns with sharing information regarding the "Do Not Call" list with Google's competitors, Mr. Schmidt responded that he preferred that it be shared "verbally[,] since I don't want to create a paper trail over which we can be sued later?" ECF No. 291-40. Ms. Brown responded: "makes sense to do orally. i agree." Id.

Google's response to competition from Facebook also demonstrates the impact of the alleged conspiracy. Google had long been concerned about Facebook hiring's effect on retention. For example, in an email to top Google executives, Mr. Brin in 2007 stated that "the facebook phenomenon creates a real retention problem." ECF No. 814-4. A month later, Mr. Brin announced a policy of making counteroffers within one hour to any Google employee who received an offer from Facebook. ECF No. 963-2.

In March 2008, Arnnon Geshuri (Google Recruiting Director) discovered that non-party Facebook had been cold calling into Google's Site Reliability Engineering ("SRE") team. Mr. Geshuri's first response was to suggest contacting Sheryl Sandberg (Chief Operating Officer for non-party Facebook) in an effort to "ask her to put a stop to the targeted sourcing effort directed at our SRE team" and "to consider establishing a mutual 'Do Not Call' agreement that specifies that we will not cold-call into each other." ECF No. 963-3. Mr. Geshuri also suggested "look[ing] internally and review[ing] the attrition rate for the SRE group," stating, "[w]e may want to consider additional individual retention incentives or team incentives to keep attrition as low as possible in SRE." Id. (emphasis added). Finally, an alternative suggestion was to "[s]tart an aggressive campaign to call into their company and go after their folks—no holds barred. We would be unrelenting and a force of nature." Id. In response, Bill Campbell (Chairman of Intuit Board of Directors, Co-Lead Director of Apple, and advisor to Google), in his capacity as an advisor to Google, suggested "Who should contact Sheryl Sandberg to get a cease fire? We have to get a truce." Id. Facebook refused.

In 2010, Google altered its salary structure with a "Big Bang" in response to Facebook's hiring, which provides additional support for Plaintiffs' theory of antitrust impact. Specifically, after a period in which Google lost a significant number of employees to Facebook, Google began to study Facebook's solicitation of Google employees. ECF No. 190 ¶ 109. One month after beginning this study, Google announced its "Big Bang," which involved an increase to the base salary of all of its salaried employees by 10% and provided an immediate cash bonus of $1,000 to all employees. ECF No. 296-18. Laszlo Bock (Google Senior Vice President of People Operations) explained that the rationale for the Big Bang included: (1) being "responsive to rising attrition;" (2) supporting higher retention because "higher salaries generate higher fixed costs;" and (3) being "very strategic because start-ups don't have the cash flow to match, and big companies are (a) too worried about internal equity and scalability to do this and (b) don't have the margins to do this." ECF No. 296-20.

Other Google documents provide further evidence of Plaintiffs' theory of antitrust impact. For example, Google's Chief Culture Officer stated that "[c]old calling into companies to recruit is to be expected unless they're on our 'don't call' list." ECF No. 291-41. Moreover, Google found that although referrals were the largest source of hires, "agencies and passively sourced candidates offer[ed] the highest yield." ECF No. 780-8. The spread of information between employees had there been active solicitations—which is central to Plaintiffs' theory of impact—is also demonstrated in Google's evidence. For example, one Google employee states that "[i]t's impossible to keep something like this a secret. The people getting counter offers talk, not just to Googlers and ex-Googlers, but also to the competitors where they received their offers (in the hopes of improving them), and those competitors talk too, using it as a tool to recruit more Googlers." ECF No. 296-23.

The wage structure and internal equity concerns at Google also support Plaintiffs' theory of impact. Google had many job families, many grades within job families, and many job titles within grades. See, e.g., ECF No. 298-7, ECF No. 298-8; see also Cisneros Decl., Ex. S (Brown Depo.) at 74-76 (discussing salary ranges utilized by Google); ECF No. 780-4 at 25-26 (testifying that Google's 2007 salary ranges had generally the same structure as the 2004 salary ranges). Throughout the Class period, Google utilized salary ranges and pay bands with minima and maxima and either means or medians. ECF No. 958-1 ¶ 66; see ECF No. 427-3 at 15-17. As explained by Shona Brown (former Google Senior Vice President, Business Operations), "if you discussed a specific role [at Google], you could understand that role was at a specific level on a certain job ladder." ECF No. 427-3 at 27-28; ECF No. 745-11. Frank Wagner (Google Director of Compensation) testified that he could locate the target salary range for jobs at Google through an internal company website. See ECF No. 780-4 at 31-32 ("Q: And if you wanted to identify what the target salary would be for a certain job within a certain grade, could you go online or go to some place . . . and pull up what that was for that job family and that grade? . . . A: Yes."). Moreover, Google considered internal equity to be an important goal. Google utilized a salary algorithm in part for the purpose of "[e]nsur[ing] internal equity by managing salaries within a reasonable range." ECF No. 814-19. Furthermore, because Google "strive[d] to achieve fairness in overall salary distribution," "high performers with low salaries [would] get larger percentage increases than high performers with high salaries." ECF No. 817-1 at 15.

In addition, Google analyzed and compared its equity compensation to Apple, Intel, Adobe, and Intuit, among other companies, each of which it designated as a "peer company" based on meeting criteria such as being a "high-tech company," a "high-growth company," and a "key labor market competitor." ECF No. 773-1. In 2007, based in part on an analysis of Google as compared to its peer companies, Mr. Bock and Dave Rolefson (Google Equity Compensation Manager) wrote that "[o]ur biggest labor market competitors are significantly exceeding their own guidelines to beat Google for talent." Id.

Finally, Google's own documents undermine Defendants' principal theory of lack of antitrust impact, that compensation decisions would be one off and not classwide. Alan Eustace (Google Senior Vice President) commented on concerns regarding competition for workers and Google's approach to counteroffers by noting that, "it sometimes makes sense to make changes in compensation, even if it introduces discontinuities in your current comp, to save your best people, and send a message to the hiring company that we'll fight for our best people." ECF No. 296-23. Because recruiting "a few really good people" could inspire "many, many others [to] follow," Mr. Eustace concluded, "[y]ou can't afford to be a rich target for other companies." Id. According to him, the "long-term . . . right approach is not to deal with these situations as one-offs but to have a systematic approach to compensation that makes it very difficult for anyone to get a better offer." Id. (emphasis added).

Google's impact on the labor market before the anti-solicitation agreements was best summarized by Meg Whitman (former CEO of eBay) who called Mr. Schmidt "to talk about [Google's] hiring practices." ECF No. 814-15. As Eric Schmidt told Google's senior executives, Ms. Whitman said "Google is the talk of the valley because [you] are driving up salaries across the board." Id. A year after this conversation, Google added eBay to its do-not-cold-call list. ECF No. 291-28.

There is also compelling evidence against Intel. Google reacted to requests regarding enforcement of the anti-solicitation agreement made by Intel executives similarly to Google's reaction to Steve Jobs' request to enforce the agreements discussed above. For example, after Paul Otellini (CEO of Intel and Member of the Google Board of Directors) received an internal complaint regarding Google's successful recruiting efforts of Intel's technical employees on September 26, 2007, ECF No. 188-8 ("Paul, I am losing so many people to Google . . . . We are countering but thought you should know."), Mr. Otellini forwarded the email to Eric Schmidt (Google Executive Chairman, Member of the Board of Directors, and former CEO) and stated "Eric, can you pls help here???" Id. Mr. Schmidt obliged and forwarded the email to his recruiting team, who prepared a report for Mr. Schmidt on Google's activities. ECF No. 291-34. The next day, Mr. Schmidt replied to Mr. Otellini, "If we find that a recruiter called into Intel, we will terminate the recruiter," the same remedy afforded to violations of the Apple-Google agreement. ECF No. 531 at 37. In another email to Mr. Schmidt, Mr. Otellini stated, "Sorry to bother you again on this topic, but my guys are very troubled by Google continuing to recruit our key players." See ECF No. 428-8.

Moreover, Mr. Otellini was aware that the anti-solicitation agreement could be legally troublesome. Specifically, Mr. Otellini stated in an email to another Intel executive regarding the Google-Intel agreement: "Let me clarify. We have nothing signed. We have a handshake 'no recruit' between eric and myself. I would not like this broadly known." Id.

Furthermore, there is evidence that Mr. Otellini knew of the anti-solicitation agreements to which Intel was not a party. Specifically, both Sergey Brin (Google Co-Founder) and Mr. Schmidt of Google testified that they would have told Mr. Otellini that Google had an anti-solicitation agreement with Apple. ECF No. 639-1 at 74:15 ("I'm sure that we would have mentioned it[.]"); ECF No. 819-12 at 60 ("I'm sure I spoke with Paul about this at some point."). Intel's own expert testified that Mr. Otellini was likely aware of Google's other bilateral agreements by virtue of Mr. Otellini's membership on Google's board. ECF No. 771 at 4. The fact that Intel was added to Google's do-not-cold-call list on the same day that Apple was added further suggests Intel's participation in an overarching conspiracy. ECF No. 291-28.

Additionally, notwithstanding the fact that Intel and Google were competitors for talent, Mr. Otellini "lifted from Google" a Google document discussing the bonus plans of peer companies including Apple and Intel. Cisneros Decl., Ex. 463. True competitors for talent would not likely share such sensitive bonus information absent agreements not to compete.

Moreover, key documents related to antitrust impact also implicate Intel. Specifically, Intel recognized the importance of cold calling and stated in its "Complete Guide to Sourcing" that "[Cold] [c]alling candidates is one of the most efficient and effective ways to recruit." ECF No. 296-22. Intel also benchmarked compensation against other "tech companies generally considered comparable to Intel," which Intel defined as a "[b]lend of semiconductor, software, networking, communications, and diversified computer companies." ECF No. 754-2. According to Intel, in 2007, these comparable companies included Apple and Google. Id. These documents suggest, as Plaintiffs contend, that the anti-solicitation agreements led to structural, rather than individual depression, of Class members' wages.

Furthermore, Intel had a "compensation structure," with job grades and job classifications. See ECF No. 745-13 at 73 ("[W]e break jobs into one of three categories—job families, we call them—R&D, tech, and nontech, there's a lot more . . . ."). The company assigned employees to a grade level based on their skills and experience. ECF No. 745-11 at 23; see also ECF No. 749-17 at 45 (explaining that everyone at Intel is assigned a "classification" similar to a job grade). Intel standardized its salary ranges throughout the company; each range applied to multiple jobs, and most jobs spanned multiple salary grades. ECF No. 745-16 at 59. Intel further broke down its salary ranges into quartiles, and compensation at Intel followed "a bell-curve distribution, where most of the employees are in the middle quartiles, and a much smaller percentage are in the bottom and top quartiles." Id. at 62-63.

Intel also used a software tool to provide guidance to managers about an employee's pay range which would also take into account market reference ranges and merit. ECF No. 758-9. As explained by Randall Goodwin (Intel Technology Development Manager), "[i]f the tool recommended something and we thought we wanted to make a proposed change that was outside its guidelines, we would write some justification." ECF No. 749-15 at 52. Similarly, Intel regularly ran reports showing the salary range distribution of its employees. ECF No. 749-16 at 64.

The evidence also supports the rigidity of Intel's wage structure. For example, in a 2004 Human Resources presentation, Intel states that, although "[c]ompensation differentiation is desired by Intel's Meritocracy philosophy," "short and long term high performer differentiation is questionable." ECF No. 758-10 at 13. Indeed, Intel notes that "[l]ack of differentiation has existed historically based on an analysis of '99 data." Id. at 19. As key "[v]ulnerability [c]hallenges," Intel identifies: (1) "[m]anagers (in)ability to distinguish at [f]ocal"—"actual merit increases are significantly reduced from system generated increases," "[l]ong term threat to retention of key players"; (2) "[l]ittle to no actual pay differentiation for HPs [high performers]"; and (3) "[n]o explicit strategy to differentiate." Id. at 24 (emphasis added).

In addition, Intel used internal equity "to determine wage rates for new hires and current employees that correspond to each job's relative value to Intel." ECF No. 749-16 at 210-11; ECF No. 961-5. To assist in that process, Intel used a tool that generates an "Internal Equity Report" when making offers to new employees. ECF No. 749-16 at 212-13. In the words of Ogden Reid (Intel Director of Compensation and Benefits), "[m]uch of our culture screams egalitarianism . . . . While we play lip service to meritocracy, we really believe more in treating everyone the same within broad bands." ECF No. 769-8.

An Intel human resources document from 2002—prior to the anti-solicitation agreements—recognized "continuing inequities in the alignment of base salaries/EB targets between hired and acquired Intel employees" and "parallel issues relating to accurate job grading within these two populations." ECF No. 750-15. In response, Intel planned to: (1) "Review exempt job grade assignments for job families with 'critical skills.' Make adjustments, as appropriate"; and (2) "Validate perception of inequities . . . . Scope impact to employees. Recommend adjustments, as appropriate." Id. An Intel human resources document confirms that, in or around 2004, "[n]ew hire salary premiums drove salary range adjustment." ECF No. 298-5 at 7 (emphasis added).

Intel would "match an Intel job code in grade to a market survey job code in grade," ECF No. 749-16 at 89, and use that as part of the process for determining its "own focal process or pay delivery," id. at 23. If job codes fell below the midpoint, plus or minus a certain percent, the company made "special market adjustment[s]." Id. at 90.

Evidence from Adobe also suggests that Adobe was aware of the impact of its anti-solicitation agreements. Adobe personnel recognized that "Apple would be a great target to look into" for the purpose of recruiting, but knew that they could not do so because, "[u]nfortunately, Bruce [Chizen (former Adobe CEO)] and Apple CEO Steve Jobs have a gentleman's agreement not to poach each other's talent." ECF No. 291-13. Adobe executives were also part and parcel of the group of high-ranking executives that entered into, enforced, and attempted to expand the anti-solicitation agreements. Specifically, Mr. Chizen, in response to discovering that Apple was recruiting employees of Macromedia (a separate entity that Adobe would later acquire), helped ensure, through an email to Mr. Jobs, that Apple would honor Apple's pre-existing anti-solicitation agreements with both Adobe and Macromedia after Adobe's acquisition of Macromedia. ECF No. 608-3 at 50.

Adobe viewed Google and Apple to be among its top competitors for talent and expressed concern about whether Adobe was "winning the talent war." ECF No. 296-3. Adobe further considered itself in a "six-horse race from a benefits standpoint," which included Google, Apple, and Intuit as among the other "horses." See ECF No. 296-4. In 2008, Adobe benchmarked its compensation against nine companies including Google, Apple, and Intel. ECF No. 296-4; cf. ECF No. 652-6 (showing that, in 2010, Adobe considered Intuit to be a "direct peer," and considered Apple, Google, and Intel to be "reference peers," though Adobe did not actually benchmark compensation against these latter companies).

Nevertheless, despite viewing other Defendants as competitors, evidence from Adobe suggests that Adobe had knowledge of the bilateral agreements to which Adobe was not a party. Specifically, Adobe shared confidential compensation information with other Defendants, despite the fact that Adobe viewed at least some of the other Defendants as competitors and did not have a bilateral agreement with them. For example, HR personnel at Intuit and at Adobe exchanged information labeled "confidential" regarding how much compensation each firm would give and to which employees that year. ECF No. 652-8. Adobe and Intuit shared confidential compensation information even though the two companies had no bilateral anti-solicitation agreement, and Adobe viewed Intuit as a direct competitor for talent. Such direct competitors for talent would not likely share such sensitive compensation information in the absence of an overarching conspiracy.

Meanwhile, Google circulated an email that expressly discussed how its "budget is comparable to other tech companies" and compared the precise percentage of Google's merit budget increases to that of Adobe, Apple, and Intel. ECF No. 807-13. Google had Adobe's precise percentage of merit budget increases even though Google and Adobe had no bilateral anti-solicitation agreement. Such sharing of sensitive compensation information among competitors is further evidence of an overarching conspiracy.

Adobe recognized that in the absence of the anti-solicitation agreements, pay increases would be necessary, echoing Plaintiffs' theory of impact. For example, out of concern that one employee—a "star performer" due to his technical skills, intelligence, and collaborative abilities—might leave Adobe because "he could easily get a great job elsewhere if he desired," Adobe considered how best to retain him. ECF No. 799-22. In so doing, Adobe expressed concern about the fact that this employee had already interviewed with four other companies and communicated with friends who worked there. Id. Thus, Adobe noted that the employee "was aware of his value in the market" as well as the fact that the employee's friends from college were "making approximately $15k more per year than he [wa]s." Id. In response, Adobe decided to give the employee an immediate pay raise. Id.

Plaintiffs' theory of impact is also supported by evidence that every job position at Adobe was assigned a job title, and every job title had a corresponding salary range within Adobe's salary structure, which included a salary minimum, middle, and maximum. See ECF No. 804-17 at 4, 8, 72, 85-86. Adobe expected that the distribution of its existing employees' salaries would fit "a bell curve." ECF No. 749-5 at 57. To assist managers in staying within the prescribed ranges for setting and adjusting salaries, Adobe had an online salary planning tool as well as salary matrices, which provided managers with guidelines based on market salary data. See ECF No. 804-17 at 29-30 ("[E]ssentially the salary planning tool is populated with employee information for a particular manager, so the employees on their team [sic]. You have the ability to kind of look at their current compensation. It shows them what the range is for the current role that they're in . . . . The tool also has the ability to provide kind of the guidelines that we recommend in terms of how managers might want to think about spending their allocated budget."). Adobe's practice, if employees were below the minimum recommended salary range, was to "adjust them to the minimum as part of the annual review" and "red flag them." Id. at 12. Deviations from the salary ranges would also result in conversations with managers, wherein Adobe's officers explained, "we have a minimum for a reason because we believe you need to be in this range to be competitive." Id.

Internal equity was important at Adobe, as it was at other Defendants. As explained by Debbie Streeter (Adobe Vice President, Total Rewards), Adobe "always look[ed] at internal equity as a data point, because if you are going to go hire somebody externally that's making . . . more than somebody who's an existing employee that's a high performer, you need to know that before you bring them in." ECF No.749-5 at 175. Similarly, when considering whether to extend a counteroffer, Adobe advised "internal equity should ALWAYS be considered." ECF No. 746-7 at 5.

Moreover, Donna Morris (Adobe Senior Vice President, Global Human Resources Division) expressed concern "about internal equity due to compression (the market driving pay for new hires above the current employees)." ECF No. 298-9 ("Reality is new hires are requiring base pay at or above the midpoint due to an increasingly aggressive market."). Adobe personnel stated that, because of the fixed budget, they may not be able to respond to the problem immediately "but could look at [compression] for FY2006 if market remains aggressive."11 Id.

D. Weaknesses in Plaintiffs' Case

Plaintiffs contend that though this evidence is compelling, there are also weaknesses in Plaintiffs' case that make trial risky. Plaintiffs contend that these risks are substantial. Specifically, Plaintiffs point to the following challenges that they would have faced in presenting their case to a jury: (1) convincing a jury to find a single overarching conspiracy among the seven Defendants in light of the fact that several pairs of Defendants did not have anti-solicitation agreements with each other; (2) proving damages in light of the fact that Defendants intended to present six expert economists that would attack the methodology of Plaintiffs' experts; and (3) overcoming the fact that Class members' compensation has increased in the last ten years despite a sluggish economy and overcoming general anti-tech worker sentiment in light of the perceived and actual wealth of Class members. Plaintiffs also point to outstanding legal issues, such as the pending motions in limine and the pending motion to determine whether the per se or rule of reason analysis should apply, which could have aided Defendants' ability to present a case that the bilateral agreements had a pro-competitive purpose. See ECF No. 938 at 10-14.

The Court recognizes that Plaintiffs face substantial risks if they proceed to trial. Nonetheless, the Court cannot, in light of the evidence above, conclude that the instant settlement amount is within the range of reasonableness, particularly compared to the settlements with the Settled Defendants and the subsequent development of the litigation. The Court further notes that there is evidence in the record that mitigate at least some of the weaknesses in Plaintiffs' case.

As to proving an overarching conspiracy, several pieces of evidence undermine Defendants' contentions that the bilateral agreements were unrelated to each other. Importantly, two individuals, Steve Jobs (Co-Founder, Former Chairman, and Former CEO of Apple) and Bill Campbell (Chairman of Intuit Board of Directors, Co-Lead Director of Apple, and advisor to Google), personally entered into or facilitated each of the bilateral agreements in this case. Specifically, Mr. Jobs and George Lucas (former Chairman and CEO of Lucasfilm), created the initial anti-solicitation agreement between Lucasfilm and Pixar when Mr. Jobs was an executive at Pixar. Thereafter, Apple, under the leadership of Mr. Jobs, entered into an agreement with Pixar, which, as discussed below, Pixar executives compared to the Lucasfilm-Pixar agreement. It was Mr. Jobs again, who, as discussed above, reached out to Sergey Brin (Google Co-Founder) and Eric Schmidt (Google Executive Chairman, Member of the Board of Directors, and former CEO) to create the Apple-Google agreement. This agreement was reached with the assistance of Mr. Campbell, who was Intuit's Board Chairman, a friend of Mr. Jobs, and an advisor to Google. The Apple-Google agreement was discussed at Google Board meetings, at which both Mr. Campbell and Paul Otellini (Chief Executive Officer of Intel and Member of the Google Board of Directors) were present. ECF No. 819-10 at 47. After discussions between Mr. Brin and Mr. Otellini and between Mr. Schmidt and Mr. Otellini, Intel was added to Google's do-not-cold-call list. Mr. Campbell then used his influence at Google to successfully lobby Google to add Intuit, of which Mr. Campbell was Chairman of the Board of Directors, to Google's do-not-cold-call list. See ECF No. 780-6 at 8-9. Moreover, it was a mere two months after Mr. Jobs entered into the Apple-Google agreement that Apple pressured Bruce Chizen (former CEO of Adobe) to enter into an Apple-Adobe agreement. ECF No. 291-17. As this discussion demonstrates, Mr. Jobs and Mr. Campbell were the individuals most closely linked to the formation of each step of the alleged conspiracy, as they were present in the process of forming each of the links.

In light of the overlapping nature of this small group of executives who negotiated and enforced the anti-solicitation agreements, it is not surprising that these executives knew of the other bilateral agreements to which their own firms were not a party. For example, both Mr. Brin and Mr. Schmidt of Google testified that they would have told Mr. Otellini of Intel that Google had an anti-solicitation agreement with Apple. ECF No. 639-1 at 74:15 ("I'm sure we would have mentioned it[.]"); ECF No. 819-12 at 60 ("I'm sure I spoke with Paul about this at some point."). Intel's own expert testified that Mr. Otellini was likely aware of Google's other bilateral agreements by virtue of Mr. Otellini's membership on Google's board. ECF No. 771 at 4. Moreover, Google recruiters knew of the Adobe-Apple agreement. Id. (Google recruiter's notation that Apple has "a serious 'hands-off policy with Adobe"). In addition, Mr. Schmidt of Google testified that it would be "fair to extrapolate" based on Mr. Schmidt's knowledge of Mr. Jobs, that Mr. Jobs "would have extended [anti-solicitation agreements] to others." ECF No. 638-8 at 170. Furthermore, it was this same mix of top executives that successfully and unsuccessfully attempted to expand the agreement to other companies in Silicon Valley, such as eBay, Facebook, Macromedia, and Palm, as discussed above, suggesting that the agreements were neither isolated nor one off agreements.

In addition, the six bilateral agreements contained nearly identical terms, precluding each pair of Defendants from affirmatively soliciting any of each other's employees. ECF No. 531 at 30. Moreover, as discussed above, Defendants recognized the similarity of the agreements. For example, Google lumped together Apple, Intel, and Intuit on Google's "do-not-cold-call" list. Furthermore, Google's "do-not-cold-call" list stated that the Apple-Google agreement and the Intel-Google agreement commenced on the same date. Finally, in an email, Lori McAdams (Pixar Vice President of Human Resources and Administration), explicitly compared the anti-solicitation agreements, stating that "effective now, we'll follow a gentleman's agreement with Apple that is similar to our Lucasfilm agreement." ECF No. 531 at 26.

As to the contention that Plaintiffs would have to rebut Defendants' contentions that the anti-solicitation agreements aided collaborations and were therefore pro-competitive, there is no documentary evidence that links the anti-solicitation agreements to any collaboration. None of the documents that memorialize collaboration agreements mentions the broad anti-solicitation agreements, and none of the documents that memorialize broad anti-solicitation agreements mentions collaborations. Furthermore, even Defendants' experts conceded that those closest to the collaborations did not know of the anti-solicitation agreements. ECF No. 852-1 at 8. In addition, Defendants' top executives themselves acknowledge the lack of any collaborative purpose. For example, Mr. Chizen of Adobe admitted that the Adobe-Apple anti-solicitation agreement was "not limited to any particular projects on which Apple and Adobe were collaborating." ECF No. 962-7 at 42. Moreover, the U.S. Department of Justice ("DOJ") also determined that the anti-solicitation agreements "were not ancillary to any legitimate collaboration," "were broader than reasonably necessary for the formation or implementation of any collaborative effort," and "disrupted the normal price-setting mechanisms that apply in the labor setting." ECF No. 93-1 ¶ 16; ECF No. 93-4 ¶ 7. The DOJ concluded that Defendants entered into agreements that were restraints of trade that were per se unlawful under the antitrust laws. ECF No. 93-1 ¶ 35; ECF No. 93-4 ¶ 3. Thus, despite the fact that Defendants have claimed since the beginning of this litigation that there were pro-competitive purposes related to collaborations for the anti-solicitation agreements and despite the fact that the purported collaborations were central to Defendants' motions for summary judgment, Defendants have failed to produce persuasive evidence that these anti-solicitation agreements related to collaborations or were pro-competitive.

IV. CONCLUSION

This Court has lived with this case for nearly three years, and during that time, the Court has reviewed a significant number of documents in adjudicating not only the substantive motions, but also the voluminous sealing requests. Having done so, the Court cannot conclude that the instant settlement falls within the range of reasonableness. As this Court stated in its summary judgment order, there is ample evidence of an overarching conspiracy between the seven Defendants, including "[t]he similarities in the various agreements, the small number of intertwining high-level executives who entered into and enforced the agreements, Defendants' knowledge about the other agreements, the sharing and benchmarking of confidential compensation information among Defendants and even between firms that did not have bilateral anti-solicitation agreements, along with Defendants' expansion and attempted expansion of the anti-solicitation agreements." ECF No. 771 at 7-8. Moreover, as discussed above and in this Court's class certification order, the evidence of Defendants' rigid wage structures and internal equity concerns, along with statements from Defendants' own executives, are likely to prove compelling in establishing the impact of the anti-solicitation agreements: a Class-wide depression of wages.

In light of this evidence, the Court is troubled by the fact that the instant settlement with Remaining Defendants is proportionally lower than the settlements with the Settled Defendants. This concern is magnified by the fact that the case evolved in Plaintiffs' favor since those settlements. At the time those settlements were reached, Defendants still could have defeated class certification before this Court, Defendants still could have successfully sought appellate review and reversal of any class certification, Defendants still could have prevailed on summary judgment, or Defendants still could have succeeded in their attempt to exclude Plaintiffs' principal expert. In contrast, the instant settlement was reached a mere month before trial was set to commence and after these opportunities for Defendants had evaporated. While the unpredictable nature of trial would have undoubtedly posed challenges for Plaintiffs, the exposure for Defendants was even more substantial, both in terms of the potential of more than $9 billion in damages and in terms of other collateral consequences, including the spotlight that would have been placed on the evidence discussed in this Order and other evidence and testimony that would have been brought to light. The procedural history and proximity to trial should have increased, not decreased, Plaintiffs' leverage from the time the settlements with the Settled Defendants were reached a year ago.

The Court acknowledges that Class counsel have been zealous advocates for the Class and have funded this litigation themselves against extraordinarily well-resourced adversaries. Moreover, there very well may be weaknesses and challenges in Plaintiffs' case that counsel cannot reveal to this Court. Nonetheless, the Court concludes that the Remaining Defendants should, at a minimum, pay their fair share as compared to the Settled Defendants, who resolved their case with Plaintiffs at a stage of the litigation where Defendants had much more leverage over Plaintiffs.

For the foregoing reasons, the Court DENIES Plaintiffs' Motion for Preliminary Approval of the settlements with Remaining Defendants. The Court further sets a Case Management Conference for September 10, 2014 at 2 p.m.

IT IS SO ORDERED.

Dated: August 8, 2014
LUCY H. KOH
United States District Judge

  1. Dr. Leamer was subject to vigorous attack in the initial class certification motion, and this Court agreed with some of Defendants' contentions with respect to Dr. Leamer and thus rejected the initial class certification motion. See ECF No. 382 at 33-43. [return]
  2. Defendants' motions in limine, Plaintiffs' motion to exclude testimony from certain experts, Defendants' motion to exclude testimony from certain experts, a motion to determine whether the per se or rule of reason analysis applied, and a motion to compel were pending at the time the 3settlement was reached. [return]
  3. Plaintiffs in the instant Motion represent that two of the letters are from non-Class members and that the third letter is from a Class member who may be withdrawing his objection. See ECF No. 920 at 18 n.11. The objection has not been withdrawn at the time of this Order. [return]
  4. Devine stated in his Opposition that the Opposition was designed to supersede a letter that he had previously sent to the Court. See ECF No. at 934 n.2. The Court did not receive any letter from Devine. Accordingly, the Court has considered only Devine's Opposition. [return]
  5. Plaintiffs also assert that administration costs for the settlement would be $160,000. [return]
  6. Devine calculated that Class members would receive an average of $3,573. The discrepancy between this number and the Court's calculation may result from the fact that Devine's calculation does not account for the fact that 147 individuals have already opted out of the Class. The Court's calculation resulted from subtracting the requested attorneys' fees ($81,125,000), costs ($1,200,000), incentive awards ($400,000), and estimated administration costs ($160,000) from the settlement amount ($324,500,000) and dividing the resulting number by the total number of 7remaining class members (64,466). [return]
  7. If the Court were to deny any portion of the requested fees, costs, or incentive payments, this would increase individual Class members' recovery. If less than 4% of the Class were to opt out, that would also increase individual Class members' recovery. [return]
  8. One way to think about this is to set up the simple equation: 5/95 = $20,000,000/x. This equation asks the question of how much 95% would be if 5% were $20,000,000. Solving for x would result in $380,000,000. [return]
  9. On the same day, Mr. Campbell sent an email to Mr. Brin and to Larry Page (Google Co-Founder) stating, "Steve just called me again and is pissed that we are still recruiting his browser guy." ECF No. 428-13. Mr. Page responded "[h]e called a few minutes ago and demanded to talk to me." Id. [return]
  10. Mr. Jobs successfully expanded the anti-solicitation agreements to Macromedia, a company acquired by Adobe, both before and after Adobe's acquisition of Macromedia. [return]
  11. Adobe also benchmarked compensation off external sources, which supports Plaintiffs' theory of Class-wide impact and undermines Defendants' theory that the anti-solicitation agreements had only one off, non-structural effects. For example, Adobe pegged its compensation structure as a "percentile" of average market compensation according to survey data from companies such as Radford. ECF No. 804-17 at 4. Mr. Chizen explained that the particular market targets that Adobe used as benchmarks for setting salary ranges "tended to be software, high-tech, those that were geographically similar to wherever the position existed." ECF No. 962-7 at 22. This demonstrated that the salary structures of the various Defendants were linked, such that the effect of one Defendant's salary structure would ripple across to the other Defendants through external sources like Radford. [return]
show more
Assembly v. intrinsics
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-10-19 00:00:00 | Created: 2026-07-23 05:18:40

Every once in a while, I hear how intrinsics have improved enough that it's safe to use them for high performance code. That would be nice. The promise of intrinsics is that you can write optimized code by calling out to functions (intrinsics) that correspond to particular assembly instructions. Since intrinsics act like normal functions, they can be cross platform. And since your compiler has access to more computational power than your brain, as well as a detailed model of every CPU, the compiler should be able to do a better job of micro-optimizations. Despite decade old claims that intrinsics can make your life easier, it never seems to work out.

The last time I tried intrinsics was around 2007; for more on why they were hopeless then (see this exploration by the author of VirtualDub). I gave them another shot recently, and while they've improved, they're still not worth the effort. The problem is that intrinsics are so unreliable that you have to manually check the result on every platform and every compiler you expect your code to be run on, and then tweak the intrinsics until you get a reasonable result. That's more work than just writing the assembly by hand. If you don't check the results by hand, it's easy to get bad results.

For example, as of this writing, the first two Google hits for popcnt benchmark (and 2 out of the top 3 bing hits) claim that Intel's hardware popcnt instruction is slower than a software implementation that counts the number of bits set in a buffer, via a table lookup using the SSSE3 pshufb instruction. This turns out to be untrue, but it must not be obvious, or this claim wouldn't be so persistent. Let's see why someone might have come to the conclusion that the popcnt instruction is slow if they coded up a solution using intrinsics.

One of the top search hits has sample code and benchmarks for both native popcnt as well as the software version using pshufb. Their code requires MSVC, which I don't have access to, but their first popcnt implementation just calls the popcnt intrinsic in a loop, which is fairly easy to reproduce in a form that gcc and clang will accept. Timing it is also pretty simple, since we're just timing a function (that happens to count the number of bits set in some fixed sized buffer).

uint32_t builtin_popcnt(const uint64_t* buf, int len) {
  int cnt = 0;
  for (int i = 0; i < len; ++i) {
    cnt += __builtin_popcountll(buf[i]);
  }
  return cnt;
}

This is slightly different from the code I linked to above, since they use the dword (32-bit) version of popcnt, and we're using the qword (64-bit) version. Since our version gets twice as much done per loop iteration, I'd expect our version to be faster than their version.

Running clang -O3 -mpopcnt -funroll-loops produces a binary that we can examine. On macs, we can use otool -tv to get the disassembly. On linux, there's objdump -d.

_builtin_popcnt:
; address                        instruction
0000000100000b30        pushq   %rbp
0000000100000b31        movq    %rsp, %rbp
0000000100000b34        movq    %rdi, -0x8(%rbp)
0000000100000b38        movl    %esi, -0xc(%rbp)
0000000100000b3b        movl    $0x0, -0x10(%rbp)
0000000100000b42        movl    $0x0, -0x14(%rbp)
0000000100000b49        movl    -0x14(%rbp), %eax
0000000100000b4c        cmpl    -0xc(%rbp), %eax
0000000100000b4f        jge     0x100000bd4
0000000100000b55        movslq  -0x14(%rbp), %rax
0000000100000b59        movq    -0x8(%rbp), %rcx
0000000100000b5d        movq    (%rcx,%rax,8), %rax
0000000100000b61        movq    %rax, %rcx
0000000100000b64        shrq    %rcx
0000000100000b67        movabsq $0x5555555555555555, %rdx
0000000100000b71        andq    %rdx, %rcx
0000000100000b74        subq    %rcx, %rax
0000000100000b77        movabsq $0x3333333333333333, %rcx
0000000100000b81        movq    %rax, %rdx
0000000100000b84        andq    %rcx, %rdx
0000000100000b87        shrq    $0x2, %rax
0000000100000b8b        andq    %rcx, %rax
0000000100000b8e        addq    %rax, %rdx
0000000100000b91        movq    %rdx, %rax
0000000100000b94        shrq    $0x4, %rax
0000000100000b98        addq    %rax, %rdx
0000000100000b9b        movabsq $0xf0f0f0f0f0f0f0f, %rax
0000000100000ba5        andq    %rax, %rdx
0000000100000ba8        movabsq $0x101010101010101, %rax
0000000100000bb2        imulq   %rax, %rdx
0000000100000bb6        shrq    $0x38, %rdx
0000000100000bba        movl    %edx, %esi
0000000100000bbc        movl    -0x10(%rbp), %edi
0000000100000bbf        addl    %esi, %edi
0000000100000bc1        movl    %edi, -0x10(%rbp)
0000000100000bc4        movl    -0x14(%rbp), %eax
0000000100000bc7        addl    $0x1, %eax
0000000100000bcc        movl    %eax, -0x14(%rbp)
0000000100000bcf        jmpq    0x100000b49
0000000100000bd4        movl    -0x10(%rbp), %eax
0000000100000bd7        popq    %rbp
0000000100000bd8        ret

Well, that's interesting. Clang seems to be calculating things manually rather than using popcnt. It seems to be using the approach described here, which is something like

x = x - ((x >> 0x1) & 0x5555555555555555);
x = (x & 0x3333333333333333) + ((x >> 0x2) & 0x3333333333333333);
x = (x + (x >> 0x4)) & 0xF0F0F0F0F0F0F0F;
ans = (x * 0x101010101010101) >> 0x38;

That's not bad for a simple implementation that doesn't rely on any kind of specialized hardware, but that's going to take a lot longer than a single popcnt instruction.

I've got a pretty old version of clang (3.0), so let me try this again after upgrading to 3.4, in case they added hardware popcnt support “recently”.

0000000100001340        pushq   %rbp         ; save frame pointer
0000000100001341        movq    %rsp, %rbp   ; new frame pointer
0000000100001344        xorl    %ecx, %ecx   ; cnt = 0
0000000100001346        testl   %esi, %esi
0000000100001348        jle     0x100001363
000000010000134a        nopw    (%rax,%rax)
0000000100001350        popcntq (%rdi), %rax ; “eax” = popcnt[rdi]
0000000100001355        addl    %ecx, %eax   ; eax += cnt
0000000100001357        addq    $0x8, %rdi   ; increment address by 64-bits (8 bytes)
000000010000135b        decl    %esi         ; decrement loop counter; sets flags
000000010000135d        movl    %eax, %ecx   ;  cnt = eax; does not set flags
000000010000135f        jne     0x100001350  ; examine flags. if esi != 0, goto popcnt
0000000100001361        jmp     0x100001365  ; goto “restore frame pointer”
0000000100001363        movl    %ecx, %eax
0000000100001365        popq    %rbp         ; restore frame pointer
0000000100001366        ret

That's better! We get a hardware popcnt! Let's compare this to the SSSE3 pshufb implementation presented here as the fastest way to do a popcnt. We'll use a table like the one in the link to show speed, except that we're going to show a rate, instead of the raw cycle count, so that the relative speed between different sizes is clear. The rate is GB/s, i.e., how many gigs of buffer we can process per second. We give the function data in chunks (varying from 1kb to 16Mb); each column is the rate for a different chunk-size. If we look at how fast each algorithm is for various buffer sizes, we get the following.

Algorithm 1k 4k 16k 65k 256k 1M 4M 16M
Intrinsic 6.9 7.3 7.4 7.5 7.5 7.5 7.5 7.5
PSHUFB 11.5 13.0 13.3 13.4 13.1 13.4 13.0 12.6

That's not so great. Relative to the the benchmark linked above, we're doing better because we're using 64-bit popcnt instead of 32-bit popcnt, but the PSHUFB version is still almost twice as fast1.

One odd thing is the way cnt gets accumulated. cnt is stored in ecx. But, instead of adding the result of the popcnt to ecx, clang has decided to add ecx to the result of the popcnt. To fix that, clang then has to move that sum into ecx at the end of each loop iteration.

The other noticeable problem is that we only get one popcnt per iteration of the loop, which means the loop isn't getting unrolled, and we're paying the entire cost of the loop overhead for each popcnt. Unrolling the loop can also let the CPU extract more instruction level parallelism from the code, although that's a bit beyond the scope of this blog post.

Using clang, that happens even with -O3 -funroll-loops. Using gcc, we get a properly unrolled loop, but gcc has other problems, as we'll see later. For now, let's try unrolling the loop ourselves by calling __builtin_popcountll multiple times during each iteration of the loop. For simplicity, let's try doing four popcnt operations on each iteration. I don't claim that's optimal, but it should be an improvement.

uint32_t builtin_popcnt_unrolled(const uint64_t* buf, int len) {
  assert(len % 4 == 0);
  int cnt = 0;
  for (int i = 0; i < len; i+=4) {
    cnt += __builtin_popcountll(buf[i]);
    cnt += __builtin_popcountll(buf[i+1]);
    cnt += __builtin_popcountll(buf[i+2]);
    cnt += __builtin_popcountll(buf[i+3]);
  }
  return cnt;
}

The core of our loop now has

0000000100001390        popcntq (%rdi,%rcx,8), %rdx
0000000100001396        addl    %eax, %edx
0000000100001398        popcntq 0x8(%rdi,%rcx,8), %rax
000000010000139f        addl    %edx, %eax
00000001000013a1        popcntq 0x10(%rdi,%rcx,8), %rdx
00000001000013a8        addl    %eax, %edx
00000001000013aa        popcntq 0x18(%rdi,%rcx,8), %rax
00000001000013b1        addl    %edx, %eax

with pretty much the same code surrounding the loop body. We're doing four popcnt operations every time through the loop, which results in the following performance:

Algorithm 1k 4k 16k 65k 256k 1M 4M 16M
Intrinsic 6.9 7.3 7.4 7.5 7.5 7.5 7.5 7.5
PSHUFB 11.5 13.0 13.3 13.4 13.1 13.4 13.0 12.6
Unrolled 12.5 14.4 15.0 15.1 15.2 15.2 15.2 15.2

Between using 64-bit popcnt and unrolling the loop, we've already beaten the allegedly faster pshufb code! But it's close enough that we might get different results with another compiler or some other chip. Let's see if we can do better.

So, what's the deal with this popcnt false dependency bug that's been getting a lot of publicity lately? Turns out, popcnt has a false dependency on its destination register, which means that even though the result of popcnt doesn't depend on its destination register, the CPU thinks that it does and will wait until the destination register is ready before starting the popcnt instruction.

x86 typically has two operand operations, e.g., addl %eax, %edx adds eax and edx, and then places the result in edx, so it's common for an operation to have a dependency on its output register. In this case, there shouldn't be a dependency, since the result doesn't depend on the contents of the output register, but that's an easy bug to introduce, and a hard one to catch2.

In this particular case, popcnt has a 3 cycle latency, but it's pipelined such that a popcnt operation can execute each cycle. If we ignore other overhead, that means that a single popcnt will take 3 cycles, 2 will take 4 cycles, 3 will take 5 cycles, and n will take n+2 cycles, as long as the operations are independent. But, if the CPU incorrectly thinks there's a dependency between them, we effectively lose the ability to pipeline the instructions, and that n+2 turns into 3n.

We can work around this by buying a CPU from AMD or VIA, or by putting the popcnt results in different registers. Let's making an array of destinations, which will let us put the result from each popcnt into a different place.

uint32_t builtin_popcnt_unrolled_errata(const uint64_t* buf, int len) {
  assert(len % 4 == 0);
  int cnt[4];
  for (int i = 0; i < 4; ++i) {
    cnt[i] = 0;
  }

  for (int i = 0; i < len; i+=4) {
    cnt[0] += __builtin_popcountll(buf[i]);
    cnt[1] += __builtin_popcountll(buf[i+1]);
    cnt[2] += __builtin_popcountll(buf[i+2]);
    cnt[3] += __builtin_popcountll(buf[i+3]);
  }
  return cnt[0] + cnt[1] + cnt[2] + cnt[3];
}

And now we get

0000000100001420        popcntq (%rdi,%r9,8), %r8
0000000100001426        addl    %ebx, %r8d
0000000100001429        popcntq 0x8(%rdi,%r9,8), %rax
0000000100001430        addl    %r14d, %eax
0000000100001433        popcntq 0x10(%rdi,%r9,8), %rdx
000000010000143a        addl    %r11d, %edx
000000010000143d        popcntq 0x18(%rdi,%r9,8), %rcx

That's better -- we can see that the first popcnt outputs into r8, the second into rax, the third into rdx, and the fourth into rcx. However, this does the same odd accumulation as the original, where instead of adding the result of the popcnt to cnt[i], it does the opposite, which necessitates moving the results back to cnt[i] afterwards.

000000010000133e        movl    %ecx, %r10d
0000000100001341        movl    %edx, %r11d
0000000100001344        movl    %eax, %r14d
0000000100001347        movl    %r8d, %ebx

Well, at least in clang (3.4). Gcc (4.8.2) is too smart to fall for this separate destination thing and “optimizes” the code back to something like our original version.

Algorithm 1k 4k 16k 65k 256k 1M 4M 16M
Intrinsic 6.9 7.3 7.4 7.5 7.5 7.5 7.5 7.5
PSHUFB 11.5 13.0 13.3 13.4 13.1 13.4 13.0 12.6
Unrolled 12.5 14.4 15.0 15.1 15.2 15.2 15.2 15.2
Unrolled 2 14.3 16.3 17.0 17.2 17.2 17.0 16.8 16.7

To get a version that works with both gcc and clang, and doesn't have these extra movs, we'll have to write the assembly by hand3:

uint32_t builtin_popcnt_unrolled_errata_manual(const uint64_t* buf, int len) {
  assert(len % 4 == 0);
  uint64_t cnt[4];
  for (int i = 0; i < 4; ++i) {
    cnt[i] = 0;
  }

  for (int i = 0; i < len; i+=4) {
    __asm__(
        "popcnt %4, %4  \n\
        "add %4, %0     \n\t"
        "popcnt %5, %5  \n\t"
        "add %5, %1     \n\t"
        "popcnt %6, %6  \n\t"
        "add %6, %2     \n\t"
        "popcnt %7, %7  \n\t"
        "add %7, %3     \n\t" // +r means input/output, r means intput
        : "+r" (cnt[0]), "+r" (cnt[1]), "+r" (cnt[2]), "+r" (cnt[3])
        : "r"  (buf[i]), "r"  (buf[i+1]), "r"  (buf[i+2]), "r"  (buf[i+3]));
  }
  return cnt[0] + cnt[1] + cnt[2] + cnt[3];
}

This directly translates the assembly into the loop:

00000001000013c3        popcntq %r10, %r10
00000001000013c8        addq    %r10, %rcx
00000001000013cb        popcntq %r11, %r11
00000001000013d0        addq    %r11, %r9
00000001000013d3        popcntq %r14, %r14
00000001000013d8        addq    %r14, %r8
00000001000013db        popcntq %rbx, %rbx

Great! The adds are now going the right direction, because we specified exactly what they should do.

Algorithm 1k 4k 16k 65k 256k 1M 4M 16M
Intrinsic 6.9 7.3 7.4 7.5 7.5 7.5 7.5 7.5
PSHUFB 11.5 13.0 13.3 13.4 13.1 13.4 13.0 12.6
Unrolled 12.5 14.4 15.0 15.1 15.2 15.2 15.2 15.2
Unrolled 2 14.3 16.3 17.0 17.2 17.2 17.0 16.8 16.7
Assembly 17.5 23.7 25.3 25.3 26.3 26.3 25.3 24.3

Finally! A version that blows away the PSHUFB implementation. How do we know this should be the final version? We can see from Agner's instruction tables that we can execute, at most, one popcnt per cycle. I happen to have run this on a 3.4Ghz Sandy Bridge, so we've got an upper bound of 8 bytes / cycle * 3.4 G cycles / sec = 27.2 GB/s. That's pretty close to the 26.3 GB/s we're actually getting, which is a sign that we can't make this much faster4.

In this case, the hand coded assembly version is about 3x faster than the original intrinsic loop (not counting the version from a version of clang that didn't emit a popcnt). It happens that, for the compiler we used, the unrolled loop using the popcnt intrinsic is a bit faster than the pshufb version, but that wasn't true of one of the two unrolled versions when I tried this with gcc.

It's easy to see why someone might have benchmarked the same code and decided that popcnt isn't very fast. It's also easy to see why using intrinsics for performance critical code can be a huge time sink5.

Thanks to Scott for some comments on the organization of this post, and to Leah for extensive comments on just about everything

If you liked this, you'll probably enjoy this post about how CPUs have changed since the 80s.


  1. see this for the actual benchmarking code. On second thought, it's an embarrassingly terrible hack, and I'd prefer that you don't look. [return]
  2. If it were the other way around, and the hardware didn't realize there was a dependency when there should be, that would be easy to catch -- any sequence of instructions that was dependent might produce an incorrect result. In this case, some sequences of instructions are just slower than they should be, which is not trivial to check for. [return]
  3. This code is a simplified version of Alex Yee's stackoverflow answer about the popcnt false dependency bug [return]
  4. That's not quite right, since the CPU has TurboBoost, but it's pretty close. Putting that aside, this example is pretty simple, but calculating this stuff by hand can get tedious for more complicated code. Luckily, the Intel Architecture Code Analyzier can figure this stuff out for us. It finds the bottleneck in the code (assuming infinite memory bandwidth at zero latency), and displays how and why the processor is bottlenecked, which is usually enough to determine if there's room for more optimization.

    You might have noticed that the performance decreases as the buffer size becomes larger than our cache. It's possible to do a back of the envelope calculation to find the upper bound imposed by the limits of memory and cache performance, but working through the calculations would take a lot more space this this footnote has available to it. You can see a good example of how do it for one simple case here. The comments by Nathan Kurz and John McCaplin are particularly good.

    [return]
  5. In the course of running these benchmarks, I also noticed that _mm_cvtsi128_si64 produces bizarrely bad code on gcc (although it's fine in clang). _mm_cvtsi128_si64 is the intrinsic for moving an SSE (SIMD) register to a general purpose register (GPR). The compiler has a lot of latitude over whether or not a variable should live in a register or in memory. Clang realizes that it's probably faster to move the value from an SSE register to a GPR if the result is about to get used. Gcc decides to save a register and move the data from the SSE register to memory, and then have the next instruction operate on memory, if that's possible. In our popcnt example, clang uses about 2x for not unrolling the loop, and the rest comes from not being up to date on a CPU bug, which is understandable. It's hard to imagine why a compiler would do a register to memory move when it's about to operate on data unless it either doesn't do optimizations at all, or it has some bug which makes it unaware of the register to register version of the instruction. But at least it gets the right result, unlike this version of MSVC.

    icc and armcc are reputed to be better at dealing with intrinsics, but they're non starters for most open source projects. Downloading icc's free non-commercial version has been disabled for the better part of a year, and even if it comes back, who's going to trust that it won't disappear again? As for armcc, I'm not sure it's ever had a free version?

    [return]
show more
Testing v. informal reasoning
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-11-03 00:00:00 | Created: 2026-07-23 05:18:40

This is an off-the-cuff comment for Hacker School's Paper of the Week Read Along series for Out of the Tar Pit.

I find the idea itself, which is presented in sections 7-10, at the end of the paper, pretty interesting. However, I have some objections to the motivation for the idea, which makes up the first 60% of the paper.

Rather than do one of those blow-by-blow rebuttals that's so common on blogs, I'll limit my comments to one widely circulated idea that I believe is not only mistaken but actively harmful.

There's a claim that “informal reasoning” is more important than “testing”1, based mostly on the strength of this quote from Dijkstra:

testing is hopelessly inadequate....(it) can be used very effectively to show the presence of bugs but never to show their absence.

They go on to make a number of related claims, like “The key problem is that a test (of any kind) on a system or component that is in one particular state tells you nothing at all about the behavior of that system or component when it happens to be in another state.”, with the conclusion that stateless simplicity is the only possible fix. Needless to say, they assume that simplicity is actually possible.

I actually agree with the bit about testing -- there's no way to avoid bugs if you create a system that's too complex to formally verify.

However, there are plenty of real systems with too much irreducible complexity to make simple. Drawing from my own experience, no human can possibly hope to understand a modern high-performance CPU well enough to informally reason about its correctness. That's not only true now, it's been true for decades. It becomes true the moment someone introduces any sort of speculative execution or caching. These things are inherently stateful and complicated. They're so complicated that the only way to model performance (in order to run experiments to design high performance chips) is to simulate precisely what will happen, since the exact results are too complex for humans to reason about and too messy to be mathematically tractable. It's possible to make a simple CPU, but not one that's fast and simple. This doesn't only apply to CPUs -- performance complexity leaks all the way up the stack.

And it's not only high performance hardware and software that's complex. Some domains are just really complicated. The tax code is 73k pages long. It's just not possible to reason effectively about something that complicated, and there are plenty of things that are that complicated.

And then there's the fact that we're human. We make mistakes. Euclid's elements contains a bug in the very first theorem. Andrew Gelman likes to use this example of an "obviously" bogus published probability result (but not obvious to the authors or the peer reviewers). One of the famous Intel CPU bugs allegedly comes from not testing something because they "knew" it was correct. No matter how smart or knowledgeable, humans are incapable of reasoning correctly all of the time.

So what do you do? You write tests! They're necessary for anything above a certain level of complexity. The argument the authors make is that they're not sufficient because the state space is huge and a test of one state tells you literally nothing about a test of any other state.

That's true if you look at your system as some kind of unknowable black box, but it turns out to be untrue in practice. There are plenty of unit testing tools that will do state space reduction based on how similar inputs affect similar states, do symbolic execution, etc. This turns out to work pretty well.

And even without resorting to formal methods, you can see this with plain old normal tests. John Regehr has noted that when Csmith finds a bug, test case reduction often finds a slew of other bugs. Turns out, tests often tell you something about nearby states.

This is not just a theoretical argument. I did CPU design/verification/test for 7.5 years at a company that relied primarily on testing. In that time I can recall two bugs that were found by customers (as opposed to our testing). One was a manufacturing bug that has no software analogue. The software equivalent would be that the software works for years and then after lots of usage at high temperature 1% of customers suddenly can't use their software anymore. Bad, but not a failure of anything analogous to software testing.

The other bug was a legitimate logical bug (in the cache memory hierarchy, of course). It's pretty embarrassing that we shipped samples of a chip with a real bug to customers, but I think that most companies would be pretty happy with one logical bug in seven and a half years.

Testing may not be sufficient to find all bugs, but it can be sufficient to achieve better reliability than pretty much any software company cares to.

Thanks (or perhaps anti-thanks) to David Albert for goading me into writing up this response and to Govert Versluis for catching a typo.


  1. These kinds of claims are always a bit odd to talk about. Like nature v. nurture, we clearly get bad results if we set either quantity to zero, and they interact in a way that makes it difficult to quantify the relative effect of non-zero quantities. [return]
show more
Caches: LRU v. random
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-11-03 00:00:00 | Created: 2026-07-23 05:18:40

Once upon a time, my computer architecture professor mentioned that using a random eviction policy for caches really isn't so bad. That random eviction isn't bad can be surprising — if your cache fills up and you have to get rid of something, choosing the least recently used (LRU) is an obvious choice, since you're more likely to use something if you've used it recently. If you have a tight loop, LRU is going to be perfect as long as the loop fits in cache, but it's going to cause a miss every time if the loop doesn't fit. A random eviction policy degrades gracefully as the loop gets too big.

In practice, on real workloads, random tends to do worse than other algorithms. But what if we take two random choices (2-random) and just use LRU between those two choices?

Here are the relative miss rates we get for SPEC CPU1 with a Sandy Bridge-like cache (8-way associative, 64k, 256k, and 2MB L1, L2, and L3 caches, respectively). These are ratios (algorithm miss rate : random miss rate); lower is better. Each cache uses the same policy at all levels of the cache.

Policy L1 (64k) L2 (256k) L3 (2MB)
2-random 0.91 0.93 0.95
FIFO 0.96 0.97 1.02
LRU 0.90 0.90 0.97
random 1.00 1.00 1.00

Random and FIFO are both strictly worse than either LRU or 2-random. LRU and 2-random are pretty close, with LRU edging out 2-random for the smaller caches and 2-random edging out LRU for the larger caches.

To see if anything odd is going on in any individual benchmark, we can look at the raw results on each sub-benchmark. The L1, L2, and L3 miss rates are all plotted in the same column for each benchmark, below:

Cache miss rates for Sandy Bridge-like cache

As we might expect, LRU does worse than 2-random when the miss rates are high, and better when the miss rates are low.

At this point, it's not clear if 2-random is beating LRU in L3 cache miss rates because it does better when the caches are large or because it does better because it's the third level in a hierarchical cache. Since a cache line that's being actively used in L1 or L2 isn't touched in L3, an eviction can happen from the L3 (which forces an eviction of both the L1 and L2) since, as far as the L3 is concerned, that line hasn't been used recently. This makes it less obvious that LRU is a good eviction policy for L3 cache.

To separate out the effects, let's look at the relative miss rates for a non-hierarchical (single level) vs. hierarchical caches at various sizes2. For the hierarchical cache, the L1 and L2 sizes are as above, 64k and 256k, and only the L3 cache size varies. Below, we've got the geometric means of the ratios3 of how each policy does (over all SPEC sub-benchmarks, compared to random eviction). A possible downside to this metric is that if we have some very low miss rates, those could dominate the mean since small fluctuations will have a large effect on the ratio, but we can look the distribution of results to see if that's the case.

Cache miss ratios for cache sizes between 64K and 16M

L3 cache miss ratios for cache sizes between 512K and 16M

Sizes below 512k are missing for the hierarchical case because of the 256k L2 — we're using an inclusive L3 cache here, so it doesn't really make sense to have an L3 that's smaller than the L2. Sizes above 16M are omitted because cache miss rates converge when the cache gets too big, which is uninteresting.

Looking at the single cache case, it seems that LRU works a bit better than 2-random for smaller caches (lower miss ratio is better), 2-random edges out LRU as the cache gets bigger. The story is similar in the hierarchical case, except that we don't really look at the smaller cache sizes where LRU is superior.

Comparing the two cases, the results are different, but similar enough that it looks our original results weren't only an artifact of looking at the last level of a hierarchical cache.

Below, we'll look at the entire distribution so we can see if the mean of the ratios is being skewed by tiny results.

L3 cache miss ratios for cache sizes between 512K and 16M

L3 cache miss ratios for cache sizes between 512K and 16M

It looks like, for a particular cache size (one column of the graph), the randomized algorithms do better when miss rates are relatively high and worse when miss rates are relatively low, so, if anything, they're disadvantaged when we just look at the geometric mean — if we were to take the arithmetic mean, the result would be dominated by the larger results, where 2 random choices and plain old random do relatively well4.

From what we've seen of the mean ratios, 2-random looks fine for large caches, and from what we've seen of the distribution of the results, that's despite 2-random being penalized by the mean ratio metric, which makes it seem pretty good for large caches.

However, it's common to implement pseudo-LRU policies because LRU can be too expensive to be workable. Since 2-random requires having at least as much information as LRU, let's take a look at what happens we use pseudo 2-random (approximately 80% accurate), and pseudo 3-random (a two-level tournament, each level of which is approximately 80% accurate).

Since random and FIFO are clearly not good replacement policies, I'll leave them out of the following graphs. Also, since the results were similar in the single cache as well as multi-level cache case, we can just look at the results from the more realistic multi-level cache case.

L3 cache miss ratios for cache sizes between 512K and 16M

Since pseudo 2-random acts like random 20% of the time and 2-random 80% of the time, we might expect it to fall somewhere between 2-random and random, which is exactly what happens. A simple tweak to try to improve pseudo 2-random is to try pseudo 3-random (evict the least recently used of 3 random choices). While that's still not quite as good as true 2-random, it's pretty close, and it's still better than LRU (and pseudo LRU) for caches larger than 1M.

The one big variable we haven't explored is the set associativity. To see how LRU compares with 2-random across different cache sizes let's look at the LRU:2-random miss ratio (higher/red means LRU is better, lower/green means 2-random is better).

Cache miss ratios for cache sizes between 64K and 16M with associativities between and 64

On average, increasing associativity increases the difference between the two policies. As before, LRU is better for small caches and 2-random is better for large caches. Associativities of 1 and 2 aren't shown because they should be identical for both algorithms.

There's still a combinatorial explosion of possibilities we haven't tried yet. One thing to do is to try different eviction policies at different cache levels (LRU for L1 and L2 with 2-random for L3 seems promising). Another thing to do is to try this for different types of caches. I happened to choose CPU caches because it's easy to find simulators and benchmark traces, but in today's “put a cache on it” world, there are a lot of other places 2-random can be applied5.

For any comp arch folks, from this data, I suspect that 2-random doesn't keep up with adaptive policies like DIP (although it might — it's in the right ballpark, but it was characterized on a different workload using a different simulator, so it's not 100% clear). However, A pseudo 2-random policy can be implemented that barely uses more resources than pseudo-LRU policies, which makes this very cheap compared to DIP. Also, we can see that pseudo 3-random is substantially better than pseudo 2-random, which indicates that k-random is probably an improvement over 2-random for the k. Some k-random policy might be an improvement over DIP.

So we've seen that this works, but why would anyone think to do this in the first place? The Power of Two Random Choices: A Survey of Techniques and Results by Mitzenmacher, Richa, and Sitaraman has a great explanation. The mathematical intuition is that if we (randomly) throw n balls into n bins, the maximum number of balls in any bin is O(log n / log log n) with high probability, which is pretty much just O(log n). But if (instead of choosing randomly) we choose the least loaded of k random bins, the maximum is O(log log n / log k) with high probability, i.e., even with two random choices, it's basically O(log log n) and each additional choice only reduces the load by a constant factor.

This turns out to have all sorts of applications; things like load balancing and hash distribution are natural fits for the balls and bins model. There are also a lot of applications that aren't obviously analogous to the balls and bins model, like circuit routing and Erdős–Rényi graphs.

Thanks to Jan Elder and Mark Hill for making dinero IV freely available, to Aleksandar Milenkovic for providing SPEC CPU traces, and to Carl Vogel, James Porter, Peter Fraenkel, Katerina Barone-Adesi, Jesse Luehrs, Lea Albaugh, and Kevin Lynagh for advice on plots and plotting packages, to Mindy Preston for finding a typo in the acknowledgments, to Lindsey Kuper for pointing out some terminology stuff, to Tom Wenisch for suggesting that I check out CMP$im for future work, and to Leah Hanson for extensive comments on the entire post.


  1. Simulations were done with dinero IV with SBC traces. These were used because professors and grad students have gotten more protective of simulator code over the past couple decades, making it hard to find a modern open source simulator on GitHub. However, dinero IV supports hierarchical caches with prefetching, so it should give a reasonable first-order approximation.

    Note that 175.vpr and 187.facerec weren't included in the traces, so they're missing from all results in this post.

    [return]
  2. Sizes are limited by dinero IV, which requires cache sizes to be a power of 2. [return]
  3. Why consider the geometric mean of the ratios? We have different “base” miss rates for different benchmarks. For example, 181.mcf has a much higher miss rate than 252.eon. If we're trying to figure out which policy is best, those differences are just noise. Looking at the ratios removes that noise.

    And if we were just comparing those two, we'd like being 2x better on both to be equivalent to being 4x better on one and just 1x on the other, or 8x better on one and 1/2x “better” on the other. Since the geometric mean is the nth-root of the product of the results, it has that property.

    [return]
  4. We can see that 2-choices tends to be better than LRU for high miss rates by looking for the high up clusters of a green triangle, red square, empty diamond, and a blue circle, and seeing that it's usually the case that the green triangle is above the red square. It's too cluttered to really tell what's going on at the lower miss rates. I admit I cheated and looked at some zoomed in plots. [return]
  5. If you know of a cache simulator for some other domain that I can use, please let me know! [return]
show more
CLWB and PCOMMIT
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-11-05 00:00:00 | Created: 2026-07-23 05:18:40

The latest version of the Intel manual has a couple of new instructions for non-volatile storage, like SSDs. What's that about?

Before we look at the instructions in detail, let's take a look at the issues that exist with super fast NVRAM. One problem is that next generation storage technologies (PCM, 3d XPoint, etc.), will be fast enough that syscall and other OS overhead can be more expensive than the actual cost of the disk access1. Another is the impedance mismatch between the x86 memory hierarchy and persistent memory. In both cases, it's basically an Amdahl's law problem, where one component has improved so much that other components have to improve to keep up.

There's a good paper by Todor Mollov, Louis Eisner, Arup De, Joel Coburn, and Steven Swanson on the first issue; I'm going to present one of their graphs below.

OS and other overhead for NVRAM operations

Everything says “Moneta” because that's the name of their system (which is pretty cool, BTW; I recommend reading the paper to see how they did it). Their “baseline” case is significantly better than you'll get out of a stock system. They did a number of optimizations (e.g., bypassing Linux's IO scheduler and removing context switches where possible), which reduces latency by 62% over plain old linux. Despite that, the hardware + DMA cost of the transaction (the white part of the bar) is dwarfed by the overhead. Note that they consider the cost of the DMA to be part of the hardware overhead.

They're able to bypass the OS entirely and reduce a lot of the overhead, but it's still true that the majority of the cost of a write is overhead.

OS bypass speedup for NVRAM operations

Despite not being able to get rid of all of the overhead, they get pretty significant speedups, both on small microbenchmarks and real code. So that's one problem. The OS imposes a pretty large tax on I/O when your I/O device is really fast.

Maybe you can bypass large parts of that problem by just mapping your NVRAM device to a region of memory and committing things to it as necessary. But that runs into another problem. which is the impedance mismatch between how caches interact with the NVRAM region if you want something like transactional semantics.

This is described in more detail in this report by Kumud Bhandari, Dhruva R. Chakrabarti, and Hans-J. Boehm. I'm going to borrow a couple of their figures, too.

Rough memory hierarchy diagram

We've got this NVRAM region which is safe and persistent, but before the CPU can get to it, it has to go through multiple layers with varying ordering guarantees. They give the following example:

Consider, for example, a common programming idiom where a persistent memory location N is allocated, initialized, and published by assigning the allocated address to a global persistent pointer p. If the assignment to the global pointer becomes visible in NVRAM before the initialization (presumably because the latter is cached and has not made its way to NVRAM) and the program crashes at that very point, a post-restart dereference of the persistent pointer will read uninitialized data. Assuming writeback (WB) caching mode, this can be avoided by inserting cache-line flushes for the freshly allocated persistent locations N before the assignment to the global persistent pointer p.

Inserting CLFLUSH instructions all over the place works, but how much overhead is that?

Persistence overhead on reads and writes

The four memory types they look at (and the four that x86 supports) are writeback (WB), writethrough (WT), write combine (WC), and uncacheable (UC). WB is what you deal with under normal circumstances. Memory can be cached and it's written back whenever it's forced to be. WT allows memory to be cached, but writes have to be written straight through to memory, i.e., memory is kept up to date with the cache. UC simply can't be cached. WC is like UC, except that writes can be coalesced before being sent out to memory.

The R, W, and RW benchmarks are just benchmarks of reading and writing memory. WB is clearly the best, by far (lower is better). If you want to get an intuitive feel for how much better WB is than the other policies, try booting an OS with anything but WB memory.

I've had to do that on occasion because I use to work for a chip company, and when we first got the chip back, we often didn't know which bits we had to disable to work around bugs. The simplest way to make progress is often to disable caches entirely. That “works”, but even minimal OSes like DOS are noticeably slow to boot without WB memory. My recollection is that Win 3.1 takes the better part of an hour, and that Win 95 is a multiple hour process.

The _b benchmarks force writes to be visible to memory. For the WB case, that involves an MFENCE followed by a CLFLUSH. WB with visibility constraints is significantly slower than the other alternatives. It's a multiple order of magnitude slowdown over WB when writes don't have to be ordered and flushed.

They also run benchmarks on some real data structures, with the constraint that data should be persistently visible.

Persistence overhead on data structure operations

The performance of regular WB memory can be terribly slow: within a factor of 2 of the performance of running without caches. And that's just the overhead around getting out of the cache hierarchy -- that's true even if your persistent storage is infinitely fast.

Now, let's look how Intel decided to address this. There are two new instructions, CLWB and PCOMMIT.

CLWB acts like CLFLUSH, in that it forces the data to get written out to memory. However, it doesn't force the cache to throw away the data, which makes future reads and writes a lot faster. Also, CLFLUSH is only ordered with respect to MFENCE, but CLWB is also ordered with respect to SFENCE. Here's their description of CLWB:

Writes back to memory the cache line (if dirty) that contains the linear address specified with the memory operand from any level of the cache hierarchy in the cache coherence domain. The line may be retained in the cache hierarchy in non-modified state. Retaining the line in the cache hierarchy is a performance optimization (treated as a hint by hardware) to reduce the possibility of cache miss on a subsequent access. Hardware may choose to retain the line at any of the levels in the cache hierarchy, and in some cases, may invalidate the line from the cache hierarchy. The source operand is a byte memory location.

It should be noted that processors are free to speculatively fetch and cache data from system memory regions that are assigned a memory-type allowing for speculative reads (such as, the WB, WC, and WT memory types). Because this speculative fetching can occur at any time and is not tied to instruction execution, the CLWB instruction is not ordered with respect to PREFETCHh instructions or any of the speculative fetching mechanisms (that is, data can be speculatively loaded into a cache line just before, during, or after the execution of a CLWB instruction that references the cache line).

CLWB instruction is ordered only by store-fencing operations. For example, software can use an SFENCE, MFENCE, XCHG, or LOCK-prefixed instructions to ensure that previous stores are included in the write-back. CLWB instruction need not be ordered by another CLWB or CLFLUSHOPT instruction. CLWB is implicitly ordered with older stores executed by the logical processor to the same address.

Executions of CLWB interact with executions of PCOMMIT. The PCOMMIT instruction operates on certain store-to-memory operations that have been accepted to memory. CLWB executed for the same cache line as an older store causes the store to become accepted to memory when the CLWB execution becomes globally visible.

PCOMMIT is applied to entire memory ranges and ensures that everything in the memory range is committed to persistent storage. Here's their description of PCOMMIT:

The PCOMMIT instruction causes certain store-to-memory operations to persistent memory ranges to become persistent (power failure protected).1 Specifically, PCOMMIT applies to those stores that have been accepted to memory.

While all store-to-memory operations are eventually accepted to memory, the following items specify the actions software can take to ensure that they are accepted:

Non-temporal stores to write-back (WB) memory and all stores to uncacheable (UC), write-combining (WC), and write-through (WT) memory are accepted to memory as soon as they are globally visible. If, after an ordinary store to write-back (WB) memory becomes globally visible, CLFLUSH, CLFLUSHOPT, or CLWB is executed for the same cache line as the store, the store is accepted to memory when the CLFLUSH, CLFLUSHOPT or CLWB execution itself becomes globally visible.

If PCOMMIT is executed after a store to a persistent memory range is accepted to memory, the store becomes persistent when the PCOMMIT becomes globally visible. This implies that, if an execution of PCOMMIT is globally visible when a later store to persistent memory is executed, that store cannot become persistent before the stores to which the PCOMMIT applies.

The following items detail the ordering between PCOMMIT and other operations:

A logical processor does not ensure previous stores and executions of CLFLUSHOPT and CLWB (by that logical processor) are globally visible before commencing an execution of PCOMMIT. This implies that software must use appropriate fencing instruction (e.g., SFENCE) to ensure the previous stores-to-memory operations and CLFLUSHOPT and CLWB executions to persistent memory ranges are globally visible (so that they are accepted to memory), before executing PCOMMIT.

A logical processor does not ensure that an execution of PCOMMIT is globally visible before commencing subsequent stores. Software that requires that such stores not become globally visible before PCOMMIT (e.g., because the younger stores must not become persistent before those committed by PCOMMIT) can ensure by using an appropriate fencing instruction (e.g., SFENCE) between PCOMMIT and the later stores.

An execution of PCOMMIT is ordered with respect to executions of SFENCE, MFENCE, XCHG or LOCK-prefixed instructions, and serializing instructions (e.g., CPUID).

Executions of PCOMMIT are not ordered with respect to load operations. Software can use MFENCE to order loads with PCOMMIT.

Executions of PCOMMIT do not serialize the instruction stream.

How much CLWB and PCOMMIT actually improve performance will be up to their implementations. It will be interesting to benchmark these and see how they do. In any case, this is an attempt to solve the WB/NVRAM impedance mismatch issue. It doesn't directly address the OS overhead issue, but that can, to a large extent, be worked around without extra hardware.

If you liked this post, you'll probably also enjoy reading about cache partitioning in Broadwell and newer Intel server parts.

Thanks to Eric Bron for spotting this in the manual and pointing it out, and to Leah Hanson, Nate Rowe, and 'unwind' for finding typos.

If you haven't had enough of papers, Zvonimir Bandic pointed out a paper by Dejan Vučinić, Qingbo Wang, Cyril Guyot, Robert Mateescu, Filip Blagojević, Luiz Franca-Neto, Damien Le Moal, Trevor Bunker, Jian Xu, and Steven Swanson on getting 1.4 us latency and 700k IOPS out of a type of NVRAM

If you liked this post, you might also like this related post on "new" CPU features.


  1. this should sound familiar to HPC and HFT folks with InfiniBand networks. [return]
show more
Literature review on the benefits of static types
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-11-07 00:00:00 | Created: 2026-07-23 05:18:40

There are some pretty strong statements about types floating around out there. The claims range from the oft-repeated phrase that when you get the types to line up, everything just works, to “not relying on type safety is unethical (if you have an SLA)”1, "It boils down to cost vs benefit, actual studies, and mathematical axioms, not aesthetics or feelings", and I think programmers who doubt that type systems help are basically the tech equivalent of an anti-vaxxer. The first and last of these statements are from "types" thought leaders who are widely quoted. There are probably plenty of strong claims about dynamic languages that I'd be skeptical of if I heard them, but I'm not in the right communities to hear the stronger claims about dynamically typed languages. Either way, it's rare to see people cite actual evidence.

Let's take a look at the empirical evidence that backs up these claims.

Click here if you just want to see the summary without having to wade through all the studies. The summary of the summary is that most studies find very small effects, if any. However, the studies probably don't cover contexts you're actually interested in. If you want the gory details, here's each study, with its abstract, and a short blurb about the study.

A Large Scale Study of Programming Languages and Code Quality in Github; Ray, B; Posnett, D; Filkov, V; Devanbu, P

Abstract

What is the effect of programming languages on software quality? This question has been a topic of much debate for a very long time. In this study, we gather a very large data set from GitHub (729 projects, 80 Million SLOC, 29,000 authors, 1.5 million commits, in 17 languages) in an attempt to shed some empirical light on this question. This reasonably large sample size allows us to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static v.s. dynamic typing, strong v.s. weak typing on software quality. By triangulating findings from different methods, and controlling for confounding effects such as team size, project size, and project history, we report that language design does have a significant, but modest effect on software quality. Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages.

Summary

The authors looked at the 50 most starred repos on github for each of the 20 most popular languages plus TypeScript (minus CSS, shell, and vim). For each of these projects, they looked at the languages used. The text in the body of the study doesn't support the strong claims made in the abstract. Additionally, the study appears to use a fundamentally flawed methodology that's not capable of revealing much information. Even if the methodology were sound, the study uses bogus data and has what Pinker calls the igon value problem.

As Gary Bernhardt points out, the authors of the study seem to confuse memory safety and implicit coercion and make other strange statements, such as

Advocates of dynamic typing may argue that rather than spend a lot of time correcting annoying static type errors arising from sound, conservative static type checking algorithms in compilers, it’s better to rely on strong dynamic typing to catch errors as and when they arise.

The study uses the following language classification scheme

Table of classifications

These classifications seem arbitrary and many people would disagree with some of these classifications. Since the results are based on aggregating results with respect to these categories, and the authors have chosen arbitrary classifications, this already makes the aggragated results suspect since they have a number of degrees of freedom here and they've made some odd choicses.

In order to get the language level results, the authors looked at commit/PR logs to determine how many bugs there were for each language used. As far as I can tell, open issues with no associated fix don't count towards the bug count. Only commits that are detected by their keyword search technique were counted. With this methodology, the number of bugs found will depend at least as strongly on the bug reporting culture as it does on the actual number of bugs found.

After determining the number of bugs, the authors ran a regression, controlling for project age, number of developers, number of commits, and lines of code.

Defect rate correlations

There are enough odd correlations here that, even if the methodology wasn't known to be flawed, I'd be skeptical that authors have captured a causal relationship. If you don't find it odd that Perl and Ruby are as reliable as each other and significantly more reliable than Erlang and Java (which are also equally reliable), which are significantly more reliable than Python, PHP, and C (which are similarly reliable), and that TypeScript is the safest language surveyed, then maybe this passes the sniff test for you, but even without reading further, this looks suspicious.

For example, Erlang and Go are rated as having a lot of concurrency bugs, whereas Perl and CoffeeScript are rated as having few concurrency bugs. Is it more plausible that Perl and CoffeeScript are better at concurrency than Erlang and Go or that people tend to use Erlang and Go more when they need concurrency? The authors note that Go might have a lot of concurrency bugs because there's a good tool to detect concurrency bugs in Go, but they don't explore reasons for most of the odd intermediate results.

As for TypeScript, Eirenarch has pointed out that the three projects they list as example TypeScript projects, which they call the "top three" TypeScript projects are bitcoin, litecoin, and qBittorrent). These are C++ projects. So the intermediate result appears to not be that TypeScript is reliable, but that projects mis-identified as TypeScript are reliable. Those projects are reliable because Qt translation files are identified as TypeScript and it turns out that, per line of code, giant dumps of config files from another project don't cause a lot of bugs. It's like saying that a project has few bugs per line of code because it has a giant README. This is the most blatant classification error, but it's far from the only one.

For example, of what they call the "top three" perl projects, one is showdown, a javascript project, and one is rails-dev-box, a shell script and a vagrant file used to launch a Rails dev environment. Without knowing anything about the latter project, one might expect it's not a perl project from its name, rails-dev-box, which correctly indicates that it's a rails related project.

Since this study uses Github's notoriously inaccurate code classification system to classify repos, it is, at best, a series of correlations with factors that are themselves only loosely correlated with actual language usage.

There's more analysis, but much of it is based on aggregating the table above into categories based on language type. Since I'm skeptical of these results, I'm at least as skeptical of any results based on aggregating these results. This section barely even scratches the surface of this study. Even with just a light skim, we see multiple serious flaws, any one of which would invalidate the results, plus numerous igon value problems. It appears that the authors didn't even look at the tables they put in the paper, since if they did, it would jump out that (just for example), they classified a project called "rails-dev-box" as one of the three biggest perl projects (it's a 70-line shell script used to spin up ruby/rails dev environments).

Do Static Type Systems Improve the Maintainability of Software Systems? An Empirical Study Kleinschmager, S.; Hanenberg, S.; Robbes, R.; Tanter, E.; Stefik, A.

Abstract

Static type systems play an essential role in contemporary programming languages. Despite their importance, whether static type systems influence human software development capabilities remains an open question. One frequently mentioned argument for static type systems is that they improve the maintainability of software systems - an often used claim for which there is no empirical evidence. This paper describes an experiment which tests whether static type systems improve the maintainability of software systems. The results show rigorous empirical evidence that static type are indeed beneficial to these activities, except for fixing semantic errors.

Summary

While the abstract talks about general classes of languages, the study uses Java and Groovy.

Subjects were given classes in which they had to either fix errors in existing code or fill out stub methods. Static classes for Java, dynamic classes for Groovy. In cases of type errors (and their respective no method errors), developers solved the problem faster in Java. For semantic errors, there was no difference.

The study used a within-subject design, with randomized task order over 33 subjects.

A notable limitation is that the study avoided using “complicated control structures”, such as loops and recursion, because those increase variance in time-to-solve. As a result, all of the bugs are trivial bugs. This can be seen in the median time to solve the tasks, which are in the hundreds of seconds. Tasks can include multiple bugs, so the time per bug is quite low.

Groovy is both better and worse than Java

This paper mentions that its results contradict some prior results, and one of the possible causes they give is that their tasks are more complex than the tasks from those other papers. The fact that the tasks in this paper don't involve using loops and recursion because they're too complicated, should give you an idea of the complexity of the tasks involved in most of these papers.

Other limitations in this experiment were that the variables were artificially named such that there was no type information encoded in any of the names, that there were no comments, and that there was zero documentation on the APIs provided. That's an unusually hostile environment to find bugs in, and it's not clear how the results generalize if any form of documentation is provided.

Additionally, even though the authors specifically picked trivial tasks in order to minimize the variance between programmers, the variance between programmers was still much greater than the variance between languages in all but two tasks. Those two tasks were both cases of a simple type error causing a run-time exception that wasn't near the type error.

A controlled experiment to assess the benefits of procedure argument type checking, Prechelt, L.; Tichy, W.F.

Abstract

Type checking is considered an important mechanism for detecting programming errors, especially interface errors. This report describes an experiment to assess the defect-detection capabilities of static, intermodule type checking.

The experiment uses ANSI C and Kernighan & Ritchie (K&R) C. The relevant difference is that the ANSI C compiler checks module interfaces (i.e., the parameter lists calls to external functions), whereas K&R C does not. The experiment employs a counterbalanced design in which each of the 40 subjects, most of them CS PhD students, writes two nontrivial programs that interface with a complex library (Motif). Each subject writes one program in ANSI C and one in K&R C. The input to each compiler run is saved and manually analyzed for defects.

Results indicate that delivered ANSI C programs contain significantly fewer interface defects than delivered K&R C programs. Furthermore, after subjects have gained some familiarity with the interface they are using, ANSI C programmers remove defects faster and are more productive (measured in both delivery time and functionality implemented)

Summary

The “nontrivial” tasks are the inversion of a 2x2 matrix (with GUI) and a file “browser” menu that has two options, select file and display file. Docs for motif were provided, but example code was deliberately left out.

There are 34 subjects. Each subjects solves one problem with the K&R C compiler (which doesn't typecheck arguments) and one with the ANSI C compiler (which does).

The authors note that the distribution of results is non-normal, with highly skewed outliers, but they present their results as box plots, which makes it impossible to see the distribution. They do some statistical significance tests on various measures, and find no difference in time to completion on the first task, a significant difference on the second task, but no difference when the tasks are pooled.

ANSI C is better, except when it's worse

In terms of how the bugs are introduced during the programming process, they do a significance test against the median of one measure of defects (which finds a significant difference in the first task but not the second), and a significance test against the 75%-quantile of another measure (which finds a significant difference in the second task but not the first).

In terms of how many and what sort of bugs are in the final program, they define a variety of measures and find that some differences on the measures are statistically significant and some aren't. In the table below, bolded values indicate statistically significant differences.

Breakdown by various metrics

Note that here, first task refers to whichever task the subject happened to perform first, which is randomized, which makes the results seem rather arbitrary. Furthermore, the numbers they compare are medians (except where indicated otherwise), which also seems arbitrary.

Despite the strong statement in the abstract, I'm not convinced this study presents strong evidence for anything in particular. They have multiple comparisons, many of which seem arbitrary, and find that some of them are significant. They also find that many of their criteria don't have significant differences. Furthermore, they don't mention whether or not they tested any other arbitrary criteria. If they did, the results are much weaker than they look, and they already don't look strong.

My interpretation of this is that, if there is an effect, the effect is dwarfed by the difference between programmers, and it's not clear whether there's any real effect at all.

An empirical comparison of C, C++, Java, Perl, Python, Rexx, and Tcl, Prechelt, L.

Abstract

80 implementations of the same set of requirements are compared for several properties, such as run time, memory consumption, source text length, comment density, program structure, reliability, and the amount of effort required for writing them. The results indicate that, for the given programming problem, which regards string manipulation and search in a dictionary, “scripting languages” (Perl, Python, Rexx, Tcl) are more productive than “conventional languages” (C, C++, Java). In terms of run time and memory consumption, they often turn out better than Java and not much worse than C or C++. In general, the differences between languages tend to be smaller than the typical differences due to different programmers within the same language.

Summary

The task was to read in a list of phone numbers and return a list of words that those phone numbers could be converted to, using the letters on a phone keypad.

This study was done in two phases. There was a controlled study for the C/C++/Java group, and a self-timed implementation for the Perl/Python/Rexx/Tcl group. The former group consisted of students while the latter group consisted of respondents from a newsgroup. The former group received more criteria they should consider during implementation, and had to implement the program when they received the problem description, whereas some people in the latter group read the problem description days or weeks before implementation.

If you take the results at face value, it looks like the class of language used imposes a lower bound on both implementation time and execution time, but that the variance between programmers is much larger than the variance between languages.

However, since the scripting language group had significantly different (and easier) environment than the C-like language group, it's hard to say how much of the measured difference in implementation time is from flaws in the experimental design and how much is real.

Static type systems (sometimes) have a positive impact on the usability of undocumented software; Mayer, C.; Hanenberg, S.; Robbes, R.; Tanter, E.; Stefik, A.

Abstract

Static and dynamic type systems (as well as more recently gradual type systems) are an important research topic in programming language design. Although the study of such systems plays a major role in research, relatively little is known about the impact of type systems on software development. Perhaps one of the more common arguments for static type systems is that they require developers to annotate their code with type names, which is thus claimed to improve the documentation of software. In contrast, one common argument against static type systems is that they decrease flexibility, which may make them harder to use. While positions such as these, both for and against static type systems, have been documented in the literature, there is little rigorous empirical evidence for or against either position. In this paper, we introduce a controlled experiment where 27 subjects performed programming tasks on an undocumented API with a static type system (which required type annotations) as well as a dynamic type system (which does not). Our results show that for some types of tasks, programmers were afforded faster task completion times using a static type system, while for others, the opposite held. In this work, we document the empirical evidence that led us to this conclusion and conduct an exploratory study to try and theorize why.

Summary

The experimental setup is very similar to the previous Hanenberg paper, so I'll just describe the main difference, which is that subjects used either Java, or a restricted subset of Groovy that was equivalent to dynamically typed Java. Subjects were students who had previous experience in Java, but not Groovy, giving some advantage for the Java tasks.

Groovy is both better and worse than Java

Task 1 was a trivial warm-up task. The authors note that it's possible that Java is superior on task 1 because the subjects had prior experience in Java. The authors speculate that, in general, Java is superior to untyped Java for more complex tasks, but they make it clear that they're just speculating and don't have enough data to conclusively support that conclusion.

How Do API Documentation and Static Typing Affect API Usability? Endrikat, S.; Hanenberg, S.; Robbes, Romain; Stefik, A.

Abstract

When developers use Application Programming Interfaces (APIs), they often rely on documentation to assist their tasks. In previous studies, we reported evidence indicating that static type systems acted as a form of implicit documentation, benefiting developer productivity. Such implicit documentation is easier to maintain, given it is enforced by the compiler, but previous experiments tested users without any explicit documentation. In this paper, we report on a controlled experiment and an exploratory study comparing the impact of using documentation and a static or dynamic type system on a development task. Results of our study both confirm previous findings and show that the benefits of static typing are strengthened with explicit documentation, but that this was not as strongly felt with dynamically typed languages.

There's an earlier study in this series with the following abstract:

In the discussion about the usefulness of static or dynamic type systems there is often the statement that static type systems improve the documentation of software. In the meantime there exists even some empirical evidence for this statement. One of the possible explanations for this positive influence is that the static type system of programming languages such as Java require developers to write down the type names, i.e. lexical representations which potentially help developers. Because of that there is a plausible hypothesis that the main benefit comes from the type names and not from the static type checks that are based on these names. In order to argue for or against static type systems it is desirable to check this plausible hypothesis in an experimental way. This paper describes an experiment with 20 participants that has been performed in order to check whether developers using an unknown API already benefit (in terms of development time) from the pure syntactical representation of type names without static type checking. The result of the study is that developers do benefit from the type names in an API's source code. But already a single wrong type name has a measurable significant negative impact on the development time in comparison to APIs without type names.

The languages used were Java and Dart. The university running the tests teaches in Java, so subjects had prior experience in Java. The task was one “where participants use the API in a way that objects need to be configured and passed to the API”, which was chosen because the authors thought that both types and documentation should have some effect. “The challenge for developers is to locate all the API elements necessary to properly configure [an] object”. The documentation was free-form text plus examples.

Taken at face value, it looks like types+documentation is a lot better than having one or the other, or neither. But since the subjects were students at a school that used Java, it's not clear how much of the effect is from familiarity with the language and how much is from the language. Moreover, the task was a single task that was chosen specifically because it was the kind of task where both types and documentation were expected to matter.

An Experiment About Static and Dynamic Type Systems; Hanenberg, S.

Abstract

Although static type systems are an essential part in teaching and research in software engineering and computer science, there is hardly any knowledge about what the impact of static type systems on the development time or the resulting quality for a piece of software is. On the one hand there are authors that state that static type systems decrease an application's complexity and hence its development time (which means that the quality must be improved since developers have more time left in their projects). On the other hand there are authors that argue that static type systems increase development time (and hence decrease the code quality) since they restrict developers to express themselves in a desired way. This paper presents an empirical study with 49 subjects that studies the impact of a static type system for the development of a parser over 27 hours working time. In the experiments the existence of the static type system has neither a positive nor a negative impact on an application's development time (under the conditions of the experiment).

Summary

This is another Hanenberg study with a basically sound experimental design, so I won't go into details about the design. Some unique parts are that, in order to control for familiarity and other things that are difficult to control for with existing languages, the author created two custom languages for this study.

The author says that the language has similarities to Smalltalk, Ruby, and Java, and that the language is a class-based OO language with single implementation inheritance and late binding.

The students had 16 hours of training in the new language before starting. The author argues that this was sufficient because “the language, its API as well as its IDE was kept very simple”. An additional 2 hours was spent to explain the type system for the static types group.

There were two tasks, a “small” one (implementing a scanner) and a “large” one (implementing a parser). The author found a statistically significant difference in time to complete the small task (the dynamic language was faster) and no difference in the time to complete the large task.

There are a number of reasons this result may not be generalizable. The author is aware of them and there's a long section on ways this study doesn't generalize as well as a good discussion on threats to validity.

Work In Progress: an Empirical Study of Static Typing in Ruby; Daly, M; Sazawal, V; Foster, J.

Abstract

In this paper, we present an empirical pilot study of four skilled programmers as they develop programs in Ruby, a popular, dynamically typed, object-oriented scripting language. Our study compares programmer behavior under the standard Ruby interpreter versus using Diamondback Ruby (DRuby), which adds static type inference to Ruby. The aim of our study is to understand whether DRuby's static typing is beneficial to programmers. We found that DRuby's warnings rarely provided information about potential errors not already evident from Ruby's own error messages or from presumed prior knowledge. We hypothesize that programmers have ways of reasoning about types that compensate for the lack of static type information, possibly limiting DRuby's usefulness when used on small programs.

Summary

Subjects came from a local Ruby user's group. Subjects implemented a simplified Sudoku solver and a maze solver. DRuby was randomly selected for one of the two problems for each subject. There were four subjects, but the authors changed the protocol after the first subject. Only three subjects had the same setup.

The authors find no benefit to having types. This is one of the studies that the first Hanenberg study mentions as a work their findings contradict. That first paper claimed that it was because their tasks were more complex, but it seems to me that this paper has a more complex task. One possible reason they found contradictory results is that the effect size is small. Another is that the specific type systems used matter, and that a DRuby v. Ruby study doesn't generalize to Java v. Groovy. Another is that the previous study attempted to remove anything hinting at type information from the dynamic implementation, including names that indicate types and API documentation. The participants of this study mention that they get a lot of type information from API docs, and the authors note that the participants encode type information in their method names.

This study was presented in a case study format, with selected comments from the participants and an analysis of their comments. The authors note that participants regularly think about types, and check types, even when programming in a dynamic language.

Haskell vs. Ada vs. C++ vs. Awk vs. ... An Experiment in Software Prototyping Productivity; Hudak, P; Jones, M.

Abstract

We describe the results of an experiment in which several conventional programming languages, together with the functional language Haskell, were used to prototype a Naval Surface Warfare Center (NSWC) requirement for a Geometric Region Server. The resulting programs and development metrics were reviewed by a committee chosen by the Navy. The results indicate that the Haskell prototype took significantly less time to develop and was considerably more concise and easier to understand than the corresponding prototypes written in several different imperative languages, including Ada and C++.

Summary

Subjects were given an informal text description for the requirements of a geo server. The requirements were behavior oriented and didn't mention performance. The subjects were “expert” programmers in the languages they used. They were asked to implement a prototype and track metrics such as dev time, lines of code, and docs. Metrics were all self reported, and no guidelines were given as to how they should be measured, so metrics varied between subjects. Also, some, but not all, subjects attended a meeting where additional information was given on the assignment.

Due to the time-frame and funding requirements, the requirements for the server were extremely simple; the median implementation was a couple hundred lines of code. Furthermore, the panel that reviewed the solutions didn't have time to evaluate or run the code; they based their findings on the written reports and oral presentations of the subjects.

Table of LOC, dev time, and lines of code

This study hints at a very interesting result, but considering all of its limitations, the fact that each language (except Haskell) was only tested once, and that other studies show much larger intra-group variance than inter-group variance, it's hard to conclude much from this study alone.

Unit testing isn't enough. You need static typing too; Farrer, E

Abstract

Unit testing and static type checking are tools for ensuring defect free software. Unit testing is the practice of writing code to test individual units of a piece of software. By validating each unit of software, defects can be discovered during development. Static type checking is performed by a type checker that automatically validates the correct typing of expressions and statements at compile time. By validating correct typing, many defects can be discovered during development. Static typing also limits the expressiveness of a programming language in that it will reject some programs which are ill-typed, but which are free of defects.

Many proponents of unit testing claim that static type checking is an insufficient mechanism for ensuring defect free software; and therefore, unit testing is still required if static type checking is utilized. They also assert that once unit testing is utilized, static type checking is no longer needed for defect detection, and so it should be eliminated.

The goal of this research is to explore whether unit testing does in fact obviate static type checking in real world examples of unit tested software.

Summary

The author took four Python programs and translated them to Haskell. Haskell's type system found some bugs. Unlike academic software engineering research, this study involves something larger than a toy program and looks at a type system that's more expressive than Java's type system. The programs were the NMEA Toolkit (9 bugs), MIDITUL (2 bugs), GrapeFruit (0 bugs), and PyFontInfo (6 bugs).

As far as I can tell, there isn't an analysis of the severity of the bugs. The programs were 2324, 2253, 2390, and 609 lines long, respectively, so the bugs found / LOC were 17 / 7576 = 1 / 446. For reference, in Code Complete, Steve McConnell estimates that 15-50 bugs per 1kLOC is normal. If you believe that estimate applies to this codebase, you'd expect that this technique caught between 4% and 15% of the bugs in this code. There's no particular reason to believe the estimate should apply, but we can keep this number in mind as a reference in order to compare to a similarly generated number from another study that we'll get to later.

The author does some analysis on how hard it would have been to find the bugs through testing, but only considers line coverage directed unit testing; the author comments that bugs might have have been caught by unit testing if they could be missed with 100% line coverage. This seems artificially weak — it's generally well accepted that line coverage is a very weak notion of coverage and that testing merely to get high line coverage isn't sufficient. In fact, it is generally considered insufficient to even test merely to get high path coverage, which is a much stronger notion of coverage than line coverage.

Gradual Typing of Erlang Programs: A Wrangler Experience; Sagonas, K; Luna, D

Abstract

Currently most Erlang programs contain no or very little type information. This sometimes makes them unreliable, hard to use, and difficult to understand and maintain. In this paper we describe our experiences from using static analysis tools to gradually add type information to a medium sized Erlang application that we did not write ourselves: the code base of Wrangler. We carefully document the approach we followed, the exact steps we took, and discuss possible difficulties that one is expected to deal with and the effort which is required in the process. We also show the type of software defects that are typically brought forward, the opportunities for code refactoring and improvement, and the expected benefits from embarking in such a project. We have chosen Wrangler for our experiment because the process is better explained on a code base which is small enough so that the interested reader can retrace its steps, yet large enough to make the experiment quite challenging and the experiences worth writing about. However, we have also done something similar on large parts of Erlang/OTP. The result can partly be seen in the source code of Erlang/OTP R12B-3.

Summary

This is somewhat similar to the study in “Unit testing isn't enough”, except that the authors of this study created a static analysis tool instead of translating the program into another language. The authors note that they spent about half an hour finding and fixing bugs after running their tool. They also point out some bugs that would be difficult to find by testing. They explicitly state “what's interesting in our approach is that all these are achieved without imposing any (restrictive) static type system in the language.” The authors have a follow-on paper, “Static Detection of Race Conditions in Erlang”, which extends the approach.

The list of papers that find bugs using static analysis without explicitly adding types is too long to list. This is just one typical example.

0install: Replacing Python; Leonard, T., pt2, pt3

Abstract

No abstract because this is a series of blog posts.

Summary

This compares ATS, C#, Go, Haskell, OCaml, Python and Rust. The author assigns scores to various criteria, but it's really a qualitative comparison. But it's interesting reading because it seriously considers the effect of language on a non-trivial codebase (30kLOC).

The author implemented parts of 0install in various languages and then eventually decided on Ocaml and ported the entire thing to Ocaml. There are some great comments about why the author chose Ocaml and what the author gained by using Ocaml over Python.

Verilog vs. VHDL design competition; Cooley, J

Abstract

No abstract because it's a usenet posting

Summary

Subjects were given 90 minutes to create a small chunk of hardware, a synchronous loadable 9-bit increment-by-3 decrement-by-5 up/down counter that generated even parity, carry and borrow, with the goal of optimizing for cycle time of the synthesized result. For the software folks reading this, this is something you'd expect to be able to do in 90 minutes if nothing goes wrong, or maybe if only a few things go wrong.

Subjects were judged purely by how optimized their result was, as long as it worked. Results that didn't pass all tests were disqualified. Although the task was quite simple, it was made substantially more complicated by the strict optimization goal. For any software readers out there, this task is approximately as complicated as implementing the same thing in assembly, where your assembler takes 15-30 minutes to assemble something.

Subjects could use Verilog (unityped) or VHDL (typed). 9 people chose Verilog and 5 chose VHDL.

During the expierment, there were a number of issues that made things easier or harder for some subjects. Overall, Verilog users were affected more negatively than VHDL users. The license server for the Verilog simulator crashed. Also, four of the five VHDL subjects were accidentally given six extra minutes. The author had manuals for the wrong logic family available, and one Verilog user spent 10 minutes reading the wrong manual before giving up and using his intuition. One of the Verilog users noted that they passed the wrong version of their code along to be tested and failed because of that. One of the VHDL users hit a bug in the VHDL simulator.

Of the 9 Verilog users, 8 got something synthesized before the 90 minute deadline; of those, 5 had a design that passed all tests. None of the VHDL users were able to synthesize a circuit in time.

Two of the VHDL users complained about issues with types “I can't believe I got caught on a simple typing error. I used IEEE std_logic_arith, which requires use of unsigned & signed subtypes, instead of std_logic_unsigned.”, and "I ran into a problem with VHDL or VSS (I'm still not sure.) This case statement doesn't analyze: ‘subtype two_bits is unsigned(1 downto 0); case two_bits'(up & down)...' But what worked was: ‘case two_bits'(up, down)...' Finally I solved this problem by assigning the concatenation first to a[n] auxiliary variable."

Comparing mathematical provers; Wiedijk, F

Abstract

We compare fifteen systems for the formalizations of mathematics with the computer. We present several tables that list various properties of these programs. The three main dimensions on which we compare these systems are: the size of their library, the strength of their logic and their level of automation.

Summary

The author compares the type systems and foundations of various theorem provers, and comments on their relative levels of proof automation.

Type systems of provers Foundations of provers Graph of automation level of provers

The author looked at one particular problem (proving the irrationality of the square root of two) and examined how different systems handle the problem, including the style of the proof and its length. There's a table of lengths, but it doesn't match the updated code examples provided here. For instance, that table claims that the ACL2 proof is 206 lines long, but there's a 21 line ACL2 proof here.

The author has a number of criteria for determining how much automation prover provides, but he freely admits that it's highly subjective. The author doesn't provide the exact rubric used for scoring, but he mentions that a more automated interaction style, user automation, powerful built-in automation, and the Poincare principle (basically whether the system lets you write programs to solve proofs algorithmically) all count towards being more automated, and more powerful logic (e.g., first-order v. higher-order), logical framework dependent types, and de Bruijn criterion (having a small guaranteed kernel) count towards being more mathematical.

Do Programming Languages Affect Productivity? A Case Study Using Data from Open Source Projects; Delory, D; Knutson, C; Chun, S

Abstract

Brooks and others long ago suggested that on average computer programmers write the same number of lines of code in a given amount of time regardless of the programming language used. We examine data collected from the CVS repositories of 9,999 open source projects hosted on SourceForge.net to test this assump- tion for 10 of the most popular programming languages in use in the open source community. We find that for 24 of the 45 pairwise comparisons, the programming language is a significant factor in determining the rate at which source code is written, even after accounting for variations between programmers and projects.

Summary

The authors say “our goal is not to construct a predictive or explanatory model. Rather, we seek only to develop a model that sufficiently accounts for the variation in our data so that we may test the significance of the estimated effect of programming language.” and that's what they do. They get some correlations, but it's hard to conclude much of anything from them.

The Unreasonable Effectiveness of Dynamic Typing for Practical Programs; Smallshire, R

Abstract

Some programming language theorists would have us believe that the one true path to working systems lies in powerful and expressive type systems which allow us to encode rich constraints into programs at the time they are created. If these academic computer scientists would get out more, they would soon discover an increasing incidence of software developed in languages such a Python, Ruby and Clojure which use dynamic, albeit strong, type systems. They would probably be surprised to find that much of this software—in spite of their well-founded type-theoretic hubris—actually works, and is indeed reliable out of all proportion to their expectations.This talk—given by an experienced polyglot programmer who once implemented Hindley Milner static type inference for “fun”, but who now builds large and successful systems in Python—explores the disconnect between the dire outcomes predicted by advocates of static typing versus the near absence of type errors in real world systems built with dynamic languages: Does diligent unit testing more than make up for the lack of static typing? Does the nature of the type system have only a low-order effect on reliability compared to the functional or imperative programming paradigm in use? How often is the dynamism of the type system used anyway? How much type information can JITs exploit at runtime? Does the unwarranted success of dynamically typed languages get up the nose of people who write Haskell?

Summary

The speaker used data from Github to determine that approximately 2.7% of Python bugs are type errors. Python's TypeError, AttributeError, and NameError were classified as type errors. The speaker rounded 2.7% down to 2% and claimed that 2% of errors were type related. The speaker mentioned that on a commercial codebase he worked with, 1% of errors were type related, but that could be rounded down from anything less than 2%. The speaker mentioned looking at the equivalent errors in Ruby, Clojure, and other dynamic languages, but didn't present any data on those other languages.

This data might be good but it's impossible to tell because there isn't enough information about the methodology. Something this has going for is that the number is in the right ballpark, compared to the made up number we got when compared the bug rate from Code Complete to the number of bugs found by Farrer. Possibly interesting, but thin.

Summary of summaries

This isn't an exhaustive list. For example, I haven't covered “An Empirical Comparison of Static and Dynamic Type Systems on API Usage in the Presence of an IDE: Java vs. Groovy with Eclipse”, and “Do developers benefit from generic types?: an empirical comparison of generic and raw types in java” because they didn't seem to add much to what we've already seen.

I didn't cover a number of older studies that are in the related work section of almost all the listed studies both because the older studies often cover points that aren't really up for debate anymore and also because the experimental design in a lot of those older papers leaves something to be desired. Feel free to ping me if there's something you think should be added to the list.

Not only is this list not exhaustive, it's not objective and unbiased. If you read the studies, you can get a pretty good handle on how the studies are biased. However, I can't provide enough information for you to decide for yourself how the studies are biased without reproducing most of the text of the papers, so you're left with my interpretation of things, filtered through my own biases. That can't be helped, but I can at least explain my biases so you can discount my summaries appropriately.

I like types. I find ML-like languages really pleasant to program in, and if I were king of the world, we'd all use F# as our default managed language. The situation with unmanaged languages is a bit messier. I certainly prefer C++ to C because std::unique_ptr and friends make C++ feel a lot safer than C. I suspect I might prefer Rust once it's more stable. But while I like languages with expressive type systems, I haven't noticed that they make me more productive or less bug prone0.

Now that you know what my biases are, let me give you my interpretation of the studies. Of the controlled experiments, only three show an effect large enough to have any practical significance. The Prechelt study comparing C, C++, Java, Perl, Python, Rexx, and Tcl; the Endrikat study comparing Java and Dart; and Cooley's experiment with VHDL and Verilog. Unfortunately, they all have issues that make it hard to draw a really strong conclusion.

In the Prechelt study, the populations were different between dynamic and typed languages, and the conditions for the tasks were also different. There was a follow-up study that illustrated the issue by inviting Lispers to come up with their own solutions to the problem, which involved comparing folks like Darius Bacon to random undergrads. A follow-up to the follow-up literally involves comparing code from Peter Norvig to code from random college students.

In the Endrikat study, they specifically picked a task where they thought static typing would make a difference, and they drew their subjects from a population where everyone had taken classes using the statically typed language. They don't comment on whether or not students had experience in the dynamically typed language, but it seems safe to assume that most or all had less experience in the dynamically typed language.

Cooley's experiment was one of the few that drew people from a non-student population, which is great. But, as with all of the other experiments, the task was a trivial toy task. While it seems damning that none of the VHDL (static language) participants were able to complete the task on time, it is extremely unusual to want to finish a hardware design in 1.5 hours anywhere outside of a school project. You might argue that a large task can be broken down into many smaller tasks, but a plausible counterargument is that there are fixed costs using VHDL that can be amortized across many tasks.

As for the rest of the experiments, the main takeaway I have from them is that, under the specific set of circumstances described in the studies, any effect, if it exists at all, is small.

Moving on to the case studies, the two bug finding case studies make for interesting reading, but they don't really make a case for or against types. One shows that transcribing Python programs to Haskell will find a non-zero number of bugs of unknown severity that might not be found through unit testing that's line-coverage oriented. The pair of Erlang papers shows that you can find some bugs that would be difficult to find through any sort of testing, some of which are severe, using static analysis.

As a user, I find it convenient when my compiler gives me an error before I run separate static analysis tools, but that's minor, perhaps even smaller than the effect size of the controlled studies listed above.

I found the 0install case study (that compared various languages to Python and eventually settled on Ocaml) to be one of the more interesting things I ran across, but it's the kind of subjective thing that everyone will interpret differently, which you can see by looking.

This fits with the impression I have (in my little corner of the world, ACL2, Isabelle/HOL, and PVS are the most commonly used provers, and it makes sense that people would prefer more automation when solving problems in industry), but that's also subjective.

And then there are the studies that mine data from existing projects. Unfortunately, I couldn't find anybody who did anything to determine causation (e.g., find an appropriate instrumental variable), so they just measure correlations. Some of the correlations are unexpected, but there isn't enough information to determine why. The lack of any causal instrument doesn't stop people like Ray et al. from making strong, unsupported, claims.

The only data mining study that presents data that's potentially interesting without further exploration is Smallshire's review of Python bugs, but there isn't enough information on the methodology to figure out what his study really means, and it's not clear why he hinted at looking at data for other languages without presenting the data2.

Some notable omissions from the studies are comprehensive studies using experienced programmers, let alone studies that have large populations of "good" or "bad" programmers, looking at anything approaching a significant project (in places I've worked, a three month project would be considered small, but that's multiple orders of magnitude larger than any project used in a controlled study), using "modern" statically typed languages, using gradual/optional typing, using modern mainstream IDEs (like VS and Eclipse), using modern radical IDEs (like LightTable), using old school editors (like Emacs and vim), doing maintenance on a non-trivial codebase, doing maintenance with anything resembling a realistic environment, doing maintenance on a codebase you're already familiar with, etc.

If you look at the internet commentary on these studies, most of them are passed around to justify one viewpoint or another. The Prechelt study on dynamic vs. static, along with the follow-ups on Lisp are perennial favorites of dynamic language advocates, and github mining study has recently become trendy among functional programmers.

Other than cherry picking studies to confirm a long-held position, the most common response I've heard to these sorts of studies is that the effect isn't quantifiable by a controlled experiment. However, I've yet to hear a specific reason that doesn't also apply to any other field that empirically measures human behavior. Compared to a lot of those fields, it's easy to run controlled experiments or do empirical studies. It's true that controlled studies only tell you something about a very limited set of circumstances, but the fix to that isn't to dismiss them, but to fund more studies. It's also true that it's tough to determine causation from ex-post empirical studies, but the solution isn't to ignore the data, but to do more sophisticated analysis. For example, econometric methods are often able to make a case for causation with data that's messier than the data we've looked at here.

The next most common response is that their viewpoint is still valid because their specific language or use case isn't covered. Maybe, but if the strongest statement you can make for your position is that there's no empirical evidence against the position, that's not much of a position.

If you've managed to read this entire thing without falling asleep, you might be interested in my opinion on tests.

Responses

Here are the responses I've gotten from people mentioned in this post. Robert Smallshire said "Your review article is very good. Thanks for taking the time to put it together." On my comment about the F# "mistake" vs. trolling, his reply was "Neither. That torque != energy is obviously solved by modeling quantities not dimensions. The point being that this modeling of quantities with types takes effort without necessarily delivering any value." Not having done much with units myself, I don't have an informed opinion on this, but my natural bias is to try to encode the information in types if at all possible.

Bartosz Milewski said "Guilty as charged!". Wow. Much Respect. But notice that, as of this update, The correction has been retweeted 1/25th as often as the original tweet. People want to believe there's evidence their position is superior. People don't want to believe the evidence is murky, or even possibly against them. Misinformation people want to believe spreads faster than information people don't want to believe.

On a related twitter conversation, Andreas Stefik said "That is not true. It depends on which scientific question. Static vs. Dynamic is well studied.", "Profound rebuttal. I had better retract my peer reviewed papers, given this new insight!", "Take a look at the papers...", and "This is a serious misrepresentation of our studies." I muted the guy since it didn't seem to be going anywhere, but it's possible there was a substantive response buried in some later tweet. It's pretty easy to take twitter comments out of context, so check out the thread yourself if you're really curious.

I have a lot of respect for the folks who do these experiments, which is, unfortunately, not mutual. But the really unfortunate thing is that some of the people who do these experiments think that static v. dynamic is something that is, at present, "well studied". There are plenty of equally difficult to study subfields in the social sciences that have multiple orders of magnitude more research going on, that are considered open problems, but at least some researchers already consider this to be well studied!

Acknowledgements

Thanks to Leah Hanson, Joe Wilder, Robert David Grant, Jakub Wilk, Rich Loveland, Eirenarch, Edward Knight, and Evan Farrer for comments/corrections/discussion.


  1. This was from a talk at Strange Loop this year. The author later clarified his statement with "To me, this follows immediately (a technical term in logic meaning the same thing as “trivially”) from the Curry-Howard Isomorphism we discussed, and from our Types vs. Tests: An Epic Battle? presentation two years ago. If types are theorems (they are), and implementations are proofs (they are), and your SLA is a guarantee of certain behavior of your system (it is), then how can using technology that precludes forbidding undesirable behavior of your system before other people use it (dynamic typing) possibly be anything but unethical?" [return]
  2. Just as an aside, I find the online responses to Smallshire's study to be pretty great. There are, of course, the usual responses about how his evidence is wrong and therefore static types are, in fact, beneficial because there's no evidence against them, and you don't need evidence for them because you can arrive at the proper conclusion using pure reason. The really interesting bit is that, at one point, Smallshire presents an example of an F# program that can't catch a certain class of bug via its type system, and the online response is basically that he's an idiot who should have written his program in a different way so that the type system should have caught the bug. I can't tell if Smallshire's bug was an honest mistake or masterful trolling. [return]
show more
How often is the build broken?
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-11-10 00:00:00 | Created: 2026-07-23 05:18:40

I've noticed that builds are broken and tests fail a lot more often on open source projects than on “work” projects. I wasn't sure how much of that was my perception vs. reality, so I grabbed the Travis CI data for a few popular categories on GitHub1.

Graph of build reliability. Props to nu, oryx, caffe, catalyst, and Scala.

For reference, at every place I've worked, two 9s of reliability (99% uptime) on the build would be considered bad. That would mean that the build is failing for over three and a half days a year, or seven hours per month. Even three 9s (99.9% uptime) is about forty-five minutes of downtime a month. That's kinda ok if there isn't a hard system in place to prevent people from checking in bad code, but it's quite bad for a place that's serious about having working builds.

By contrast, 2 9s of reliability is way above average for the projects I pulled data for2 -- only 8 of 40 projects are that reliable. Almost twice as many projects -- 15 of 40 -- don't even achieve one 9 of uptime. And my sample is heavily biased towards reliable projects. There are projects that were well-known enough to be “featured” in a hand curated list by GitHub. That's already biases the data right there. And then I only grabbed data from the projects that care enough about testing to set up TravisCI3, which introduces an even stronger bias.

To make sure I wasn't grabbing bad samples, I removed any initial set of failing tests (there are often a lot of fails as people try to set up Travis and have it misconfigured) and projects that that use another system for tracking builds that only have Travis as an afterthought (like Rust)4.

Why doesn't the build fail all the time at work? Engineers don't like waiting for someone else to unbreak the build and managers can do the back of the envelope calculation which says that N idle engineers * X hours of build breakage = $Y of wasted money.

But that same logic applies to open source projects! Instead of wasting dollars, contributor's time is wasted.

Web programmers are hyper-aware of how 100ms of extra latency on a web page load has a noticeable effect on conversion rate. Well, what's the effect on conversion rate when a potential contributor to your project spends 20 minutes installing dependencies and an hour building your project only to find the build is broken?

I used to dig through these kinds of failures to find the bug, usually assuming that it must be some configuration issue specific to my machine. But having spent years debugging failures I run into with make check on a clean build, I've found that it's often just that someone checked in bad code. Nowadays, if I'm thinking about contributing to a project or trying to fix a bug and the build doesn't work, I move on to another project.

The worst thing about regular build failures is that they're easy5 to prevent. Graydon Hoare literally calls keeping a clean build the “not rocket science rule”, and wrote an open source tool (bors) anyone can use to do not-rocket-science. And yet, most open source projects still suffer through broken and failed builds, along with the associated cost of lost developer time and lost developer “conversions”.

Please don't read too much into the individual data in the graph. I find it interesting that DevOps projects tend to be more reliable than languages, which tend to be more reliable than web frameworks, and that ML projects are all over the place (but are mostly reliable). But when it comes to individual projects, all sorts of stuff can cause a project to have bad numbers.

Thanks to Kevin Lynagh, Leah Hanson, Michael Smith, Katerina Barone-Adesi, and Alexey Romanov for comments.

Also, props to Michael Smith of Puppetlabs for a friendly ping and working through the build data for puppet to make sure there wasn't a bug in my scripts. This is one of my most maligned blog posts because no one wants to believe the build for their project is broken more often than the build for other projects. But even though it only takes about a minute to pull down the data for a project and sanity check it using the links in this post, only one person actually looked through the data with me, while a bunch of people told me how it must quite obviously be incorrect without ever checking the data.

This isn't to say that I don't have any bugs. This is a quick hack that probably has bugs and I'm always happy to get bugreports! But some non-bugs that have been repeatedly reported are getting data from all branches instead of the main branch, getting data for all PRs and not just code that's actually checked in to the main branch, and using number of failed builds instead of the amount of time that the build is down. I'm pretty sure that you can check that any of those claims are false in about the same amount of time that it takes to make the claim, but that doesn't stop people from making the claim.


  1. Categories determined from GitHub's featured projects lists, which seem to be hand curated. [return]
  2. Wouldn't it be nice if I had test coverage data, too? But I didn't try to grab it since this was a quick 30-minute project and coming up with cross-language test coverage comparisons isn't trivial. However, I spot checked some projects and the ones that do poorly conform to an engineering version of what Tyler Cowen calls "The Law of Below Averages" -- projects that often have broken/failed builds also tend to have very spotty test coverage. [return]
  3. I used the official Travis API script, modified to return build start time instead of build finish time. Even so, build start time isn't exactly the same as check-in time, which introduces some noise. Only data against the main branch (usually master) was used. Some data was incomplete because their script either got a 500 error from the Travis API server, or ran into a runtime syntax error. All errors happened with and without my modifications, which is pretty appropriate for this blog post.

    If you want to reproduce the results, apply this patch to the official script, run it with the appropriate options (usually with --branch master, but not always), and then aggregate the results. You can use this script, but if you don't have Julia it may be easier to just do it yourself.

    [return]
  4. I think I filtered all the projects that were actually using a different testing service out. Please let me know if there are any still in my list. This removed one project with one-tenth of a 9 and two projects with about half a 9. BTW, removing the initial Travis fails for these projects bumped some of them up between half a 9 and a full 9 and completely eliminated a project that's had failing Travis tests for over a year. The graph shown looks much better than the raw data, and it's still not good. [return]
  5. Easy technically. Hard culturally. Michael Smith brought up the issue of intermittent failures. When you get those, whether that's because the project itself is broken or because the CI build is broken, people will start checking in bad code. There are environments where people don't do that -- for the better part of a decade, I worked at a company where people would track down basically any test failure ever, even (or especially) if the failure was something that disappeared with no explanation. How do you convince people to care that much? That's hard.

    How do you convince people to use a system like bors, where you don't have to care to avoid breaking the build? That's much easier, though still harder than the technical problems involved in building bors.

    [return]
show more
Speeding up this site by 50x
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-11-17 00:00:00 | Created: 2026-07-23 05:18:40

I've seen all these studies that show how a 100ms improvement in page load time has a significant effect on page views, conversion rate, etc., but I'd never actually tried to optimize my site. This blog is a static Octopress site, hosted on GitHub Pages. Static sites are supposed to be fast, and GitHub Pages uses Fastly, which is supposed to be fast, so everything should be fast, right?

Not having done this before, I didn't know what to do. But in a great talk on how the internet works, Dan Espeset suggested trying webpagetest; let's give it a shot.

Here's what it shows with my nearly stock Octopress setup1. The only changes I'd made were enabling Google Analytics, the social media buttons at the bottom of posts, and adding CSS styling for tables (which are, by default, unstyled and unreadable).

12 seconds to the first page view! What happened? I thought static sites were supposed to be fast. The first byte gets there in less than half a second, but the page doesn't start rendering until 9 seconds later.

Lots of js, CSS, and fonts

Looks like the first thing that happens is that we load a bunch of js and CSS. Looking at the source, we have all this js in source/_includes/head.html.

<script src="{{ root_url }}/javascripts/modernizr-2.0.js"></script>
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script>!window.jQuery && document.write(unescape('%3Cscript src="./javascripts/lib/jquery.min.js"%3E%3C/script%3E'))</script>
<script src="{{ root_url }}/javascripts/octopress.js" type="text/javascript"></script>
{% include google_analytics.html %}

I don't know anything about web page optimization, but Espeset mentioned that js will stall page loading and rendering. What if we move the scripts to source/_includes/custom/after_footer.html?

That's a lot better! We've just saved about 4 seconds on load time and on time to start rendering.

Those script tags load modernizer, jquery, octopress.js, and some google analytics stuff. What is in this octopress.js anyway? It's mostly code to support stuff like embedding flash videos, delicious integration, and github repo integration. There are a few things that do get used for my site, but most of that code is dead weight.

Also, why are there multiple js files? Espeset also mentioned that connections are finite resources, and that we'll run out of simultaneous open connections if we have a bunch of different files. Let's strip out all of that unused js and combine the remaining js into a single file.

Much better! But wait a sec. What do I need js for? As far as I can tell, the only thing my site is still using octopress's js for is so that you can push the right sidebar back and forth by clicking on it, and jquery and modernizer are only necessary for the js used in octopress. I never use that, and according to in-page analytics no one else does either. Let's get rid of it.

That didn't change total load time much, but the browser started rendering sooner. We're down to having the site visually complete after 1.2s, compared to 9.6s initially -- an 8x improvement.

What's left? There's still some js for the twitter and fb widgets at the bottom of each post, but those all get loaded after things are rendered, so they don't really affect the user's experience, even though they make the “Load Time” number look bad.

That's a lot of fonts!

This is a pie chart of how many bytes of my page are devoted to each type of file. Apparently, the plurality of the payload is spent on fonts. Despite my reference post being an unusually image heavy blog post, fonts are 43.8% and images are such a small percentage that webpagetest doesn't even list the number. Doesn't my browser already have some default fonts? Can we just use those?

Turns out, we can. The webpage is now visually complete in 0.9s -- a 12x improvement. The improvement isn't quite as dramatic for “Repeat View”2 -- it's only an 8.6x improvement there -- but that's still pretty good.

The one remaining “obvious” issue is that the header loads two css files, one of which isn't minified. This uses up two connections and sends more data than necessary. Minifying the other css file and combining them speeds this up even further.

Time to visually complete is now 0.7s -- a 15.6x improvement3. And that's on a page that's unusually image heavy for my site.

Mostly image load time

At this point the only things that happen before the page starts displaying are:, loading the HTML, loading the one css file, and loading the giant image (reliability.png).

We've already minified the css, so the main thing left to do is to make giant image better. I already ran optipng -o7 -zm1-9 on all my images, but ImageOptim was able to shave off another 4% of the image, giving a slight improvement. Across all the images in all my posts, ImageOptim was able to reduce images by an additional 20% over optipng, but it didn't help much in this case.

I also tried specifying the size of the image to see if that would let the page render before the image was finished downloading, but it didn't result in much of a difference.

After that, I couldn't think of anything else to try, but webpagetest had some helpful suggestions.

Blargh github pages

Apparently, the server I'm on is slow (it gets a D in sending the first byte after the initial request). It also recommends caching static content, but when I look at the individual suggestions, they're mostly for widgets I don't host/control. I should use a CDN, but Github Pages doesn't put content on a CDN for bare domains unless you use a DNS alias record, and my DNS provider doesn't support alias records. That's two reasons to stop servering from Github Pages (or perhaps one reason to move off Github Pages and one reason to get another DNS provider), so I switched to Cloudflare, which shaved over 100ms off the time to first byte.

Note that if you use Cloudflare for a static site, you'll want to create a "Page Rule" and enable "Cache Everything". By default, Cloudflare doesn't cache HTML, which is sort of pointless on a static blog that's mostly HTML. If you've done the optimizations here, you'll also want to avoid their "Rocket Loader" thing which attempts to load js asynchronously by loading blocking javascript. "Rocket Loader" is like AMP, in that it can speed up large, bloated, websites, but is big enough that it slows down moderately optimized websites.

Here's what happened after I initally enabled Cloudflare without realizing that I needed to create a "Page Rule".

Cloudflare saves 80MB out of 1GB

That's about a day's worth of traffic in 2013. Initially, Cloudflare was serving my CSS and redirecting to Github Pages for the HTML. Then I inlined my CSS and Cloudflare literally did nothing. Overall, Cloudflare served 80MB out of 1GB of traffic because it was only caching images and this blog is relatively light on images.

I haven't talked about inlining CSS, but it's easy and gives a huge speedup on the first visit since it means only one connection is required to display the page, instead of two sequentialy connections. It's a disadvantage on future visits since it means that the CSS has to be re-downloaded for each page, but since most of my traffic is from people running across a single blog post, who don't click through to anything else, it's a net win. In _includes/head.html

<link href="{{ root_url }}/stylesheets/all.css" media="screen, projection" rel="stylesheet" type="text/css">

should change to

{\% include all.css %}

In addition, there's a lot of pointless cruft in the css. Removing the stuff that, as someone who doesn't know CSS can spot as pointless (like support for delicious, support for Firefox 3.5 and below, lines that firefox flags as having syntax errors such as no-wrap instead of nowrap) cuts down the remaining CSS by about half. There's a lot of duplication remaining and I expect that the CSS could be reduced by another factor of 4, but that would require actually knowing CSS. Just doing those things, we get down to .4s before the webpage is visually complete.

Inlining css

That's a 10.9/.4 = 27.5 fold speedup. The effect on mobile is a lot more dramatic; there, it's closer to 50x.

I'm not sure what to think about all this. On the one hand, I'm happy that I was able to get a 25x-50x speedup on my site. On the other hand, I associate speedups of that magnitude with porting plain Ruby code to optimized C++, optimized C++ to a GPU, or GPU to quick-and-dirty exploratory ASIC. How is it possible that someone with zero knowledge of web development can get that kind of speedup by watching one presentation and then futzing around for 25 minutes? I was hoping to maybe find 100ms of slack, but it turns out there's not just 100ms, or even 1000ms, but 10000ms of slack in a Octopress setup. According to a study I've seen, going from 1000ms to 3000ms costs you 20% of your readers and 50% of your click-throughs. I haven't seen a study that looks at going from 400ms to 10900ms because the idea that a website would be that slow is so absurd that people don't even look into the possibility. But many websites are that slow!4

Update

I found it too hard to futz around with trimming down the massive CSS file that comes with Octopress, so I removed all of the CSS and then added a few lines to allow for a nav bar. This makes almost no difference on the desktop benchmark above, but it's a noticable improvement for slow connections. The difference is quite dramatic for 56k connections as well as connections with high packetloss.

Starting the day I made this change, my analytics data shows a noticeable improvement in engagement and traffic. There are too many things confounded here to say what caused this change (performance increase, total lack of styling, etc.), but there are a couple of things find interesting about this. First, it seems to likely show that the advice that it's very important to keep line lengths short is incorrect since, if that had a very large impact, it would've overwhelmed the other changes and resulted in reduced engagement and not increased engagement. Second, despite the Octopress design being widely used and lauded (it appears to have been the most widely used blog theme for programmers when I started my blog), it appears to cause a blog (or at least this blog) to get less readership than literally having no styling at all. Having no styling is surely not optimal, but there's something a bit funny about no styling beating the at-the-time most widely used programmer blog styling, which means it likely also beat wordpress, svtble, blogspot, medium, etc., since those have most oof the same ingredients as Octopress.

Resources

Unfortunately, the video of the presentation I'm referring to is restricted RC alums. If you're an RC alum, check this out. Otherwise high-performance browser networking is great, but much longer.

Acknowledgements

Thanks to Leah Hanson, Daniel Espeset, and Hugo Jobling for comments/corrections/discussion.

I'm not a front-end person, so I might be totally off in how I'm looking at these benchmarks. If so, please let me know.


  1. From whatever version was current in September 2013. It's possible some of these issues have been fixed, but based on the extremely painful experience of other people who've tried to update their Octopress installs, it didn't seem worth making the attempt to get a newer version of Octopress. [return]
  2. Why is “Repeat View” slower than “First View”? [return]
  3. If you look at a video of loading the original vs. this version, the difference is pretty dramatic. [return]
  4. For example, slashdot takes 15s to load over FIOS. The tests shown above were done on Cable, which is substantially slower. [return]
show more
Everything is broken
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-11-18 00:00:00 | Created: 2026-07-23 05:18:40

If I had to guess, I'd say I probably work around hundreds of bugs in an average week, and thousands in a bad week. It's not unusual for me to run into a hundred new bugs in a single week. But I often get skepticism when I mention that I run into multiple new (to me) bugs per day, and that this is inevitable if we don't change how we write tests. Well, here's a log of one week of bugs, limited to bugs that were new to me that week. After a brief description of the bugs, I'll talk about what we can do to improve the situation. The obvious answer to spend more effort on testing, but everyone already knows we should do that and no one does it. That doesn't mean it's hopeless, though.

One week of bugs

Ubuntu

When logging into my machine, I got a screen saying that I entered my password incorrectly. After a five second delay, it logged me in anyway. This is probably at least two bugs, perhaps more.

GitHub

GitHub switched from Pygments to whatever they use for Atom, breaking syntax highlighting for most languages. The HN comments on this indicate that it's not just something that affects obscure languages; Java, PHP, C, and C++ all have noticeable breakage.

In a GitHub issue, a GitHub developer says

You're of course free to fork the Racket bundle and improve it as you see fit. I'm afraid nobody at GitHub works with Racket so we can't judge what proper highlighting looks like. But we'll of course pull your changes thanks to the magic of O P E N S O U R C E.

A bit ironic after the recent keynote talk by another GitHub employee titled “move fast and break nothing”. Not to mention that it's unlikely to work. The last time I submitted a PR to linguist, it only got merged after I wrote a blog post pointing out that they had 100s of open PRs, some of which were a year old, which got them to merge a bunch of PRs after the post hit reddit. As far as I can tell, "the magic of O P E N S O U R C E" is code for the magic of hitting the front page of reddit/HN or having lots of twitter followers.

Also, icons were broken for a while. Was that this past week?

LinkedIn

After replying to someone's “InMail”, I checked on it a couple days later, and their original message was still listed as unread (with no reply). Did it actually send my reply? I had no idea, until the other person responded.

Inbox

The Inbox app (not to be confused with Inbox App) notifies you that you have a new message before it actually downloads the message. It takes an arbitrary amount of time before the app itself gets the message, and refreshing in the app doesn't cause the message to download.

The other problem with notifications is that they sometimes don't show up when you get a message. About half the time I get a notification from the gmail app, I also get a notification from the Inbox app. The other half of the time, the notification is dropped.

Overall, I get a notification for a message that I can read maybe 1/3 of the time.

Google Analytics

Some locations near the U.S. (like Mexico City and Toronto) aren't considered worthy of getting their own country. The location map shows these cities sitting in the blue ocean that's outside of the U.S.

Octopress

Footnotes don't work correctly on the main page if you allow posts on the main page (instead of the index) and use the syntax to put something below the fold. Instead of linking to the footnote, you get a reference to anchor text that goes nowhere. This is in addition to the other footnote bug I already knew about.

Tags are only downcased in some contexts but not others, which means that any tags with capitalized letters (sometimes) don't work correctly. I don't even use tags, but I noticed this on someone else's blog.

My Atom feed doesn't work correctly.

If you consider performance bugs to be problems, I noticed so many of those this past week that they have their own blog post.

Running with Rifles (Game)

Weapons that are supposed to stun injure you instead. I didn't even realize that was a bug until someone mentioned that would be fixed in the next version.

It's possible to stab people through walls.

If you're holding a key when the level changes, your character keeps doing that action continuously during the next level, even after you've released the key.

Your character's position will randomly get out of sync from the server. When that happens, the only reliable fix I've found is to randomly shoot for a while. Apparently shooting causes the client to do something like send a snapshot of your position to the server? Not sure why that doesn't just happen regularly.

Vehicles can randomly spawn on top of you, killing you.

You can randomly spawn under a vehicle, killing you.

AI teammates don't consider walls or buildings when throwing grenades, which often causes them to kill themselves.

Grenades will sometimes damage the last vehicle you were in even when you're nowhere near the vehicle.

AI vehicles can get permanently stuck on pretty much any obstacle.

This is the first video game I've played in about 15 years. I tend to think of games as being pretty reliable, but that's probably because games were much simpler 15 years ago. MS Paint doesn't have many bugs, either.

Update: The sync issue above is caused by memory leaks. I originally thought that the game just had very poor online play code, but it turns out it's actually ok for the first 6 hours or so after a server restart. There are scripts around to restart the servers periodically, but they sometimes have bugs which cause them to stop running. When that happens on the official servers, the game basically becomes unplayable online.

Julia

Unicode sequence causes match/ismatch to blow up with a bounds error.

Unicode sequence causes using a string as a hash index to blow up with a bounds error.

Exception randomly not caught by catch. This sucks because putting things in a try/catch was the workaround for the two bugs above. I've seen other variants of this before; it's possible this shouldn't count as a new bug because it might be the same root cause as some bug I've already seen.

Function (I forget which) returns completely wrong results when given bad length arguments. You can even give it length arguments of the wrong type, and it will still “work” instead of throwing an exception or returning an error.

If API design bugs count, methods that work operation on iterables sometimes take the stuff as the first argument and sometimes don't. There are way too many of these to list. To take one example, match takes a regex first and a string second, whereas search takes a string first and a regex second. This week, I got bit by something similar on a numerical function.

And of course I'm still running into the 1+ month old bug that breaks convert, which is pervasive enough that anything that causes it to happen renders Julia unusable.

Here's one which might be an OS X bug? I had some bad code that caused an infinite loop in some Julia code. Nothing actually happened in the while loop, so it would just run forever. Oops. The bug is that this somehow caused my system to run out of memory and become unresponsive. Activity monitor showed that the kernel was taking an ever increasing amount of memory, which went away when I killed the Julia process.

I won't list bugs in packages because there are too many. Even in core Julia, I've run into so many Julia bugs that I don't file bugs any more. It's just too much of an interruption. When I have some time, I should spend a day filing all the bugs I can remember, but I think it would literally take a whole day to write up a decent, reproducible, bug report for each bug.

See this post for more on why I run into so many Julia bugs.

Google Hangouts

On starting a hangout: "This video call isn't available right now. Try again in a few minutes.".

Same person appears twice in contacts list. Both copies have the same email listed, and double clicking on either brings me to the same chat window.

UW Health

The latch mechanism isn't quite flush to the door on about 10% of lockers, so your locker won't actually be latched unless you push hard against the door while moving the latch to the closed position (which means that, unless you check, it's easy to think you locked the locker door without having actually locked it; no doubt there's a helpful sign saying that they're not responsible for any theft that might occur).

There's no visual (or other) indication that the latch failed to latch. As far as I can tell, the only way to check is to tug on the handle to see if the door opens after you've tried to latch it.

Coursera, Mining Massive Data Sets

Selecting the correct quiz answer gives you 0 points. The workaround (independently discovered by multiple people on the forums) is to keep submitting until the correct answer gives you 1 point. This is a week after a quiz had incorrect answer options which resulted in there being no correct answers.

Facebook

If you do something “wrong” with the mouse while scrolling down on someone's wall, the blue bar at the top can somehow transform into a giant block the size of your cover photo that doesn't go away as you scroll down.

Clicking on the activity sidebar on the right pops something that's under other UI elements, making it impossible to read or interact with.

Pandora

A particular station keeps playing electronic music, even though I hit thumbs down every time an electronic song comes on. The seed song was a song from a Disney musical.

Zulip

An old issue is that you can't disable notifications from @all mentions. Since literally none of them have been relevant to me for as long as I can remember, and @all notifications outnumber other notifications, it means that the majority of notifications I get are spam.

The new thing is that I tried muting the streams that regularly spam me, but the notification blows through the mute. My fix for that is that I've disabled all notifications, but now I don't get a notification if someone DMs me or uses @danluu.

Chrome

The Rust guide is unreadable with my version of chrome (no plug-ins).

Unreadable quoted blocks

Google Docs

I tried co-writing a doc with Rose Ames. Worked fine for me, but everything displayed as gibberish for her, so we switched to hackpad.

I didn't notice this until after I tried hackpad, but Docs is really slow. Hackpad feels amazingly responsive, but it's really just that Docs is laggy. It's the same feeling I had after I tried fastmail. Gmail doesn't seem slow until you use something that isn't slow.

Hackpad

Hours after the doc was created, it says “ROSE AMES CREATED THIS 1 MINUTE AGO.”

The right hand side list, which shows who's in the room, has a stack of N people even though there are only 2 people.

Rust

After all that, Rose and I worked through the Rust guide. I won't list the issues here because they're so long that our hackpad doc that's full of bugs is at least twice as long as this blog post. And this isn't a knock against the Rust docs, the docs are actually much better than for almost any other language.

WAT

I'm in a super good mood. Everything is still broken, but now it's funny instead of making me mad.

— Gary Bernhardt (@garybernhardt) January 28, 2013

What's going on here? If you include the bugs I'm not listing because the software is so buggy that listing all of the bugs would triple the length of this post, that's about 80 bugs in one week. And that's only counting bugs I hadn't seen before. How come there are so many bugs in everything?

A common response to this sort of comment is that it's open source, you ungrateful sod, why don't you fix the bugs yourself? I do fix some bugs, but there literally aren't enough hours in a week for me to debug and fix every bug I run into. There's a tragedy of the commons effect here. If there are only a few bugs, developers are likely to fix the bugs they run across. But if there are so many bugs that making a dent is hopeless, a lot of people won't bother.

I'm going to take a look at Julia because I'm already familiar with it, but I expect that it's no better or worse tested than most of these other projects (except for Chrome, which is relatively well tested). As a rough proxy for how much test effort has gone into it, it has 18k lines of test code. But that's compared to about 108k lines of code in src plus Base.

At every place I've worked, a 2k LOC prototype that exists just so you can get preliminary performance numbers and maybe play with the API is expected to have at least that much in tests because otherwise how do you know that it's not so broken that your performance estimates aren't off by an order of magnitude? Since complexity doesn't scale linearly in LOC, folks expect a lot more test code as the prototype gets bigger.

At 18k LOC in tests for 108k LOC of code, users are going to find bugs. A lot of bugs.

Here's where I'm supposed to write an appeal to take testing more seriously and put real effort into it. But we all know that's not going to work. It would take 90k LOC of tests to get Julia to be as well tested as a poorly tested prototype (falsely assuming linear complexity in size). That's two person-years of work, not even including time to debug and fix bugs (which probably brings it closer to four of five years). Who's going to do that? No one. Writing tests is like writing documentation. Everyone already knows you should do it. Telling people they should do it adds zero information1.

Given that people aren't going to put any effort into testing, what's the best way to do it?

Property-based testing. Generative testing. Random testing. Concolic Testing (which was done long before the term was coined). Static analysis. Fuzzing. Statistical bug finding. There are lots of options. Some of them are actually the same thing because the terminology we use is inconsistent and buggy. I'm going to arbitrarily pick one to talk about, but they're all worth looking into.

People are often intimidated by these, though. I've seen a lot of talks on these and they often make it sound like this stuff is really hard. Csmith is 40k LOC. American Fuzzy Lop's compile-time instrumentation is smart enough to generate valid JPEGs. Sixth Sense has the same kind of intelligence as American Fuzzy Lop in terms of exploration, and in addition, uses symbolic execution to exhaustively explore large portions of the state space; it will formally verify that your asserts hold if it's able to collapse the state space enough to exhaustively search it, otherwise it merely tries to get the best possible test coverage by covering different paths and states. In addition, it will use symbolic equivalence checking to check different versions of your code against each other.

That's all really impressive, but you don't need a formal methods PhD to do this stuff. You can write a fuzzer that will shake out a lot of bugs in an hour2. Seriously. I'm a bit embarrassed to link to this, but this fuzzer was written in about an hour and found 20-30 bugs3, including incorrect code generation, and crashes on basic operations like multiplication and exponentiation. My guess is that it would take another 2-3 hours to shake out another 20-30 bugs (with support for more types), and maybe another day of work to get another 20-30 (with very basic support for random expressions). I don't mention this because it's good. It's not. It's totally heinous. But that's the point. You can throw together an absurd hack in an hour and it will turn out to be pretty useful.

Compared to writing unit tests by hand: even if I knew what the bugs were in advance, I'd be hard pressed to code fast enough to generate 30 bugs in an hour. 30 bugs in a day? Sure, but not if I don't already know what the bugs are in advance. This isn't to say that unit testing isn't valuable, but if you're going to spend a few hours writing tests, a few hours writing a fuzzer is going to go a longer way than a few hours writing unit tests. You might be able to hit 100 words a minute by typing, but your CPU can easily execute 200 billion instructions a minute. It's no contest.

What does it really take to write a fuzzer? Well, you need to generate random inputs for a program. In this case, we're generating random function calls in some namespace. Simple. The only reason it took an hour was because I don't really get Julia's reflection capabilities well enough to easily generate random types, which resulted in my writing the type generation stuff by hand.

This applies to a lot of different types of programs. Have a GUI? It's pretty easy to prod random UI elements. Read files or things off the network? Generating (or mutating) random data is straightforward. This is something anyone can do.

But this isn't a silver bullet. Lackadaisical testing means that your users will find bugs. However, even given that developers aren't going to spend nearly enough time on testing, we can do a lot better than we're doing right now.

Resources

There are a lot of great resources out there, but if you're just getting started, I found this description of types of fuzzers to be one of those most helpful (and simplest) things I've read.

John Regehr has a udacity course on software testing. I haven't worked through it yet (Pablo Torres just pointed to it), but given the quality of Dr. Regehr's writing, I expect the course to be good.

For more on my perspective on testing, there's this.

Acknowledgments

Thanks to Leah Hanson and Mindy Preston for catching writing bugs, to Steve Klabnik for explaining the cause/fix of the Chrome bug (bad/corrupt web fonts), and to Phillip Joseph for finding a markdown bug.

I'm experimenting with blogging more by spending less time per post and just spewing stuff out in 30-90 minute sitting. Please let me know if something is unclear or just plain wrong. Seriously.


  1. If I were really trying to convince you of this, I'd devote a post to the business case, diving into the data and trying to figure out the cost of bugs. The short version of that unwritten post is that response times are well studied and it's known that a 100ms of extra latency will cost you a noticeable amount of revenue. A 1s latency hit is a disaster. How do you think that compares to having your product not work at all?

    Compared to 100ms of latency, how bad is it when your page loads and then bugs out in a way that makes it totally unusable? What if it destroys user state and makes the user re-enter everything they wanted to buy into their cart? Removing one extra click is worth a huge amount of revenue, and now we're talking about adding 10 extra clicks or infinite latency to a random subset of users. And not a small subset, either. Want to stop lighting piles of money on fire? Write tests. If that's too much work, at least use the data you already have to find bugs.

    Of course it's sometimes worth it to light pile of money on fire. Maybe your rocket ship is powered by flaming piles of money. If you're a very rapidly growing startup, a 20% increase in revenue might not be worth that much. It could be better to focus on adding features that drive growth. The point isn't that you should definitely write more tests, it's that you should definitely do the math to see if you should write more tests.

    [return]
  2. Plus debugging time. [return]
  3. I really need to update the readme with more bugs. [return]
show more
TF-IDF linux commits
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-11-24 00:00:00 | Created: 2026-07-23 05:18:40

I was curious what different people worked on in Linux, so I tried grabbing data from the current git repository to see if I could pull that out of commit message data. This doesn't include history from before they switched to git, so it only goes back to 2005, but that's still a decent chunk of history.

Here's a list of the most commonly used words (in commit messages), by the top four most frequent committers, with users ordered by number of commits.

User 1 2 3 4 5
viro to in of and the
tiwai alsa the - for to
broonie the to asoc for a
davem the to in and sparc64


Alright, so their most frequently used words are to, alsa, the, and the. Turns out, Takashi Iwai (tiwai) often works on audio (alsa), and by going down the list we can see that David Miller's (davem) fifth most frequently used term is sparc64, which is a pretty good indicator that he does a lot of sparc work. But the table is mostly noise. Of course people use to, in, and other common words all the time! Putting that into a table provides zero information.

There are a number of standard techniques for dealing with this. One is to explicitly filter out "stop words", common words that we don't care about. Unfortunately, that doesn't work well with this dataset without manual intervention. Standard stop-word lists are going to miss things like Signed-off-by and cc, which are pretty uninteresting. We can generate a custom list of stop words using some threshold for common words in commit messages, but any threshold high enough to catch all of the noise is also going to catch commonly used but interesting terms like null and driver.

Luckily, it only takes about a minute to do by hand. After doing that, the result is that many of the top words are the same for different committers. I won't reproduce the table of top words by committer because it's just many of the same words repeated many times. Instead, here's the table of the top words (ranked by number of commit messages that use the word, not raw count), with stop words removed, which has the same data without the extra noise of being broken up by committer.

Word Count
driver 49442
support 43540
function 43116
device 32915
arm 28548
error 28297
kernel 23132
struct 18667
warning 17053
memory 16753
update 16088
bit 15793
usb 14906
bug 14873
register 14547
avoid 14302
pointer 13440
problem 13201
x86 12717
address 12095
null 11555
cpu 11545
core 11038
user 11038
media 10857
build 10830
missing 10508
path 10334
hardware 10316


Ok, so there's been a lot of work on arm, lots of stuff related to memory, null, pointer, etc. But if want to see what individuals work on, we'll need something else.

That something else could be penalizing more common words without eliminating them entirely. A standard metric to normalize by is the inverse document frequency (IDF), log(# of messages / # of messages with word). So instead of ordering by term count or term frequency, let's try ordering by (term frequency) * log(# of messages / # of messages with word), which is commonly called TF-IDF1. This gives us words that one person used that aren't commonly used by other people.

Here's a list of the top 40 linux committers and their most commonly used words, according to TF-IDF.

User 1 2 3 4 5
viro switch annotations patch of endianness
tiwai alsa hda codec codecs hda-codec
broonie asoc regmap mfd regulator wm8994
davem sparc64 sparc we kill fix
gregkh cc staging usb remove hank
mchehab v4l/dvb media at were em28xx
tglx x86 genirq irq prepare shared
hsweeten comedi staging tidy remove subdevice
mingo x86 sched zijlstra melo peter
joe unnecessary checkpatch convert pr_ use
tj cgroup doesnt which it workqueue
lethal sh up off sh64 kill
axel.lin regulator asoc convert thus use
hch xfs sgi-pv sgi-modid remove we
sachin.kamat redundant remove simpler null of_match_ptr
bzolnier ide shtylyov sergei acked-by caused
alan tty gma500 we up et131x
ralf mips fix build ip27 of
johannes.berg mac80211 iwlwifi it cfg80211 iwlagn
trond.myklebust nfs nfsv4 sunrpc nfsv41 ensure
shemminger sky2 net_device_ops skge convert bridge
bunk static needlessly global patch make
hartleys comedi staging remove subdevice driver
jg1.han simpler device_release unnecessary clears thus
akpm cc warning fix function patch
rmk+kernel arm acked-by rather tested-by we
daniel.vetter drm/i915 reviewed-by v2 wilson vetter
bskeggs drm/nouveau drm/nv50 drm/nvd0/disp on chipsets
acme galbraith perf weisbecker eranian stephane
khali hwmon i2c driver drivers so
torvalds linux commit just revert cc
chris drm/i915 we gpu bugzilla whilst
neilb md array so that we
lars asoc driver iio dapm of
kaber netfilter conntrack net_sched nf_conntrack fix
dhowells keys rather key that uapi
heiko.carstens s390 since call of fix
ebiederm namespace userns hallyn serge sysctl
hverkuil v4l/dvb ivtv media v4l2 convert


That's more like it. Some common words still appear -- this would really be improved with manual stop words to remove things like cc and of. But for the most part, we can see who works on what. Takashi Iwai (tiwai) spends a lot of time in hda land and workig on codecs, David S. Miller (davem) has spent a lot of time on sparc64, Ralf Baechle (ralf) does a lot of work with mips, etc. And then again, maybe it's interesting that some, but not all, people cc other folks so much that it shows up in their top 5 list even after getting penalized by IDF.

We can also use this to see the distribution of what people talk about in their commit messages vs. how often they commit.

This graph has people on the x-axis and relative word usage (ranked by TF-IDF) y-axis. On the x-axis, the most frequent committers on the left and least frequent on the right. On the y-axis, points are higher up if that committer used the word null more frequently, and lower if the person used the word null less frequently.

Relatively, almost no one works on POSIX compliance. You can actually count the individual people who mentioned POSIX in commit messages.

This is the point of the blog post where you might expect some kind of summary, or at least a vague point. Sorry. No such luck. I just did this because TF-IDF is one of a zillion concepts presented in the Mining Massive Data Sets course running now, and I knew it wouldn't really stick unless I wrote some code.

If you really must have a conclusion, TF-IDF is sometimes useful and incredibly easy to apply. You should use it when you should use it (when you want to see what words distinguish different documents/people from each other) and you shouldn't use it when you shouldn't use it (when you want to see what's common to documents/people). The end.

I'm experimenting with blogging more by spending less time per post and just spewing stuff out in 30-90 minute sitting. Please let me know if something is unclear or just plain wrong. Seriously. I went way over time on this one, but that's mostly because argh data and tables and bugs in Julia, not because of proofreading. I'm sure there are bugs!

Thanks to Leah Hanson for finding a bunch of writing bugs in this post and to Zack Maril for a conversation on how to maybe display change over time in the future.


  1. I actually don't understand why it's standard to take the log here. Sometimes you want to take the log so you can work with smaller numbers, or so that you can convert a bunch of multiplies into a bunch of adds, but neither of those is true here. Please let me know if this is obvious to you. [return]
show more
Markets, discrimination, and "lowering the bar"
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-12-01 00:00:00 | Created: 2026-07-23 05:18:40

Public discussions of discrimination in tech often result in someone claiming that discrimination is impossible because of market forces. Here's a quote from Marc Andreessen that sums up a common view1.

Let's launch right into it. I think the critique that Silicon Valley companies are deliberately, systematically discriminatory is incorrect, and there are two reasons to believe that that's the case. ... No. 2, our companies are desperate for talent. Desperate. Our companies are dying for talent. They're like lying on the beach gasping because they can't get enough talented people in for these jobs. The motivation to go find talent wherever it is unbelievably high.

Marc Andreessen's point is that the market is too competitive for discrimination to exist. But VC funded startups aren't the first companies in the world to face a competitive hiring market. Consider the market for PhD economists from, say, 1958 to 1987. Alan Greenspan had this to say about how that market looked to his firm, Townsend-Greenspan.

Townsend-Greenspan was unusual for an economics firm in that the men worked for the women (we had about twenty-five employees in all). My hiring of women economists was not motivated by women's liberation. It just made great business sense. I valued men and women equally, and found that because other employers did not, good women economists were less expensive than men. Hiring women . . . gave Townsend-Greenspan higher-quality work for the same money . . .

Not only did competition not end discrimination, there was enough discrimination that the act of not discriminating provided a significant competitive advantage for Townsend-Greenspan. And this is in finance, which is known for being cutthroat. And not just any part of finance, but one where it's PhD economists hiring other PhD economists. This is one of the industries where the people doing the hiring are the most likely to be familiar with both the theoretical models and the empirical research showing that discrimination opens up market opportunities by suppressing wages of some groups. But even that wasn't enough to equalize wages between men and women when Greenspan took over Townsend-Greenspan in 1958 and it still wasn't enough when Greenspan left to become chairman of the Fed in 1987. That's the thing about discrimination. When it's part of a deep-seated belief, it's hard for people to tell that they're discriminating.

And yet, in discussions on tech hiring, people often claim that, since markets and hiring are perfectly competitive or efficient, companies must already be hiring the best people presented to them. A corollary of this is that anti-discrimination or diversity oriented policies necessarily mean "lowering the bar since these would mean diverging from existing optimal hiring practices. And conversely, even when "market forces" aren't involved in the discussion, claiming that increasing hiring diversity necessarily means "lowering the bar" relies on an assumption of a kind of optimality in hiring. I think that an examination of tech hiring practices makes it pretty clear that practices are far from optimal, but rather than address this claim based on practices (which has been done in the linked posts), I'd look to look at the meta-claim that market forces make discrimination impossible. People make vauge claims about market efficiency and economics, like this influential serial founder who concludes his remarks on hiring with "Capitalism is real and markets are efficient."2. People seem to love handwave-y citations of "the market" or "economists".

But if we actually read what economists have to say on how hiring markets work, they do not, in general, claim that markets are perfectly efficient or that discrimination does not occur in markets that might colloquially be called highly competitive. Since we're talking about discrimination, a good place to start might be Becker's seminal work on discrimination. What Becker says is that markets impose a cost on discrimination, and that under certain market conditions, what Becker calls "taste-based"3 discrimination occuring on average doesn't mean there's discrimination at the margin. This is quite a specific statement and, if you read other papers in the literature on discrimination, they also make similarly specific statements. What you don't see is anything like the handwave-y claims in tech discussions, that "market forces" or "competition" is incompatible with discrimination or non-optimal hiring. Quite frankly, I've never had a discussion with someone who says things like "Capitalism is real and markets are efficient" where it appears that they have even a passing familiarity with Becker's seminal work in the field of the economics of discrimination or, for that matter, any other major work on the topic.

In discussions among the broader tech community, I have never seen anyone make a case that the tech industry (or any industry) meets the conditions under which taste-based discrimination on average doesn't imply marginal taste-based discrimination. Nor have I ever seen people make the case that we only have taste-based discrimination or that we also meet the conditions for not having other forms of discrimination. When people cite "efficient markets" with respect to hiring or other parts of tech, it's generally vague handwaving that sounds like an appeal to authority, but the authority is what someone might call a teenage libertarian's idea of how markets might behave.

Since people often don't find abstract reasoning of the kind you see in Becker's work convincing, let's look at a few concrete examples. You can see discrimination in a lot of fields. A problem is that it's hard to separate out the effect of discrimination from confounding variables because it's hard to get good data on employee performance v. compensation over time. Luckily, there's one set of fields where that data is available: sports. And before we go into the examples, it's worth noting that we should, directionally, expect much less discrimination in sports than in tech. Not only is there much better data available on employee performance, it's easier to predict future employee performance from past performance, the impact of employee performance on "company" performance is greater and easier quantify, and the market is more competitive. Relatively to tech, these forces both increase the cost of discrimination while making the cost more visible.

In baseball, Gwartney and Haworth (1974) found that teams that discriminated less against non-white players in the decade following de-segregation performed better. Studies of later decades using “classical” productivity metrics mostly found that salaries equalize. However, Swartz (2014), using newer and more accurate metrics for productivity, found that Latino players are significantly underpaid for their productivity level. Compensation isn't the only way to discriminate -- Jibou (1988) found that black players had higher exit rates from baseball after controlling for age and performance. This should sound familiar to anyone who's wondered about exit rates in tech fields.

This slow effect of the market isn't limited to baseball; it actually seems to be worse in other sports. A review article by Kahn (1991) notes that in basketball, the most recent studies (up to the date of the review) found an 11%-25% salary penalty for black players as well as a higher exit rate. Kahn also noted multiple studies showing discrimination against French-Canadians in hockey, which is believed to be due to stereotypes about how French-Canadian men are less masculine than other men4.

In tech, some people are concerned that increasing diversity will "lower the bar", but in sports, which has a more competitive hiring market than tech, we saw the opposite, increasing diversity raised the level instead of lowering it because it means hiring people on their qualifications instead of on what they look like. I don't disagree with people who say that it would be absurd for tech companies to leave money on the table by not hiring qualified minorities. But this is exactly what we saw in the sports we looked at, where that's even more absurd due to the relative ease of quantifying performance. And yet, for decades, teams left huge amounts of money on the table by favoring white players (and, in the case of hockey, non-French Canadian players) who were, quite simply, less qualified than their peers. The world is an absurd place.

In fields where there's enough data to see if there might be discrimination, we often find discrimination. Even in fields that are among the most competitive fields in existence, like major professional sports. Studies on discrimination aren't limited to empirical studies and data mining. There have been experiments showing discrimination at every level, from initial resume screening to phone screening to job offers to salary negotiation to workplace promotions. And those studies are mostly in fields where there's something resembling gender parity. In fields where discrimination is weak enough that there's gender parity or racial parity in entrance rates, we can see steadily decreasing levels of discrimination over the last two generations. Discrimination hasn't been eliminated, but it's much reduced.

Graph of enrollment by gender in med school, law school, the sciences, and CS. Graph courtesy of NPR.

And then we have computer science. The disparity in entrance rates is about what it was for medicine, law, and the physical sciences in the 70s. As it happens, the excuses for the gender disparity are the exact same excuses that were trotted out in the 70s to justify why women didn't want to go into or couldn't handle technical fields like medicine, economics, finance, and biology.

One argument that's commonly made is that women are inherently less interested in the "harder" sciences, so you'd expect more women to go into biology or medicine than programming. There are two major reasons I don't find that line of reasoning to be convincing. First, proportionally more women go into fields like math and chemical engineering than go into programming. I think it's pointless to rank math and the sciences by how "hard science" they are, but if you ask people to rank these things, most people will put math above programming and if they know what's involved in a chemical engineering degree, I think they'll also put chemical engineering above programming and yet those fields have proportionally more women than programming. Second, if you look at other countries, they have wildly different proportions of people who study computer science for reasons that seem to mostly be cultural. Given that we do see all of this variation, I don't see any reason to think that the U.S. reflects the "true" rate that women want to study programming and that countries where (proportionally) many more women want to study programming have rates that are distorted from the "true" rate by cultural biases.

Putting aside theoretical arguments, I wonder how it is that I've had such a different lived experience than Andreessen. His reasoning must sound reasonable in his head and stories of discrimination from women and minorities must not ring true. But to me, it's just the opposite.

Just the other day, I was talking to John (this and all other names were chosen randomly in order to maintain anonymity), a friend of mine who's a solid programmer. It took him two years to find a job, which is shocking in today's job market for someone my age, but sadly normal for someone like him, who's twice my age.

You might wonder if it's something about John besides his age, but when a Google coworker and I mock interviewed him he did fine. I did the standard interview training at Google and I interviewed for Google, and when I compare him to that bar, I'd say that his getting hired at Google would pretty much be a coin flip. Yes on a good day; no on a bad day. And when he interviewed at Google, he didn't get an offer, but he passed the phone screen and after the on-site they strongly suggested that he apply again in a year, which is a good sign. But most places wouldn't even talk to John.

And even at Google, which makes a lot of hay about removing bias from their processes, the processes often fail to do so. When I referred Mary to Google, she got rejected in the recruiter phone screen as not being technical enough and I saw William face increasing levels of ire from a manager because of a medical problem, which eventually caused him to quit.

Of course, in online discussions, people will call into question the technical competency of people like Mary. Well, Mary is one of the most impressive engineers I've ever met in any field. People mean different things when they say that, so let me provide a frame of reference: the other folks who fall into that category for me include an IBM Fellow, the person that IBM Fellow called the best engineer at IBM, a Math Olympiad medalist who's now a professor at CMU, a distinguished engineer at Sun, and a few other similar folks.

So anyway, Mary gets on the phone with a Google recruiter. The recruiter makes some comments about how Mary has a degree in math and not CS, and might not be technical enough, and questions Mary's programming experience: was it “algorithms” or “just coding”? It goes downhill from there.

Google has plenty of engineers without a CS degree, people with degrees in history, music, and the arts, and lots of engineers without any degree at all, not even a high school diploma. But somehow a math degree plus my internal referral mentioning that this was one of the best engineers I've ever seen resulted in the decision that Mary wasn't technical enough.

You might say that, like the example with John, this is some kind of a fluke. Maybe. But from what I've seen, if Mary were a man and not a woman, the odds of a fluke would have been lower.

This dynamic isn't just limited to hiring. I notice it every time I read the comments on one of Anna's blog posts. As often as not, someone will question Anna's technical chops. It's not even that they find a "well, actually" in the current post (although that sometimes happens); it's usually that they dig up some post from six months ago which, according to them, wasn't technical enough.

I'm no more technical than Anna, but I have literally never had that happen to me. I've seen it happen to men, but only those who are extremely high profile (among the top N most well-known tech bloggers, like Steve Yegge or Jeff Atwood), or who are pushing an agenda that's often condescended to (like dynamic languages). But it regularly happens to moderately well-known female bloggers like Anna.

Differential treatment of women and minorities isn't limited to hiring and blogging. I've lost track of the number of times a woman has offhandedly mentioned to me that some guy assumed she was a recruiter, a front-end dev, a wife, a girlfriend, or a UX consultant. It happens everywhere. At conferences. At parties full of devs. At work. Everywhere. Not only has that never happened to me, the opposite regularly happens to me -- if I'm hanging out with physics or math grad students, people assume I'm a fellow grad student.

When people bring up the market in discussions like these, they make it sound like it's a force of nature. It's not. It's just a word that describes the collective actions of people under some circumstances. Mary's situation didn't automatically get fixed because it's a free market. Mary's rejection by the recruiter got undone when I complained to my engineering director, who put me in touch with an HR director who patiently listened to the story and overturned the decision5. The market is just humans. It's humans all the way down.

We can fix this, if we stop assuming the market will fix it for us.

Also, note that although this post was originally published in 2014, it was updated in 2020 with links to some more recent comments and a bit of re-organization.

Thanks to Leah Hanson, Kelley Eskridge, Lindsey Kuper, Nathan Kurz, Scott Feeney, Katerina Barone-Adesi, Yuri Vishnevsky, @teles_dev, "Negative12DollarBill", and Patrick Roberts for feedback on this post, and to Julia Evans for encouraging me to post this when I was on the fence about writing this up publicly.

Note that all names in this post are aliases, taken from a list common names in the U.S. as of 1880.


  1. If you're curious what his “No. 1” was, it was that there can't be discrimination because just look at all the diversity we have. Chinese. Indians. Vietnamese. And so on. The argument is that it's not possible that we're discriminating against some groups because we're not discriminating against other groups. In particular, it's not possible that we're discriminating against groups that don't fit the stereotypical engineer mold because we're not discriminating against groups that do fit the stereotypical engineer mold. [return]
  2. See also, this comment by Benedict Evans "refuting" a comment that SV companies may have sub-optimal hiring practices for employees by saying "I don’t have to tell you that there is a ferocious war for talent in the valley.". That particular comment isn't one about diversity or discrimination, but the general idea that the SV job market somehow enfores a kind of optimality is pervasive among SV thought leaders. [return]
  3. "taste-based" discrimination is discrimination based on preferences that are unrelated to any actual productivity differences between groups that might exist. Of course, it's common for people to claim that they've never seen racism or sexism in some context, often with the implication and sometimes with an explicit claim that any differences we see are due to population level differences. If that were the case, we'd want to look at the literature on "statistical" discrimination. However, statistical discrimination doesn't seem like it should be relevant to this discussion. A contrived example of a case where statistical discrimination would be relevant is if we had to hire basketball players solely off of their height and weight with no ability to observe their play, either directly or statistically.

    In that case, teams would want to exclusively hire tall basketball players, since, if all you have to go on is height, height is a better proxy for basketball productivity than nothing. However, if we consider the non-contrived example of actual basketball productivity and compare the actual productivity of NBA basketball players vs. their height, there is (with the exception of outliers who are very unusually short for basketball players), no correlation between height and performance. The reason is that, if we can measure performance directly, we can simply hire based on performance, which takes height out of the performance equation. The exception to this is for very short players, who have to overcome biases (taste-based discrimination) that cause people to overlook them.

    While measure of programming productivity are quite poor, the actual statistical correlation between race and gender and productivity among the entire population is zero as best as anyone can tell, making statistical discrimination irrelevant.

    [return]
  4. The evidence here isn't totally unequivocal. In the review, Kahn notes that for some areas, there are early studies finding no pay gap, but those were done with small samples of players. Also, Kahn notes that (at the time), there wasn't enough evidence in football to say much either way. [return]
  5. In the interest of full disclosure, this didn't change the end result, since Mary didn't want to have anything to do with Google after the first interview. Given that the first interview went how it did, making that Mary's decision and not Google's was probably the best likely result, though, and from the comments I heard from the HR director, it sounded like there might be a lower probability of the same thing happening again in the future. [return]
show more
Malloc tutorial
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-12-04 00:00:00 | Created: 2026-07-23 05:18:40

Let's write a malloc and see how it works with existing programs!

This is basically an expanded explanation of what I did after reading this tutorial by Marwan Burelle and then sitting down and trying to write my own implementation, so the steps are going to be fairly similar. The main implementation differences are that my version is simpler and more vulnerable to memory fragmentation. In terms of exposition, my style is a lot more casual.

This tutorial is going to assume that you know what pointers are, and that you know enough C to know that *ptr dereferences a pointer, ptr->foo means (*ptr).foo, that malloc is used to dynamically allocate space, and that you're familiar with the concept of a linked list. For a basic intro to C, Pointers on C is one of my favorite books. If you want to look at all of this code at once, it's available here.

Preliminaries aside, malloc's function signature is

void *malloc(size_t size);

It takes as input a number of bytes and returns a pointer to a block of memory of that size.

There are a number of ways we can implement this. We're going to arbitrarily choose to use sbrk. The OS reserves stack and heap space for processes and sbrk lets us manipulate the heap. sbrk(0) returns a pointer to the current top of the heap. sbrk(foo) increments the heap size by foo and returns a pointer to the previous top of the heap.

Diagram of linux memory layout, courtesy of Gustavo Duarte.

If we want to implement a really simple malloc, we can do something like

#include <assert.h>
#include <string.h>
#include <sys/types.h>
#include <unistd.h>


void *malloc(size_t size) {
  void *p = sbrk(0);
  void *request = sbrk(size);
  if (request == (void*) -1) {
    return NULL; // sbrk failed.
  } else {
    assert(p == request); // Not thread safe.
    return p;
  }
}

When a program asks malloc for space, malloc asks sbrk to increment the heap size and returns a pointer to the start of the new region on the heap. This is missing a technicality, that malloc(0) should either return NULL or another pointer that can be passed to free without causing havoc, but it basically works.

But speaking of free, how does free work? Free's prototype is

void free(void *ptr);

When free is passed a pointer that was previously returned from malloc, it's supposed to free the space. But given a pointer to something allocated by our malloc, we have no idea what size block is associated with it. Where do we store that? If we had a working malloc, we could malloc some space and store it there, but we're going to run into trouble if we need to call malloc to reserve space each time we call malloc to reserve space.

A common trick to work around this is to store meta-information about a memory region in some space that we squirrel away just below the pointer that we return. Say the top of the heap is currently at 0x1000 and we ask for 0x400 bytes. Our current malloc will request 0x400 bytes from sbrk and return a pointer to 0x1000. If we instead save, say, 0x10 bytes to store information about the block, our malloc would request 0x410 bytes from sbrk and return a pointer to 0x1010, hiding our 0x10 byte block of meta-information from the code that's calling malloc.

That lets us free a block, but then what? The heap region we get from the OS has to be contiguous, so we can't return a block of memory in the middle to the OS. Even if we were willing to copy everything above the newly freed region down to fill the hole, so we could return space at the end, there's no way to notify all of the code with pointers to the heap that those pointers need to be adjusted.

Instead, we can mark that the block has been freed without returning it to the OS, so that future calls to malloc can use re-use the block. But to do that we'll need be able to access the meta information for each block. There are a lot of possible solutions to that. We'll arbitrarily choose to use a single linked list for simplicity.

So, for each block, we'll want to have something like

struct block_meta {
  size_t size;
  struct block_meta *next;
  int free;
  int magic; // For debugging only. TODO: remove this in non-debug mode.
};

#define META_SIZE sizeof(struct block_meta)

We need to know the size of the block, whether or not it's free, and what the next block is. There's a magic number here for debugging purposes, but it's not really necessary; we'll set it to arbitrary values, which will let us easily see which code modified the struct last.

We'll also need a head for our linked list:

void *global_base = NULL;

For our malloc, we'll want to re-use free space if possible, allocating space when we can't re-use existing space. Given that we have this linked list structure, checking if we have a free block and returning it is straightforward. When we get a request of some size, we iterate through our linked list to see if there's a free block that's large enough.

struct block_meta *find_free_block(struct block_meta **last, size_t size) {
  struct block_meta *current = global_base;
  while (current && !(current->free && current->size >= size)) {
    *last = current;
    current = current->next;
  }
  return current;
}

If we don't find a free block, we'll have to request space from the OS using sbrk and add our new block to the end of the linked list.

struct block_meta *request_space(struct block_meta* last, size_t size) {
  struct block_meta *block;
  block = sbrk(0);
  void *request = sbrk(size + META_SIZE);
  assert((void*)block == request); // Not thread safe.
  if (request == (void*) -1) {
    return NULL; // sbrk failed.
  }

  if (last) { // NULL on first request.
    last->next = block;
  }
  block->size = size;
  block->next = NULL;
  block->free = 0;
  block->magic = 0x12345678;
  return block;
}

As with our original implementation, we request space using sbrk. But we add a bit of extra space to store our struct, and then set the fields of the struct appropriately.

Now that we have helper functions to check if we have existing free space and to request space, our malloc is simple. If our global base pointer is NULL, we need to request space and set the base pointer to our new block. If it's not NULL, we check to see if we can re-use any existing space. If we can, then we do; if we can't, then we request space and use the new space.

void *malloc(size_t size) {
  struct block_meta *block;
  // TODO: align size?

  if (size <= 0) {
    return NULL;
  }

  if (!global_base) { // First call.
    block = request_space(NULL, size);
    if (!block) {
      return NULL;
    }
    global_base = block;
  } else {
    struct block_meta *last = global_base;
    block = find_free_block(&last, size);
    if (!block) { // Failed to find free block.
      block = request_space(last, size);
      if (!block) {
        return NULL;
      }
    } else {      // Found free block
      // TODO: consider splitting block here.
      block->free = 0;
      block->magic = 0x77777777;
    }
  }

  return(block+1);
}

For anyone who isn't familiar with C, we return block+1 because we want to return a pointer to the region after block_meta. Since block is a pointer of type struct block_meta, +1 increments the address by one sizeof(struct block_meta).

If we just wanted a malloc without a free, we could have used our original, much simpler malloc. So let's write free! The main thing free needs to do is set ->free.

Because we'll need to get the address of our struct in multiple places in our code, let's define this function.

struct block_meta *get_block_ptr(void *ptr) {
  return (struct block_meta*)ptr - 1;
}

Now that we have that, here's free:

void free(void *ptr) {
  if (!ptr) {
    return;
  }

  // TODO: consider merging blocks once splitting blocks is implemented.
  struct block_meta* block_ptr = get_block_ptr(ptr);
  assert(block_ptr->free == 0);
  assert(block_ptr->magic == 0x77777777 || block_ptr->magic == 0x12345678);
  block_ptr->free = 1;
  block_ptr->magic = 0x55555555;
}

In addition to setting ->free, it's valid to call free with a NULL ptr, so we need to check for NULL. Since free shouldn't be called on arbitrary addresses or on blocks that are already freed, we can assert that those things never happen.

You never really need to assert anything, but it often makes debugging a lot easier. In fact, when I wrote this code, I had a bug that would have resulted in silent data corruption if these asserts weren't there. Instead, the code failed at the assert, which make it trivial to debug.

Now that we've got malloc and free, we can write programs using our custom memory allocator! But before we can drop our allocator into existing code, we'll need to implement a couple more common functions, realloc and calloc. Calloc is just malloc that initializes the memory to 0, so let's look at realloc first. Realloc is supposed to adjust the size of a block of memory that we've gotten from malloc, calloc, or realloc.

Realloc's function prototype is

void *realloc(void *ptr, size_t size)

If we pass realloc a NULL pointer, it's supposed to act just like malloc. If we pass it a previously malloced pointer, it should free up space if the size is smaller than the previous size, and allocate more space and copy the existing data over if the size is larger than the previous size.

Everything will still work if we don't resize when the size is decreased and we don't free anything, but we absolutely have to allocate more space if the size is increased, so let's start with that.

void *realloc(void *ptr, size_t size) {
  if (!ptr) {
    // NULL ptr. realloc should act like malloc.
    return malloc(size);
  }

  struct block_meta* block_ptr = get_block_ptr(ptr);
  if (block_ptr->size >= size) {
    // We have enough space. Could free some once we implement split.
    return ptr;
  }

  // Need to really realloc. Malloc new space and free old space.
  // Then copy old data to new space.
  void *new_ptr;
  new_ptr = malloc(size);
  if (!new_ptr) {
    return NULL; // TODO: set errno on failure.
  }
  memcpy(new_ptr, ptr, block_ptr->size);
  free(ptr);
  return new_ptr;
}

And now for calloc, which just clears the memory before returning a pointer.

void *calloc(size_t nelem, size_t elsize) {
  size_t size = nelem * elsize; // TODO: check for overflow.
  void *ptr = malloc(size);
  memset(ptr, 0, size);
  return ptr;
}

Note that this doesn't check for overflow in nelem * elsize, which is actually required by the spec. All of the code here is just enough to get something that kinda sorta works.

Now that we have something that kinda works, we can use our with existing programs (and we don't even need to recompile the programs)!

First, we need to compile our code. On linux, something like

clang -O0 -g -W -Wall -Wextra -shared -fPIC malloc.c -o malloc.so

should work.

-g adds debug symbols, so we can look at our code with gdb or lldb. -O0 will help with debugging, by preventing individual variables from getting optimized out. -W -Wall -Wextra adds extra warnings. -shared -fPIC will let us dynamically link our code, which is what lets us use our code with existing binaries!

On macs, we'll want something like

clang -O0 -g -W -Wall -Wextra -dynamiclib malloc.c -o malloc.dylib

Note that sbrk is deprecated on recent versions of OS X. Apple uses an unorthodox definition of deprecated -- some deprecated syscalls are badly broken. I didn't really test this on a Mac, so it's possible that this will cause weird failures or or just not work on a mac.

Now, to use get a binary to use our malloc on linux, we'll need to set the LD_PRELOAD environment variable. If you're using bash, you can do that with

export LD_PRELOAD=/absolute/path/here/malloc.so

If you've got a mac, you'll want

export DYLD_INSERT_LIBRARIES=/absolute/path/here/malloc.so

If everything works, you can run some arbitrary binary and it will run as normal (except that it will be a bit slower).

$ ls
Makefile  malloc.c  malloc.so  README.md  test  test-0  test-1  test-2  test-3  test-4

If there's a bug, you might get something like

$ ls
Segmentation fault (core dumped)

Debugging

Let's talk about debugging! If you're familiar with using a debugger to set breakpoints, inspect memory, and step through code, you can skip this section and go straight to the exercises.

This section assumes you can figure out how to install gdb on your system. If you're on a mac, you may want to just use lldb and translate the commands appropriately. Since I don't know what bugs you might run into, I'm going to introduce a couple of bugs and show how I'd track them down.

First, we need to figure out how to run gdb without having it segfault. If ls segfaults, and we try to run gdb ls, gdb is almost certainly going to segfault, too. We could write a wrapper to do this, but gdb also supports this. If we start gdb and then run set environment LD_PRELOAD=./malloc.so before running the program, LD_PRELOAD will work as normal.

$ gdb /bin/ls
(gdb) set environment LD_PRELOAD=./malloc.so
(gdb) run
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7bd7dbd in free (ptr=0x0) at malloc.c:113
113       assert(block_ptr->free == 0);

As expected, we get a segfault. We can look around with list to see the code near the segfault.

(gdb) list
108     }
109
110     void free(void *ptr) {
111       // TODO: consider merging blocks once splitting blocks is implemented.
112       struct block_meta* block_ptr = get_block_ptr(ptr);
113       assert(block_ptr->free == 0);
114       assert(block_ptr->magic == 0x77777777 || block_ptr->magic == 0x12345678);
115       block_ptr->free = 1;
116       block_ptr->magic = 0x55555555;
117     }

And then we can use p (for print) to see what's going on with the variables here:

(gdb) p ptr
$6 = (void *) 0x0
(gdb) p block_ptr
$7 = (struct block_meta *) 0xffffffffffffffe8

ptr is 0, i.e., NULL, which is the cause of the problem: we forgot to check for NULL.

Now that we've figured that out, let's try a slightly harder bug. Let's say that we decided to replace our struct with

struct block_meta {
  size_t size;
  struct block_meta *next;
  int free;
  int magic;    // For debugging only. TODO: remove this in non-debug mode.
  char data[1];
};

and then return block->data instead of block+1 from malloc, with no other changes. This seems pretty similar to what we're already doing -- we just define a member that points to the end of the struct, and return a pointer to that.

But here's what happens if we try to use our new malloc:

$ /bin/ls
Segmentation fault (core dumped)
gdb /bin/ls
(gdb) set environment LD_PRELOAD=./malloc.so
(gdb) run

Program received signal SIGSEGV, Segmentation fault.
_IO_vfprintf_internal (s=s@entry=0x7fffff7ff5f0, format=format@entry=0x7ffff7567370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", ap=ap@entry=0x7fffff7ff718) at vfprintf.c:1332
1332    vfprintf.c: No such file or directory.
1327    in vfprintf.c

This isn't as nice as our last error -- we can see that one of our asserts failed, but gdb drops us into some print function that's being called when the assert fails. But that print function uses our buggy malloc and blows up!

One thing we could do from here would be to inspect ap to see what assert was trying to print:

(gdb) p *ap
$4 = {gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7fffff7ff7f0, reg_save_area = 0x7fffff7ff730}

That would work fine; we could poke around until we figure out what's supposed to get printed and figure out the fail that way. Some other solutions would be to write our own custom assert or to use the right hooks to prevent assert from using our malloc.

But in this case, we know there are only a few asserts in our code. The one in malloc checking that we don't try to use this in a multithreaded program and the two in free checking that we're not freeing something we shouldn't. Let's look at free first, by setting a breakpoint.

$ gdb /bin/ls
(gdb) set environment LD_PRELOAD=./malloc.so
(gdb) break free
Breakpoint 1 at 0x400530
(gdb) run /bin/ls

Breakpoint 1, free (ptr=0x61c270) at malloc.c:112
112       if (!ptr) {

block_ptr isn't set yet, but if we use s a few times to step forward to after it's set, we can see what the value is:

(gdb) s
(gdb) s
(gdb) s
free (ptr=0x61c270) at malloc.c:118
118       assert(block_ptr->free == 0);
(gdb) p/x *block_ptr
$11 = {size = 0, next = 0x78, free = 0, magic = 0, data = ""}

I'm using p/x instead of p so we can see it in hex. The magic field is 0, which should be impossible for a valid struct that we're trying to free. Maybe get_block_ptr is returning a bad offset? We have ptr available to us, so we can just inspect different offsets. Since it's a void *, we'll have to cast it so that gdb knows how to interpret the results.

(gdb) p sizeof(struct block_meta)
$12 = 32
(gdb) p/x *(struct block_meta*)(ptr-32)
$13 = {size = 0x0, next = 0x78, free = 0x0, magic = 0x0, data = {0x0}}
(gdb) p/x *(struct block_meta*)(ptr-28)
$14 = {size = 0x7800000000, next = 0x0, free = 0x0, magic = 0x0, data = {0x78}}
(gdb) p/x *(struct block_meta*)(ptr-24)
$15 = {size = 0x78, next = 0x0, free = 0x0, magic = 0x12345678, data = {0x6e}}

If we back off a bit from the address we're using, we can see that the correct offset is 24 and not 32. What's happening here is that structs get padded, so that sizeof(struct block_meta) is 32, even though the last valid member is at 24. If we want to cut out that extra space, we need to fix get_block_ptr.

That's it for debugging!

Exercises

Personally, this sort of thing never sticks with me unless I work through some exercises, so I'll leave a couple exercises here for anyone who's interested.

  1. malloc is supposed to return a pointer “which is suitably aligned for any built-in type”. Does our malloc do that? If so, why? If not, fix the alignment. Note that “any built-in type” is basically up to 8 bytes for C because SSE/AVX types aren't built-in types.

  2. Our malloc is really wasteful if we try to re-use an existing block and we don't need all of the space. Implement a function that will split up blocks so that they use the minimum amount of space necessary

  3. After doing 2, if we call malloc and free lots of times with random sizes, we'll end up with a bunch of small blocks that can only be re-used when we ask for small amounts of space. Implement a mechanism to merge adjacent free blocks together so that any consecutive free blocks will get merged into a single block.

  4. Find bugs in the existing code! I haven't tested this much, so I'm sure there are bugs, even if this basically kinda sorta works.

Resources

As noted above, there's Marwan Burelle tutorial.

For more on how Linux deals with memory management, see this post by Gustavo Duarte.

For more on how real-world malloc implementations work, dlmalloc and tcmalloc are both great reading. I haven't read the code for jemalloc, and I've heard that it's a bit more more difficult to understand, but it's also the most widely used high-performance malloc implementation around.

For help debugging, Address Sanitizer is amazing. If you want to write a thread-safe version, Thread Sanitizer is also a great tool.

There's a Spanish translation of this post here thanks to Matias Garcia Isaia.

Acknowledgements

Thanks to Gustavo Duarte for letting me use one of his images to illustrate sbrk, and to Ian Whitlock, Danielle Sucher, Nathan Kurz, "tedu", @chozu@fedi.absturztau.be, and David Farrel for comments/corrections/discussion. Please let me know if you find other bugs in this post (whether they're in the writing or the code).

show more
Integer overflow checking cost
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-12-17 00:00:00 | Created: 2026-07-23 05:18:40

How much overhead should we expect from enabling integer overflow checks? Using a compiler flag or built-in intrinsics, we should be able to do the check with a conditional branch that branches based on the overflow flag that add and sub set. Code that looks like

add     %esi, %edi

should turn into something like

add     %esi, %edi
jo      <handle_overflow>

Assuming that branch is always correctly predicted (which should be the case for most code), the costs of the branch are the cost of executing that correctly predicted not-taken branch, the pollution the branch causes in the branch history table, and the cost of decoding the branch (on x86, jo and jno don't fuse with add or sub, which means that on the fast path, the branch will take up one of the 4 opcodes that can come from the decoded instruction cache per cycle). That's probably less than a 2x penalty per add or sub on front-end limited in the worst case (which might happen in a tightly optimized loop, but should be rare in general), plus some nebulous penalty from branch history pollution which is really difficult to measure in microbenchmarks. Overall, we can use 2x as a pessimistic guess for the total penalty.

2x sounds like a lot, but how much time do applications spend adding and subtracting? If we look at the most commonly used benchmark of “workstation” integer workloads, SPECint, the composition is maybe 40% load/store ops, 10% branches, and 50% other operations. Of the 50% “other” operations, maybe 30% of those are integer add/sub ops. If we guesstimate that load/store ops are 10x as expensive as add/sub ops, and other ops are as expensive as add/sub, a 2x penalty on add/sub should result in a (40*10+10+50 + 12) / (40*10+10+50) = 3% penalty. That the penalty for a branch is 2x, that add/sub ops are only 10x faster than load/store ops, and that add/sub ops aren't faster than other "other" ops are all pessimistic assumptions, so this estimate should be on the high end for most workloads.

John Regehr, who's done serious analysis on integer overflow checks estimates that the penalty should be about 5%, which is in the same ballpark as our napkin sketch estimate.

A spec license costs $800, so let's benchmark bzip2 (which is a component of SPECint) instead of paying $800 for SPECint. Compiling bzip2 with clang -O3 vs. clang -O3 -fsanitize=signed-integer-overflow,unsigned-integer-overflow (which prints out a warning on overflow) vs. -fsanitize-undefined-trap-on-error with undefined overflow checks (which causes a crash on an undefined overflow), we get the following results on compressing and decompressing 1GB of code and binaries that happened to be lying around on my machine.

options zip (s) unzip (s) zip (ratio) unzip (ratio)
normal 93 45 1.0 1.0
fsan 119 49 1.28 1.09
fsan ud 94 45 1.01 1.00

In the table, ratio is the relative ratio of the run times, not the compression ratio. The difference between fsan ud, unzip and normal, unzip isn't actually 0, but it rounds to 0 if we measure in whole seconds. If we enable good error messages, decompression doesn't slow down all that much (45s v. 49s), but compression is a lot slower (93s v. 119s). The penalty for integer overflow checking is 28% for compression and 9% decompression if we print out nice diagnostics, but almost nothing if we don't. How is that possible? Bzip2 normally has a couple of unsigned integer overflows. If I patch the code to remove those so that the diagnostic printing code path is never executed it still causes a large performance hit.

Let's check out the penalty when we just do some adds with something like

for (int i = 0; i < n; ++i) {
  sum += a[i];
}

On my machine (a 3.4 GHz Sandy Bridge), this turns out to be about 6x slower with -fsanitize=signed-integer-overflow,unsigned-integer-overflow. Looking at the disassembly, the normal version uses SSE adds, whereas the fsanitize version uses normal adds. Ok, 6x sounds plausible for unchecked SSE adds v. checked adds.

But if I try different permutations of the same loop that don't allow the the compiler to emit SSE instructions for the unchecked version, I still get a 4x-6x performance penalty for versions compiled with fsanitize. Since there are a lot of different optimizations in play, including loop unrolling, let's take a look at a simple function that does a single add to get a better idea of what's going on.

Here's the disassembly for a function that adds two ints, first compiled with -O3 and then compiled with -O3 -fsanitize=signed-integer-overflow,unsigned-integer-overflow.

0000000000400530 <single_add>:
  400530:       01 f7                   add    %esi,%edi
  400532:       89 f8                   mov    %edi,%eax
  400534:       c3                      retq

The compiler does a reasonable job on the -O3 version. Per the standard AMD64 calling convention, the arguments are passed in via the esi and edi registers, and passed out via the eax register. There's some overhead over an inlined add instruction because we have to move the result to eax and then return from the function call, but considering that it's a function call, it's a totally reasonable implementation.

000000000041df90 <single_add>:
  41df90:       53                      push   %rbx
  41df91:       89 fb                   mov    %edi,%ebx
  41df93:       01 f3                   add    %esi,%ebx
  41df95:       70 04                   jo     41df9b <single_add+0xb>
  41df97:       89 d8                   mov    %ebx,%eax
  41df99:       5b                      pop    %rbx
  41df9a:       c3                      retq
  41df9b:       89 f8                   mov    %edi,%eax
  41df9d:       89 f1                   mov    %esi,%ecx
  41df9f:       bf a0 89 62 00          mov    $0x6289a0,%edi
  41dfa4:       48 89 c6                mov    %rax,%rsi
  41dfa7:       48 89 ca                mov    %rcx,%rdx
  41dfaa:       e8 91 13 00 00          callq  41f340
<__ubsan_handle_add_overflow>
  41dfaf:       eb e6                   jmp    41df97 <single_add+0x7>

The compiler does not do a reasonable job on the -O3 -fsanitize=signed-integer-overflow,unsigned-integer-overflow version. Optimization wizard Nathan Kurz, had this to say about clang's output:

That's awful (although not atypical) compiler generated code. For some reason the compiler decided that it wanted to use %ebx as the destination of the add. Once it did this, it has to do the rest. The question would by why it didn't use a scratch register, why it felt it needed to do the move at all, and what can be done to prevent it from doing so in the future. As you probably know, %ebx is a 'callee save' register, meaning that it must have the same value when the function returns --- thus the push and pop. Had the compiler just done the add without the additional mov, leaving the input in %edi/%esi as it was passed (and as done in the non-checked version), this wouldn't be necessary. I'd guess that it's a residue of some earlier optimization pass, but somehow the ghost of %ebx remained.

However, adding -fsanitize-undefined-trap-on-error changes this to

0000000000400530 <single_add>:
  400530:       01 f7                   add    %esi,%edi
  400532:       70 03                   jo     400537 <single_add+0x7>
  400534:       89 f8                   mov    %edi,%eax
  400536:       c3                      retq
  400537:       0f 0b                   ud2

Although this is a tiny, contrived, example, we can see a variety of mis-optimizations in other code compiled with options that allow fsanitize to print out diagnostics.

While a better C compiler could do better, in theory, gcc 4.82 doesn't do better than clang 3.4 here. For one thing, gcc's -ftrapv only checks signed overflow. Worse yet, it doesn't work, and this bug on ftrapv has been open since 2008. Despite doing fewer checks and not doing them correctly, gcc's -ftrapv slows things down about as much as clang's -fsanitize=signed-integer-overflow,unsigned-integer-overflow on bzip2, and substantially more than -fsanitize=signed-integer-overflow.

Summing up, integer overflow checks ought to cost a few percent on typical integer-heavy workloads, and they do, as long as you don't want nice error messages. The current mechanism that produces nice error messages somehow causes optimizations to get screwed up in a lot of cases1.

Update

On clang 3.8.0 and after, and gcc 5 and after, register allocation seems to work as expected (although you may need to pass -fno-sanitize-recover. I haven't gone back and re-run my benchmarks across different versions of clang and gcc, but I'd like to do that when I get some time.

CPU internals series

Thanks to Nathan Kurz for comments on this topic, including, but not limited to, the quote that's attributed to him, and to Stan Schwertly, Nick Bergson-Shilcock, Scott Feeney, Marek Majkowski, Adrian and Juan Carlos Borras for typo corrections and suggestions for clarification. Also, huge thanks to Richard Smith, who pointed out the -fsanitize-undefined-trap-on-error option to me. This post was updated with results for that option after Richard's comment. Also, thanks to Filipe Cabecinhas for noticing that clang fixed this behavior in clang 3.8 (released approximately 1.5 years after this post).

John Regehr has some more comments here on why clang's implementation of integer overflow checking isn't fast (yet).


  1. People often call for hardware support for integer overflow checking above and beyond the existing overflow flag. That would add expense and complexity to every chip made to get, at most, a few percent extra performance in the best case, on optimized code. That might be worth it -- there are lots of features Intel adds that only speed up a subset of applications by a few percent.

    This is often described as a chicken and egg problem; people would use overflow checks if checks weren't so slow, and hardware support is necessary to make the checks fast. But there's already hardware support to get good-enough performance for the vast majority of applications. It's just not taken advantage of because people don't actually care about this problem.

    [return]
show more
A review of the Julia language
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2014-12-28 00:00:00 | Created: 2026-07-23 05:18:40

Here's a language that gives near-C performance that feels like Python or Ruby with optional type annotations (that you can feed to one of two static analysis tools) that has good support for macros plus decent-ish support for FP, plus a lot more. What's not to like? I'm mostly not going to talk about how great Julia is, though, because you can find plenty of blog posts that do that all over the internet.

The last time I used Julia (around Oct. 2014), I ran into two new (to me) bugs involving bogus exceptions when processing Unicode strings. To work around those, I used a try/catch, but of course that runs into a non-deterministic bug I've found with try/catch. I also hit a bug where a function returned a completely wrong result if you passed it an argument of the wrong type instead of throwing a "no method" error. I spent half an hour writing a throwaway script and ran into four bugs in the core language.

The second to last time I used Julia, I ran into too many bugs to list; the worst of them caused generating plots to take 30 seconds per plot, which caused me to switch to R/ggplot2 for plotting. First there was this bug with plotting dates didn't work. When I worked around that I ran into a regression that caused plotting to break large parts of the core language, so that data manipulation had to be done before plotting. That would have been fine if I knew exactly what I wanted, but for exploratory data analysis I want to plot some data, do something with the data, and then plot it again. Doing that required restarting the REPL for each new plot. That would have been fine, except that it takes 22 seconds to load Gadfly on my 1.7GHz Haswell (timed by using time on a file that loads Gadfly and does no work), plus another 10-ish seconds to load the other packages I was using, turning my plotting workflow into: restart REPL, wait 30 seconds, make a change, make a plot, look at a plot, repeat.

It's not unusual to run into bugs when using a young language, but Julia has more than its share of bugs for something at its level of maturity. If you look at the test process, that's basically inevitable.

As far as I can tell, FactCheck is the most commonly used thing resembling a modern test framework, and it's barely used. Until quite recently, it was unmaintained and broken, but even now the vast majority of tests are written using @test, which is basically an assert. It's theoretically possible to write good tests by having a file full of test code and asserts. But in practice, anyone who's doing that isn't serious about testing and isn't going to write good tests.

Not only are existing tests not very good, most things aren't tested at all. You might point out that the coverage stats for a lot of packages aren't so bad, but last time I looked, there was a bug in the coverage tool that caused it to only aggregate coverage statistics for functions with non-zero coverage. That is to say, code in untested functions doesn't count towards the coverage stats! That, plus the weak notion of test coverage that's used (line coverage1) make the coverage stats unhelpful for determining if packages are well tested.

The lack of testing doesn't just mean that you run into regression bugs. Features just disappear at random, too. When the REPL got rewritten a lot of existing shortcut keys and other features stopped working. As far as I can tell, that wasn't because anyone wanted it to work differently. It was because there's no way to re-write something that isn't tested without losing functionality.

Something that goes hand-in-hand with the level of testing on most Julia packages (and the language itself) is the lack of a good story for error handling. Although you can easily use Nullable (the Julia equivalent of Some/None) or error codes in Julia, the most common idiom is to use exceptions. And if you use things in Base, like arrays or /, you're stuck with exceptions. I'm not a fan, but that's fine -- plenty of reliable software uses exceptions for error handling.

The problem is that because the niche Julia occupies doesn't care2 about error handling, it's extremely difficult to write a robust Julia program. When you're writing smaller scripts, you often want to “fail-fast” to make debugging easier, but for some programs, you want the program to do something reasonable, keep running, and maybe log the error. It's hard to write a robust program, even for this weak definition of robust. There are problems at multiple levels. For the sake of space, I'll just list two.

If I'm writing something I'd like to be robust, I really want function documentation to include all exceptions the function might throw. Not only do the Julia docs not have that, it's common to call some function and get a random exception that has to do with an implementation detail and nothing to do with the API interface. Everything I've written that actually has to be reliable has been exception free, so maybe that's normal when people use exceptions? Seems pretty weird to me, though.

Another problem is that catching exceptions doesn't work (sometimes, at random). I ran into one bug where using exceptions caused code to be incorrectly optimized out. You might say that's not fair because it was caught using a fuzzer, and fuzzers are supposed to find bugs, but the fuzzer wasn't fuzzing exceptions or even expressions. The implementation of the fuzzer just happens to involve eval'ing function calls, in a loop, with a try/catch to handle exceptions. Turns out, if you do that, the function might not get called. This isn't a case of using a fuzzer to generate billions of tests, one of which failed. This was a case of trying one thing, one of which failed. That bug is now fixed, but there's still a nasty bug that causes exceptions to sometimes fail to be caught by catch, which is pretty bad news if you're putting something in a try/catch block because you don't want an exception to trickle up to the top level and kill your program.

When I grepped through Base to find instances of actually catching an exception and doing something based on the particular exception, I could only find a single one. Now, it's me scanning grep output in less, so I might have missed some instances, but it isn't common, and grepping through common packages finds a similar ratio of error handling code to other code. Julia folks don't care about error handling, so it's buggy and incomplete. I once asked about this and was told that it didn't matter that exceptions didn't work because you shouldn't use exceptions anyway -- you should use Erlang style error handling where you kill the entire process on an error and build transactionally robust systems that can survive having random processes killed. Putting aside the difficulty of that in a language that doesn't have Erlang's support for that kind of thing, you can easily spin up a million processes in Erlang. In Julia, if you load just one or two commonly used packages, firing up a single new instance of Julia can easily take half a minute or a minute. To spin up a million independent instances would at 30 seconds a piece would take approximately two years.

Since we're broadly on the topic of APIs, error conditions aren't the only place where the Base API leaves something to be desired. Conventions are inconsistent in many ways, from function naming to the order of arguments. Some methods on collections take the collection as the first argument and some don't (e.g., replace takes the string first and the regex second, whereas match takes the regex first and the string second).

More generally, Base APIs outside of the niche Julia targets often don't make sense. There are too many examples to list them all, but consider this one: the UDP interface throws an exception on a partial packet. This is really strange and also unhelpful. Multiple people stated that on this issue but the devs decided to throw the exception anyway. The Julia implementers have great intuition when it comes to linear algebra and other areas they're familiar with. But they're only human and their intuition isn't so great in areas they're not familiar with. The problem is that they go with their intuition anyway, even in the face of comments about how that might not be the best idea.

Another thing that's an issue for me is that I'm not in the audience the package manager was designed for. It's backed by git in a clever way that lets people do all sorts of things I never do. The result of all that is that it needs to do git status on each package when I run Pkg.status(), which makes it horribly slow; most other Pkg operations I care about are also slow for a similar reason.

That might be ok if it had the feature I most wanted, which is the ability to specify exact versions of packages and have multiple, conflicting, versions of packages installed3. Because of all the regressions in the core language libraries and in packages, I often need to use an old version of some package to make some function actually work, which can require old versions of its dependencies. There's no non-hacky way to do this.

Since I'm talking about issues where I care a lot more than the core devs, there's also benchmarking. The website shows off some impressive sounding speedup numbers over other languages. But they're all benchmarks that are pretty far from real workloads. Even if you have a strong background in workload characterization and systems architecture (computer architecture, not software architecture), it's difficult to generalize performance results on anything resembling real workload from microbenchmark numbers. From what I've heard, performance optimization of Julia is done from a larger set of similar benchmarks, which has problems for all of the same reasons. Julia is actually pretty fast, but this sort of ad hoc benchmarking basically guarantees that performance is being left on the table. Moreover, the benchmarks are written in a way that stacks the deck against other languages. People from other language communities often get rebuffed when they submit PRs to rewrite the benchmarks in their languages idiomatically. The Julia website claims that "all of the benchmarks are written to test the performance of specific algorithms, expressed in a reasonable idiom", and that making adjustments that are idiomatic for specific languages would be unfair. However, if you look at the Julia code, you'll notice that they're written in a way to avoid doing one of a number of things that would crater performance. If you follow the mailing list, you'll see that there are quite a few intuitive ways to write Julia code that has very bad performance. The Julia benchmarks avoid those pitfalls, but the code for other languages isn't written with anywhere near that care; in fact, it's just the opposite.

I've just listed a bunch of issues with Julia. I believe the canonical response for complaints about an open source project is, why don't you fix the bugs yourself, you entitled brat? Well, I tried that. For one thing, there are so many bugs that I often don't file bugs, let alone fix them, because it's too much of an interruption. But the bigger issue are the barriers to new contributors. I spent a few person-days fixing bugs (mostly debugging, not writing code) and that was almost enough to get me into the top 40 on GitHub's list of contributors. My point isn't that I contributed a lot. It's that I didn't, and that still put me right below the top 40.

There's lots of friction that keeps people from contributing to Julia. The build is often broken or has failing tests. When I polled Travis CI stats for languages on GitHub, Julia was basically tied for last in uptime. This isn't just a statistical curiosity: the first time I tried to fix something, the build was non-deterministically broken for the better part of a week because someone checked bad code directly into master without review. I spent maybe a week fixing a few things and then took a break. The next time I came back to fix something, tests were failing for a day because of another bad check-in and I gave up on the idea of fixing bugs. That tests fail so often is even worse than it sounds when you take into account the poor test coverage. And even when the build is "working", it uses recursive makefiles, and often fails with a message telling you that you need to run make clean and build again, which takes half an hour. When you do so, it often fails with a message telling you that you need to make clean all and build again, with takes an hour. And then there's some chance that will fail and you'll have to manually clean out deps and build again, which takes even longer. And that's the good case! The bad case is when the build fails non-deterministically. These are well-known problems that occur when using recursive make, described in Recursive Make Considered Harmful circa 1997.

And that's not even the biggest barrier to contributing to core Julia. The biggest barrier is that the vast majority of the core code is written with no markers of intent (comments, meaningful variable names, asserts, meaningful function names, explanations of short variable or function names, design docs, etc.). There's a tax on debugging and fixing bugs deep in core Julia because of all this. I happen to know one of the Julia core contributors (presently listed as the #2 contributor by GitHub's ranking), and when I asked him about some of the more obtuse functions I was digging around in, he couldn't figure it out either. His suggestion was to ask the mailing list, but for the really obscure code in the core codebase, there's perhaps one to three people who actually understand the code, and if they're too busy to respond, you're out of luck.

I don't mind spending my spare time working for free to fix other people's bugs. In fact, I do quite a bit of that and it turns out I often enjoy it. But I'm too old and crotchety to spend my leisure time deciphering code that even the core developers can't figure out because it's too obscure.

None of this is to say that Julia is bad, but the concerns of the core team are pretty different from my concerns. This is the point in a complain-y blog post where you're supposed to suggest an alternative or make a call to action, but I don't know that either makes sense here. The purely technical problems, like slow load times or the package manager, are being fixed or will be fixed, so there's not much to say there. As for process problems, like not writing tests, not writing internal documentation, and checking unreviewed and sometimes breaking changes directly into master, well, that's “easy”4 to fix by adding a code review process that forces people to write tests and documentation for code, but that's not free.

A small team of highly talented developers who can basically hold all of the code in their collective heads can make great progress while eschewing anything that isn't just straight coding at the cost of making it more difficult for other people to contribute. Is that worth it? It's hard to say. If you have to slow down Jeff, Keno, and the other super productive core contributors and all you get out of it is a couple of bums like me, that's probably not worth it. If you get a thousand people like me, that's probably worth it. The reality is in the ambiguous region in the middle, where it might or might not be worth it. The calculation is complicated by the fact that most of the benefit comes in the long run, whereas the costs are disproportionately paid in the short run. I once had an engineering professor who claimed that the answer to every engineering question is "it depends". What should Julia do? It depends.

2022 Update

This post originally mentioned how friendly the Julia community is, but I removed that since it didn't seem accurate in light of the responses. Many people were highly supportive, such as this Julia core developer:

However, a number of people had some pretty nasty responses and I don't think it's accurate to say that a community is friendly when the response is mostly positive, but with a significant fraction of nasty responses, since it doesn't really take a lot of nastiness to make a group seem unfriendly. Also, sentiment about this post has gotten more negative over time as communities tend to take their direction from the top and a couple of the Julia co-creators have consistently been quite negative about this post.

Now, onto the extent to which these issues have been fixed. The initial response from the co-founders was that the issues aren't really real and the post is badly mistaken. Over time, as some of the issues had some work done on them, the response changed to being that this post is out of date and the issues were all fixed, e.g., here's a response from one of the co-creators of Julia in 2016:

The main valid complaints in Dan's post were:

  1. Insufficient testing & coverage. Code coverage is now at 84% of base Julia, from somewhere around 50% at the time he wrote this post. While you can always have more tests (and that is happening), I certainly don't think that this is a major complaint at this point.

  2. Package issues. Julia now has package precompilation so package loading is pretty fast. The package manager itself was rewritten to use libgit2, which has made it much faster, especially on Windows where shelling out is painfully slow.

  3. Travis uptime. This is much better. There was a specific mystery issue going on when Dan wrote that post. That issue has been fixed. We also do Windows CI on AppVeyor these days.

  4. Documentation of Julia internals. Given the quite comprehensive developer docs that now exist, it's hard to consider this unaddressed: http://julia.readthedocs.org/en/latest/devdocs/julia/

So the legitimate issues raised in that blog post are fixed.

The top response to that is:

The main valid complaints [...] the legitimate issues raised [...]

This is a really passive-aggressive weaselly phrasing. I’d recommend reconsidering this type of tone in public discussion responses.

Instead of suggesting that the other complaints were invalid or illegitimate, you could just not mention them at all, or at least use nicer language in brushing them aside. E.g. “... the main actionable complaints...” or “the main technical complaints ...”

Putting aside issues of tone, I would say that the main issue from the post, the core team's attitude towards correctness, is both a legitimate issue and one that's unfixed, as we'll see when we look at how the specific issues mentioned as fixed are also unfixed.

On correctness, if the correctness issues were fixed, we wouldn't continue to see showstopping bugs in Julia, but I have a couple of friends who continued to use Julia for years until they got fed up with correctness issues and sent me quite a few bugs that they personally ran into that were serious well after the 2016 comment about correctness being fixed, such as getting an incorrect result when sampling from a distribution, sampling from an array produces incorrect results, the product function, i.e., multiplication, produces incorrect results, quantile produces incorrect results, mean produces incorrect results, incorrect array indexing, divide produces incorrect results, converting from float to int produces incorrect results, quantile produces incorrect results (again), mean produces incorrect results (again), etc.

There has been a continued flow of very serious bugs from Julia and numerous other people noting that they've run into serious bugs, such as here:

I remember all too un-fondly a time in which one of my Julia models was failing to train. I spent multiple months on-and-off trying to get it working, trying every trick I could think of.

Eventually – eventually! – I found the error: Julia/Flux/Zygote was returning incorrect gradients. After having spent so much energy wrestling with points 1 and 2 above, this was the point where I simply gave up. Two hours of development work later, I had the model successfully training… in PyTorch.

And here

I have been bit by incorrect gradient bugs in Zygote/ReverseDiff.jl. This cost me weeks of my life and has thoroughly shaken my confidence in the entire Julia AD landscape. [...] In all my years of working with PyTorch/TF/JAX I have not once encountered an incorrect gradient bug.

And here

Since I started working with Julia, I’ve had two bugs with Zygote which have slowed my work by several months. On a positive note, this has forced me to plunge into the code and learn a lot about the libraries I’m using. But I’m finding myself in a situation where this is becoming too much, and I need to spend a lot of time debugging code instead of doing climate research.

Despite this continued flow of bugs, public responses from the co-creators of Julia as well as a number of core community members generally claim, as they did for this post, that the issues will be fixed very soon (e.g., see the comments here by some core devs on a recent post, saying that all of the issues are being addressed and will be fixed soon, or this 2020 comment about how the there were serious correctness issues in 2016 but things are now good, etc.).

Instead of taking the correctness issues or other issues seriously, the developers make statements like the following comments from a co-creator of Julia, passed to me by a friend of mine as my friend ran into yet another showstopping bug:

takes that Julia doesn't take testing seriously... I don't get it. the amount of time and energy we spend on testing the bejeezus out of everything. I literally don't know any other open source project as thoroughly end-to-end tested.

The general claim is that, not only has Julia fixed its correctness issues, it's as good as it gets for correctness.

On the package issues, the claim was that package load times were fixed by 2016. But this continues to be a major complaint of the people I know who use Julia, e.g., Jamie Brandon switched away from using Julia in 2022 because it took two minutes for his CSV parsing pipeline to run, where most of the time was package loading. Another example is that, in 2020, on a benchmark where the Julia developers bragged that Julia is very fast at the curious workload of repeatedly loading the same CSV over and over again (in a loop, not by running a script repeatedly) compared to R, some people noted that this was unrealistic due to Julia's very long package load times, saying that it takes 2 seconds to open the CSV package and then 104 seconds to load a plotting library. In 2022, in response to comments that package loading is painfully slow, a Julia developer responds to each issue saying each one will be fixed; on package loading, they say

We're getting close to native code caching, and more: https://discourse.julialang.org/t/precompile-why/78770/8. As you'll also read, the difficulty is due to important tradeoffs Julia made with composability and aggressive specialization...but it's not fundamental and can be surmounted. Yes there's been some pain, but in the end hopefully we'll have something approximating the best of both worlds.

It's curious that these problems could exist in 2020 and 2022 after a co-creator of Julia claimed, in 2016, that the package load time problems were fixed. But this is the general pattern of Julia PR that we see. On any particular criticism, the criticism is one of: illegitimate, fixed soon or, when the criticism is more than a year old, already fixed. But we can see by looking at responses over time that the issues that are "already fixed" or "will be fixed soon" are, in fact, not fixed many years after claims that they were fixed. It's true that there is progress on the issues, but it wasn't really fair to say that package load time issues were fixed and "package loading is pretty fast" when it takes nearly two minutes to load a CSV and use a standard plotting library (an equivalent to ggplot2) to generate a plot in Julia. And likewise for correctness issues when there's still a steady stream of issues in core libraries, Julia itself, and libraries that are named as part of the magic that makes Julia great (e.g., autodiff is frequently named as a huge advantage of Julia when it comes to features, but then when it comes to bugs, those bugs don't count because they're not in Julia itself (that last comment, of course, has a comment from a Julia developer noting that all of the issues will be addressed soon).

There's a sleight of hand here where the reflexive response from a number of the co-creators as well as core developers of Julia is to brush off any particular issue with a comment that sounds plausible if read on HN or Twitter by someone who doesn't know people who've used Julia. This makes for good PR since, with an emerging language like Julia, most potential users won't have real connections who've used it seriously and the reflexive comments sound plausible if you don't look into them.

I use the word reflexive here because it seems that some co-creators of Julia respond to any criticism with a rebuttal, such as here, where a core developer responds to a post about showstopping bugs by saying that having bugs is actually good, and here, where in response to my noting that some people had commented that they were tired of misleading benchmarking practices by Julia developers, a co-creator of Julia drops in to say "I would like to let it be known for the record that I do not agree with your statements about Julia in this thread." But my statements in the thread were merely that there existed comments like https://news.ycombinator.com/item?id=24748582. It's quite nonsensical to state, for the record, a disagreement that those kinds of comments exist because they clearly do exist.

Another example of a reflexive response is this 2022 thread, where someone who tried Julia but stopped using it for serious work after running into one too many bugs that took weeks to debug suggests that the Julia ecosystem needs a rewrite because the attitude and culture in the community results in a large number of correctness issues. A core Julia developer "rebuts" the comment by saying that things are re-written all the time and gives examples of things that were re-written for performance reasons. Performance re-writes are, famously, a great way to introduce bugs, making the "rebuttal" actually a kind of anti-rebuttal. But, as is typical for many core Julia developers, the person saw that there was an issue (not enough re-writes) and reflexively responded with a denial, that there are enough re-writes.

These reflexive responses are pretty obviously bogus if you spend a bit of time reading them and looking at the historical context but this kind of "deny deny deny" response is generally highly effective PR and has been effective for Julia, so it's understandable that it's done. For example, on this 2020 comment that belies the 2016 comment about correctness being fixed that says that there were serious issues in 2016 but things are "now" good in 2020, someone responds "Thank you, this is very heartening." since it relieves them of their concern that there are still issues. Of course, you can see basically the same discussion on discussions in 2022, but people reading the discussion in 2022 generally won't go back to see that this same discussion happened in 2020, 2016, 2013, etc.

On the build uptime, the claim is that the issue causing uptime issues was fixed, but my comment there was on the attitude of brushing off the issue for an extended period of time with "works on my machine". As we can see from the examples above, the meta-issue of brushing off issues continued.

On the last issue that was claimed to legitimate, which was also claimed to be fixed, documentation, this is still a common complaint from the community, e.g., here in 2018, 2 years after it was claimed that documentation was fixed in 2016, here in 2019, here in 2022, etc. In a much lengthier complaint, one person notes

The biggest issue, and one they seem unwilling to really address, is that actually using the type system to do anything cool requires you to rely entirely on documentation which may or may not exist (or be up-to-date).

And another echoes this sentiment with

This is truly an important issue.

Of course, there's a response saying this will be fixed soon, as is generally the case. And yet, you can still find people complaining about the documentation.

If you go back and read discussions on Julia correctness issues, three more common defenses are that everything has bugs, bugs are quickly fixed, and testing is actually great because X is well tested. You can see examples of "everything has bugs" here in 2014 as well as here in 2022 (and in between as well, of course), as if all non-zero bug rates are the same, even though a number of developers have noted that they stopped using Julia for work and switched to other ecosystems because, while everything has bugs, all non-zero numbers are, of course, not the same. Bugs getting fixed quickly is sometimes not true (e.g., many of the bugs linked in this post have been open for quite a while and are still open) and is also a classic defense that's used to distract from the issue of practices that directly lead to the creation of an unusually large number of new bugs. As noted in a number of links, above, it can take weeks or months to debug correctness issues since many of the correctness issues are of the form "silently return incorrect results" and, as noted above, I ran into a bug where exceptions were non-deterministically incorrectly not caught. It may be true that, in some cases, these sorts of bugs are quickly fixed when found, but those issues still cost users a lot of time to track down. We saw an example of "testing is actually great because X is well tested" above. If you'd like a more recent example, here's one from 2022 where, in response to someone saying that ran into more correctness bugs in Julia than than in any other ecosystem they've used in their decades of programming, a core Julia dev responds by saying that a number of things are very well tested in Julia, such as libuv, as if testing some components well is a talisman that can be wielded against bugs in other components. This is obviously absurd, in that it's like saying that a building with an open door can't be insecure because it also has very sturdy walls, but it's a common defense used by core Julia developers. And, of course, there's also just straight-up FUD about writing about Julia. For example, in 2022, on Yuri Vishnevsky's post on Julia bugs, a co-creator of Julia said "Yuri's criticism was not that Julia has correctness bugs as a language, but that certain libraries when composed with common operations had bugs (many of which are now addressed).". This is, of course, completely untrue. In conversations with Yuri, he noted to me that he specifically included examples of core language and core library bugs because those happened so frequently, and it was frustrating that core Julia people pretended those didn't exist and that their FUD seemed to work since people would often respond as if their comments weren't untrue. As mentioned above, this kind of flat denial of simple matters of fact is highly effective, so it's understandable that people employ it but, personally, it's not to my taste.

To be clear, I don't inherently have a problem with software being buggy. As I've mentioned, I think move fast and break things can be a good value because it clearly states that velocity is more valued than correctness. Comments from the creators of Julia as well as core developers broadcast that Julia is not just highly reliable and correct, but actually world class ("the amount of time and energy we spend on testing the bejeezus out of everything. I literally don't know any other open source project as thoroughly end-to-end tested.", etc.). But, by revealed preference, we can see that Julia's values are "move fast and break things".

Appendix: blog posts on Julia

  • 2014: this post
  • 2016: Victor Zverovich
    • Julia brags about high performance in unrepresentative microbenchmarks but often has poor performance in practice
    • Complex codebase leading to many bugs
  • 2022: Volker Weissman
    • Poor documentation
    • Unclear / confusing error messages
    • Benchmarks claim good performance but benchmarks are of unrealistic workloads and performance is often poor in practice
  • 2022: Patrick Kidger comparison of Julia to JAX and PyTorch
    • Poor documentation
    • Correctness issues in widely relied on, important, libraries
    • Inscrutable error messages
    • Poor code quality, leading to bugs and other issues
  • 2022: Yuri Vishnevsky
    • Many very serious correctness bugs in both the language runtime and core libraries that are heavily relied on
    • Culture / attitude has persistently caused a large number of bugs, "Julia and its packages have the highest rate of serious correctness bugs of any programming system I’ve used, and I started programming with Visual Basic 6 in the mid-2000s"
      • Stream of serious bugs is in stark contrast to comments from core Julia developers and Julia co-creators saying that Julia is very solid and has great correctness properties

Thanks (or anti-thanks) to Leah Hanson for pestering me to write this for the past few months. It's not the kind of thing I'd normally write, but the concerns here got repeatedly brushed off when I brought them up in private. For example, when I brought up testing, I was told that Julia is better tested than most projects. While that's true in some technical sense (the median project on GitHub probably has zero tests, so any non-zero number of tests is above average), I didn't find that to be a meaningful rebuttal (as opposed to a reply that Julia is still expected to be mostly untested because it's in an alpha state). After getting a similar response on a wide array of topics I stopped using Julia. Normally that would be that, but Leah really wanted these concerns to stop getting ignored, so I wrote this up.

Also, thanks to Leah Hanson, Julia Evans, Joe Wilder, Eddie V, David Andrzejewski, @sasuke___420@mastodon.social, and Yuri Vishnevsky for comments/corrections/discussion.


  1. What I mean here is that you can have lots of bugs pop up despite having 100% line coverage. It's not that line coverage is bad, but that it's not sufficient, not even close. And because it's not sufficient, it's a pretty bad sign when you not only don't have 100% line coverage, you don't even have 100% function coverage. [return]
  2. I'm going to use the word care a few times, and when I do I mean something specific. When I say care, I mean that in the colloquial revealed preference sense of the word. There's another sense of the word, in which everyone cares about testing and error handling, the same way every politician cares about family values. But that kind of caring isn't linked to what I care about, which involves concrete actions. [return]
  3. It's technically possible to have multiple versions installed, but the process is a total hack. [return]
  4. By "easy", I mean extremely hard. Technical fixes can be easy, but process and cultural fixes are almost always hard. [return]
show more
What's new in CPUs since the 80s?
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-01-11 00:00:00 | Created: 2026-07-23 05:18:40

This is a response to the following question from David Albert:

My mental model of CPUs is stuck in the 1980s: basically boxes that do arithmetic, logic, bit twiddling and shifting, and loading and storing things in memory. I'm vaguely aware of various newer developments like vector instructions (SIMD) and the idea that newer CPUs have support for virtualization (though I have no idea what that means in practice).

What cool developments have I been missing? What can today's CPU do that last year's CPU couldn't? How about a CPU from two years ago, five years ago, or ten years ago? The things I'm most interested in are things that programmers have to manually take advantage of (or programming environments have to be redesigned to take advantage of) in order to use and as a result might not be using yet. I think this excludes things like Hyper-threading/SMT, but I'm not honestly sure. I'm also interested in things that CPUs can't do yet but will be able to do in the near future.

Everything below refers to x86 and linux, unless otherwise indicated. History has a tendency to repeat itself, and a lot of things that were new to x86 were old hat to supercomputing, mainframe, and workstation folks.

The Present

Miscellania

For one thing, chips have wider registers and can address more memory. In the 80s, you might have used an 8-bit CPU, but now you almost certainly have a 64-bit CPU in your machine. I'm not going to talk about this too much, since I assume you're familiar with programming a 64-bit machine. In addition to providing more address space, 64-bit mode provides more registers and more consistent floating point results (via the avoidance of pseudo-randomly getting 80-bit precision for 32 and 64 bit operations via x87 floating point). Other things that you're very likely to be using that were introduced to x86 since the early 80s include paging / virtual memory, pipelining, and floating point.

Esoterica

I'm also going to avoid discussing things that are now irrelevant (like A20M) and things that will only affect your life if you're writing drivers, BIOS code, doing security audits, or other unusually low-level stuff (like APIC/x2APIC, SMM, NX, or SGX).

Memory / Caches

Of the remaining topics, the one that's most likely to have a real effect on day-to-day programming is how memory works. My first computer was a 286. On that machine, a memory access might take a few cycles. A few years back, I used a Pentium 4 system where a memory access took more than 400 cycles. Processors have sped up a lot more than memory. The solution to the problem of having relatively slow memory has been to add caching, which provides fast access to frequently used data, and prefetching, which preloads data into caches if the access pattern is predictable.

A few cycles vs. 400+ cycles sounds really bad; that's well over 100x slower. But if I write a dumb loop that reads and operates on a large block of 64-bit (8-byte) values, the CPU is smart enough to prefetch the correct data before I need it, which lets me process at about 22 GB/s on my 3GHz processor. A calculation that can consume 8 bytes every cycle at 3GHz only works out to 24GB/s, so getting 22GB/s isn't so bad. We're losing something like 8% performance by having to go to main memory, not 100x.

As a first-order approximation, using predictable memory access patterns and operating on chunks of data that are smaller than your CPU cache will get you most of the benefit of modern caches. If you want to squeeze out as much performance as possible, this document is a good starting point. After digesting that 100 page PDF, you'll want to familiarize yourself with the microarchitecture and memory subsystem of the system you're optimizing for, and learn how to profile the performance of your application with something like likwid.

TLBs

There are lots of little caches on the chip for all sorts of things, not just main memory. You don't need to know about the decoded instruction cache and other funny little caches unless you're really going all out on micro-optimizations. The big exception is the TLBs, which are caches for virtual memory lookups (done via a 4-level page table structure on x86). Even if the page tables were in the l1-data cache, that would be 4 cycles per lookup, or 16 cycles to do an entire virtual address lookup each time around. That's totally unacceptable for something that's required for all user-mode memory accesses, so there are small, fast, caches for virtual address lookups.

Because the first level TLB cache has to be fast, it's severely limited in size (perhaps 64 entries on a modern chip). If you use 4k pages, that limits the amount of memory you can address without incurring a TLB miss. x86 also supports 2MB and 1GB pages; some applications will benefit a lot from using larger page sizes. It's something worth looking into if you've got a long-running application that uses a lot of memory.

Also, first-level caches are usually limited by the page size times the associativity of the cache. If the cache is smaller than that, the bits used to index into the cache are the same regardless if whether you're looking at the virtual address or the physical address, so you don't have to do a virtual to physical translation before indexing into the cache. If the cache is larger than that, you have to first do a TLB lookup to index into the cache (which will cost at least one extra cycle), or build a virtually indexed cache (which is possible, but adds complexity and coupling to software). You can see this limit in modern chips. Haswell has an 8-way associative cache and 4kB pages. Its l1 data cache is 8 * 4kB = 32kB.

Out of Order Execution / Serialization

For a couple decades now, x86 chips have been able to speculatively execute and re-order execution (to avoid blocking on a single stalled resource). This sometimes results in odd performance hiccups. But x86 is pretty strict in requiring that, for a single CPU, externally visible state, like registers and memory, must be updated as if everything were executed in order. The implementation of this involves making sure that, for any pair of instructions with a dependency, those instructions execute in the correct order with respect to each other.

That restriction that things look like they executed in order means that, for the most part, you can ignore the existence of OoO execution unless you're trying to eke out the best possible performance. The major exceptions are when you need to make sure something not only looks like it executed in order externally, but actually executed in order internally.

An example of when you might care would be if you're trying to measure the execution time of a sequence of instructions using rdtsc. rdtsc reads a hidden internal counter and puts the result into edx and eax, externally visible registers.

Say we do something like

foo
rdtsc
bar
mov %eax, [%ebx]
baz

where foo, bar, and baz don't touch eax, edx, or [%ebx]. The mov that follows the rdtsc will write the value of eax to some location in memory, and because eax is an externally visible register, the CPU will guarantee that the mov doesn't execute until after rdtsc has executed, so that everything looks like it happened in order.

However, since there isn't an explicit dependency between the rdtsc and either foo or bar, the rdtsc could execute before foo, between foo and bar, or after bar. It could even be the case that baz executes before the rdtsc, as long as baz doesn't affect the move instruction in any way. There are some circumstances where that would be fine, but it's not fine if the rdtsc is there to measure the execution time of foo.

To precisely order the rdtsc with respect to other instructions, we need to an instruction that serializes execution. Precise details on how exactly to do that are provided in this document by Intel.

Memory / Concurrency

In addition to the ordering restrictions above, which imply that loads and stores to the same location can't be reordered with respect to each other, x86 loads and stores have some other restrictions. In particular, for a single CPU, stores are never reordered with other stores, and stores are never reordered with earlier loads, regardless of whether or not they're to the same location.

However, loads can be reordered with earlier stores. For example, if you write

mov 1, [%esp]
mov [%ebx], %eax

it can be executed as if you wrote

mov [%ebx], %eax
mov 1, [%esp]

But the converse isn't true — if you write the latter, it can never be executed as if you wrote the former.

You could force the first example to execute as written by inserting a serializing instruction. But that requires the CPU to serialize all instructions. But that's slow, since it effectively forces the CPU to wait until all instructions before the serializing instruction are done before executing anything after the serializing instruction. There's also an mfence instruction that only serializes loads and stores, if you only care about load/store ordering.

I'm not going to discuss the other memory fences, lfence and sfence, but you can read more about them here.

We've looked at single core ordering, where loads and stores are mostly ordered, but there's also multi-core ordering. The above restrictions all apply; if core0 is observing core1, it will see that all of the single core rules apply to core1's loads and stores. However, if core0 and core1 interact, there's no guarantee that their interaction is ordered.

For example, say that core0 and core 1 start with eax and edx set to 0, and core 0 executes

mov 1, [_foo]
mov [_foo], %eax
mov [_bar], %edx

while core1 executes

mov 1, [_bar]
mov [_bar], %eax
mov [_foo], %edx

For both cores, eax has to be 1 because of the within-core dependency between the first instruction and the second instruction. However, it's possible for edx to be 0 in both cores because line 3 of core0 can execute before core0 sees anything from core1, and visa versa.

That covers memory barriers, which serialize memory accesses within a core. Since stores are required to be seen in a consistent order across cores, they can, they also have an effect on cross-core concurrency, but it's pretty difficult to reason about that kind of thing correctly. Linus has this to say on using memory barriers instead of locking:

The real cost of not locking also often ends up being the inevitable bugs. Doing clever things with memory barriers is almost always a bug waiting to happen. It's just really hard to wrap your head around all the things that can happen on ten different architectures with different memory ordering, and a single missing barrier. … The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it.

And it turns out that on modern x86 CPUs, using locking to implement concurrency primitives is often cheaper than using memory barriers, so let's look at locks.

If we set _foo to 0 and have two threads that both execute incl (_foo) 10000 times each, incrementing the same location with a single instruction 20000 times, is guaranteed not to exceed 20000, but it could (theoretically) be as low as 2. If it's not obvious why the theoretical minimum is 2 and not 10000, figuring that out is a good exercise. If it is obvious, my bonus exercise for you is, can any reasonable CPU implementation get that result, or is that some silly thing the spec allows that will never happen? There isn't enough information in this post to answer the bonus question, but I believe I've linked to enough information.

We can try this with a simple code snippet

#include <stdlib.h>
#include <thread>

#define NUM_ITERS 10000
#define NUM_THREADS 2

int counter = 0;
int *p_counter = &counter;

void asm_inc() {
  int *p_counter = &counter;
  for (int i = 0; i < NUM_ITERS; ++i) {
    __asm__("incl (%0) \n\t" : : "r" (p_counter));
  }
}

int main () {
  std::thread t[NUM_THREADS];
  for (int i = 0; i < NUM_THREADS; ++i) {
    t[i] = std::thread(asm_inc);
  }
  for (int i = 0; i < NUM_THREADS; ++i) {
    t[i].join();
  }
  printf("Counter value: %i\n", counter);
  return 0;
}

Compiling the above with clang++ -std=c++11 -pthread, I get the following distribution of results on two of my machines:

Different distributions of non-determinism on Haswell and Sandy Bridge

Not only do the results vary between runs, the distribution of results is different on different machines. We never hit the theoretical minimum of 2, or for that matter, anything below 10000, but there's some chance of getting a final result anywhere between 10000 and 20000.

Even though incl is a single instruction, it's not guaranteed to be atomic. Internally, incl is implemented as a load followed by an add followed by an store. It's possible for an increment on cpu0 to sneak in and execute between the load and the store on cpu1 and visa versa.

The solution Intel has for this is the lock prefix, which can be added to a handful of instructions to make them atomic. If we take the above code and turn incl into lock incl, the resulting output is always 20000.

So, that's how we make a single instruction atomic. To make a sequence atomic, we can use xchg or cmpxchg, which are always locked as compare-and-swap primitives. I won't go into detail about how that works, but see this article by David Dalrymple if you're curious..

In addition to making a memory transaction atomic, locks are globally ordered with respect to each other, and loads and stores aren't re-ordered with respect to locks.

For a rigorous model of memory ordering, see the x86 TSO doc.

All of this discussion has been how about how concurrency works in hardware. Although there are limitations on what x86 will re-order, compilers don't necessarily have those same limitations. In C or C++, you'll need to insert the appropriate primitives to make sure the compiler doesn't re-order anything. As Linus points out here, if you have code like

local_cpu_lock = 1;
// .. do something critical ..
local_cpu_lock = 0;

the compiler has no idea that local_cpu_lock = 0 can't be pushed into the middle of the critical section. Compiler barriers are distinct from CPU memory barriers. Since the x86 memory model is relatively strict, some compiler barriers are no-ops at the hardware level that tell the compiler not to re-order things. If you're using a language that's higher level than microcode, assembly, C, or C++, your compiler probably handles this for you without any kind of annotation.

Memory / Porting

If you're porting code to other architectures, it's important to note that x86 has one of the strongest memory models of any architecture you're likely to encounter nowadays. If you write code that just works without thinking it through and port it to architectures that have weaker guarantees (PPC, ARM, or Alpha), you'll almost certainly have bugs.

Consider this example:

Initial
-----
x = 1;
y = 0;
p = &x;

CPU1         CPU2
----         ----
i = *p;      y = 1;
             MB;
             p = &y;

MB is a memory barrier. On an Alpha 21264 system, this can result in i = 0.

Kourosh Gharachorloo explains how:

CPU2 does y=1 which causes an "invalidate y" to be sent to CPU1. This invalidate goes into the incoming "probe queue" of CPU1; as you will see, the problem arises because this invalidate could theoretically sit in the probe queue without doing an MB on CPU1. The invalidate is acknowledged right away at this point (i.e., you don't wait for it to actually invalidate the copy in CPU1's cache before sending the acknowledgment). Therefore, CPU2 can go through its MB. And it proceeds to do the write to p. Now CPU1 proceeds to read p. The reply for read p is allowed to bypass the probe queue on CPU1 on its incoming path (this allows replies/data to get back to the 21264 quickly without needing to wait for previous incoming probes to be serviced). Now, CPU1 can derefence p to read the old value of y that is sitting in its cache (the invalidate y in CPU1's probe queue is still sitting there).

How does an MB on CPU1 fix this? The 21264 flushes its incoming probe queue (i.e., services any pending messages in there) at every MB. Hence, after the read of p, you do an MB which pulls in the invalidate to y for sure. And you can no longer see the old cached value for y.

Even though the above scenario is theoretically possible, the chances of observing a problem due to it are extremely minute. The reason is that even if you setup the caching properly, CPU1 will likely have ample opportunity to service the messages (i.e., invalidate) in its probe queue before it receives the data reply for "read p". Nonetheless, if you get into a situation where you have placed many things in CPU1's probe queue ahead of the invalidate to y, then it is possible that the reply to p comes back and bypasses this invalidate. It would be difficult for you to set up the scenario though and actually observe the anomaly.

This is long enough without my talking about other architectures so I won't go into detail, but if you're wondering why anyone would create a spec that allows this kind of optimization, consider that before rising fab costs crushed DEC, their chips were so fast that they could run industry standard x86 benchmarks of real workloads in emulation faster than x86 chips could run the same benchmarks natively. For more explanation of why the most RISC-y architecture of the time made the decisions it did, see this paper on the motivations behind the Alpha architecture.

BTW, this is a major reason I'm skeptical of the Mill architecture. Putting aside arguments about whether or not they'll live up to their performance claims, being technically excellent isn't, in and of itself, a business model.

Memory / Non-Temporal Stores / Write-Combine Memory

The set of restrictions outlined in the previous section apply to cacheable (i.e., “write-back” or WB) memory. That, itself, was new at one time. Before that, there was only uncacheable (UC) memory.

One of the interesting things about UC memory is that all loads and stores are expected to go out to the bus. That's perfectly reasonable in a processor with no cache and little to no on-board buffering. A result of that is that devices that have access to memory can rely on all accesses to UC memory regions creating separate bus transactions, in order (because some devices will use a memory read or write as as trigger to do something). That worked great in 1982, but it's not so great if you have a video card that just wants to snarf down whatever the latest update is. If multiple writes happen to the same UC location (or different bytes of the same word), the CPU is required to issue a separate bus transaction for each write, even though a video card doesn't really care about seeing each intervening result.

The solution to that was to create a memory type called write combine (WC). WC is a kind of eventually consistent UC. Writes have to eventually make it to memory, but they can be buffered internally. WC memory also has weaker ordering guarantees than UC.

For the most part, you don't have to deal with this unless you're talking directly with devices. The one exception are “non-temporal” load and store operations. These make particular loads and stores act like they're to WC memory, even if the address is in a memory region that's marked WB.

This is useful if you don't want to pollute your caches with something. This is often useful if you're doing some kind of streaming calculation where you know you're not going to use a particular piece of data more than once.

Memory / NUMA

Non-uniform memory access, where memory latencies and bandwidth are different for different processors, is so common that we mostly don't talk about NUMA or ccNUMA anymore because they're so common that it's assumed to be the default.

The takeaway here is that threads that share memory should be on the same socket, and a memory-mapped I/O heavy thread should make sure it's on the socket that's closest to the I/O device it's talking to.

I've mostly avoided explaining the why behind things because that would make this post at least an order of magnitude longer than it's going to be. But I'll give a vastly oversimplified explanation of why we have NUMA systems, partially because it's a self-contained thing that's relatively easy to explain and partially to demonstrate how long the why is compared to the what.

Once upon a time, there was just memory. Then CPUs got fast enough relative to memory that people wanted to add a cache. It's bad news if the cache is inconsistent with the backing store (memory), so the cache has to keep some information about what it's holding on to so it knows if/when it needs to write things to the backing store.

That's not too bad, but once you get 2 cores with their own caches, it gets a little more complicated. To maintain the same programming model as the no-cache case, the caches have to be consistent with each other and with the backing store. Because existing load/store instructions have nothing in their API that allows them to say sorry! this load failed because some other CPU is holding onto the address you want, the simplest thing was to have every CPU send a message out onto the bus every time it wanted to load or store something. We've already got this memory bus that both CPUs are connected to, so we just require that other CPUs respond with the data (and invalidate the appropriate cache line) if they have a modified version of the data in their cache.

That works ok. Most of the time, each CPU only touches data the other CPU doesn't care about, so there's some wasted bus traffic. But it's not too bad because once a CPU puts out a message saying Hi! I'm going to take this address and modify the data, it can assume it completely owns that address until some other CPU asks for it, which will probably won't happen. And instead of doing things on a single memory address, we can operate on cache lines that have, say, 64 bytes. So, the overall overhead is pretty low.

It still works ok for 4 CPUs, although the overhead is a bit worse. But this thing where each CPU has to respond to every other CPU's fails to scale much beyond 4 CPUs, both because the bus gets saturated and because the caches will get saturated (the physical size/cost of a cache is O(n^2) in the number of simultaneous reads and write supported, and the speed is inversely correlated to the size).

A “simple” solution to this problem is to have a single centralized directory that keeps track of all the information, instead of doing N-way peer-to-peer broadcast. Since we're packing 2-16 cores on a chip now anyway, it's pretty natural to have a single directory per chip (socket) that tracks the state of the caches for every core on a chip.

This only solves the problem for each chip, and we need some way for the chips to talk to each other. Unfortunately, while we were scaling these systems up the bus speeds got fast enough that it's really difficult to drive a signal far enough to connect up a bunch of chips and memory all on one bus, even for small systems. The simplest solution to that is to have each socket own a region of memory, so every socket doesn't need to be connected to every part of memory. This also avoids the complexity of needed a higher level directory of directories, since it's clear which directory owns any particular piece of memory.

The disadvantage of this is that if you're sitting in one socket and want some memory owned by another socket, you have a significant performance penalty. For simplicity, most “small” (< 128 core) systems use ring-like busses, so the performance penalty isn't just the direct latency/bandwidth penalty you pay for walking through a bunch of extra hops to get to memory, it also uses up a finite resource (the ring-like bus) and slows down other cross-socket accesses.

In theory, the OS handles this transparently, but it's often inefficient.

Context Switches / Syscalls

Here, syscall refers to a linux system call, not the SYSCALL or SYSENTER x86 instructions.

A side effect of all the caching that modern cores have is that context switches are expensive, which causes syscalls to be expensive. Livio Soares and Michael Stumm discuss the cost in great detail in their paper. I'm going to use a few of their figures, below. Here's a graph of how many instructions per clock (IPC) a Core i7 achieves on Xalan, a sub-benchmark from SPEC CPU.

Long tail of overhead from a syscall. 14,000 cycles.

14,000 cycles after a syscall, code is still not quite running at full speed.

Here's a table of the footprint of a few different syscalls, both the direct cost (in instructions and cycles), and the indirect cost (from the number of cache and TLB evictions).

Some of these syscalls cause 40+ TLB evictions! For a chip with a 64-entry d-TLB, that nearly wipes out the TLB. The cache evictions aren't free, either.

The high cost of syscalls is the reason people have switched to using batched versions of syscalls for high-performance code (e.g., epoll, or recvmmsg) and the reason that people who need very high performance I/O often use user space I/O stacks. More generally, the cost of context switches is why high-performance code is often thread-per-core (or even single threaded on a pinned thread) and not thread-per-logical-task.

This high cost was also the driver behind vDSO, which turns some simple syscalls that don't require any kind of privilege escalation into simple user space library calls.

SIMD

Basically all modern x86 CPUs support SSE, 128-bit wide vector registers and instructions. Since it's common to want to do the same operation multiple times, Intel added instructions that will let you operate on a 128-bit chunk of data as 2 64-bit chunks, 4 32-bit chunks, 8 16-bit chunks, etc. ARM supports the same thing with a different name (NEON), and the instructions supported are pretty similar.

It's pretty common to get a 2x-4x speedup from using SIMD instructions; it's definitely worth looking into if you've got a computationally heavy workload.

Compilers are good enough at recognizing common patterns that can be vectorized that simple code, like the following, will automatically use vector instructions with modern compilers

for (int i = 0; i < n; ++i) {
  sum += a[i];
}

But compilers will often produce non-optimal code if you don't write the assembly by hand, especially for SIMD code, so you'll want to look at the disassembly and check for compiler optimization bugs if you really care about getting the best possible performance.

Power Management

There are a lot of fancy power management feature on modern CPUs that optimize power usage in different scenarios. The result of these is that “race to idle”, completing work as fast as possible and then letting the CPU go back to sleep is the most power efficient way to work.

There's been a lot of work that's shown that specific microoptmizations can benefit power consumption, but applying those microoptimizations on real workloads often results in smaller than expected benefits.

GPU / GPGPU

I'm even less qualified to talk about this than I am about the rest of this stuff. Luckily, Cliff Burdick volunteered to write a section on GPUs, so here it is.

Prior to the mid-2000's, Graphical Processing Units (GPUs) were restricted to an API that allowed only a very limited amount of control of the hardware. As the libraries became more flexible, programmers began using the processors for more general-purpose tasks, such as linear algebra routines. The parallel architecture of the GPU could work on large chunks of a matrix by launching hundreds of simultaneous threads. However, the code had to use traditional graphics APIs and was still limited in how much of the hardware it could control. Nvidia and ATI took notice and released frameworks that allowed the user to access more of the hardware with an API familiar with people outside of the graphics industry. The libraries gained popularity, and today GPUs are widely used for high-performance computing (HPC) alongside CPUs.

Compared to CPUs, the hardware on GPUs have a few major differences, outlined below:

Processors

At the top level, a GPU processor contains one or many streaming multiprocessors (SMs). Each streaming multiprocessor on a modern GPU typically contains over 100 floating point units, or what are typically referred to as cores in the GPU world. Each core is typically clocked around 800MHz, although, like CPUs, processors with higher clock rates but fewer cores are also available. GPU processors lack many features of their CPU counterparts, including large caches and branch prediction. Between the layers of cores, SMs, and the overall processor, communicating becomes increasingly slower. For this reason, problems that perform well on GPUs are typically highly-parallel, but have some amount of data that can be shared between a small number of threads. We'll get into why this is in the memory section below.

Memory

Memory on modern GPU is broken up into 3 main categories: global memory, shared memory, and registers. Global memory is the GDDR memory that's advertised on the box of the GPU and is typically around 2-12GB in size, and has a throughput of 300-400GB/s. Global memory can be accessed by all threads across all SMs on the processor, and is also the slowest type of memory on the card. Shared memory is, as the name says, memory that's shared between all threads within the same SM. It is usually at least twice as fast as global memory, but is not accessible between threads on different SMs. Registers are much like registers on a CPU in that they are the fastest way to access data on a GPU, but they are local per thread and the data is not visible to any other running thread. Both shared memory and global memory have very strict rules on how they can be accessed, with severe performance penalties for not following them. To reach the throughputs mentioned above, memory accesses must be completely coalesced between threads within the same thread group. Similar to a CPU reading into a single cache line, GPUs have cache lines sized so that a single access can serve all threads in a group if aligned properly. However, in the worst case where all threads in a group access memory in a different cache line, a separate memory read will be required for each thread. This usually means that most of the data in the cache line is not used by the thread, and the usable throughput of the memory goes down. A similar rule applies to shared memory as well, with a couple exceptions that we won't cover here.

Threading Model

GPU threads run in a SIMT (Single Instruction Multiple Thread) fashion, and each thread runs in a group with a pre-defined size in the hardware (typically 32). That last part has many implications; every thread in that group must be working on the same instruction at the same time. If any of the threads in a group need to take a divergent path (an if statement, for example) of code from the others, all threads not part of the branch suspend execution until the branch is complete. As a trivial example:

if (threadId < 5) {
   // Do something
}
// Do More

In the code above, this branch would cause 27 of our 32 threads in the group to suspend execution until the branch is complete. You can imagine if many groups of threads all run this code, the overall performance will take a large hit while most of the cores sit idle. Only when an entire group of threads is stalled is the hardware allowed to swap in another group to run on those cores.

Interfaces

Modern GPUs must have a CPU to copy data to and from CPU and GPU memory, and to launch and code on the GPU. At the highest throughput, a PCIe 3.0 bus with 16 lanes can achieves rates of about 13-14GB/s. This may sound high, but when compared to the memory speeds residing on the GPU itself, they're over an order of magnitude slower. In fact, as GPUs get more powerful, the PCIe bus is increasingly becoming a bottleneck. To see any of the performance benefits the GPU has over a CPU, the GPU must be loaded with a large amount of work so that the time the GPU takes to run the job is significantly higher than the time it takes to copy the data to and from.

Newer GPUs have features to launch work dynamically in GPU code without returning to the CPU, but it's fairly limited in its use at this point.

GPU Conclusion

Because of the major architectural differences between CPUs and GPUs, it's hard to imagine either one replacing the other completely. In fact, a GPU complements a CPU well for parallel work and allows the CPU to work independently on other tasks as the GPU is running. AMD is attempting to merge the two technologies with their "Heterogeneous System Architecture" (HSA), but taking existing CPU code and determining how to split it between the CPU and GPU portion of the processor will be a big challenge not only for the processor, but for compilers as well.

Virtualization

Since you mentioned virtualization, I'll talk about it a bit, but Intel's implementation of virtualization instructions generally isn't something you need to think about unless you're writing very low-level code that directly deals with virtualization.

Dealing with that stuff is pretty messy, as you can see from this code. Setting stuff up to use Intel's VT instructions to launch a VM guest is about 1000 lines of low-level code, even for the very simple case shown there.

Virtual Memory

If you look at Vish's VT code, you'll notice that there's a decent chunk of code dedicated to page tables / virtual memory. That's another “new” feature that you don't have to worry about unless you're writing an OS or other low-level systems code. Using virtual memory is much simpler than using segmented memory, but that's not relevant nowadays so I'll just leave it at that.

SMT / Hyper-threading

Since you brought it up, I'll also mention SMT. As you said, this is mostly transparent for programmers. A typical speedup for enabling SMT on a single core is around 25%. That's good for overall throughput, but it means that each thread might only get 60% of its original performance. For applications where you care a lot about single-threaded performance, you might be better off disabling SMT. It depends a lot on the workload, though, and as with any other changes, you should run some benchmarks on your exact workload to see what works best.

One side effect of all this complexity that's been added to chips (and software) is that performance is a lot less predictable than it used to be; the relative importance of benchmarking your exact workload on the specific hardware it's going to run on has gone up.

Just for example, people often point to benchmarks from the Computer Languages Benchmarks Game as evidence that one language is faster than another. I've tried reproducing the results myself, and on my mobile Haswell (as opposed to the server Kentsfield that's used in the results), I get results that are different by as much as 2x (in relative speed). Running the same benchmark on the same machine, Nathan Kurz recently pointed me to an example where gcc -O3 is 25% slower than gcc -O2. Changing the linking order on C++ programs can cause a 15% performance change. Benchmarking is a hard problem.

Branches

Old school conventional wisdom is that branches are expensive, and should be avoided at all (or most) costs. On a Haswell, the branch misprediction penalty is 14 cycles. Branch mispredict rates depend on the workload. Using perf stat on a few different things (bzip2, top, mysqld, regenerating my blog), I get branch mispredict rates of between 0.5% and 4%. If we say that a correctly predicted branch costs 1 cycle, that's an average cost of between .995 * 1 + .005 * 14 = 1.065 cycles to .96 * 1 + .04 * 14 = 1.52 cycles. That's not so bad.

This actually overstates the penalty since about 1995, since Intel added conditional move instructions that allow you to conditionally move data without a branch. This instruction was memorably panned by Linus, which has given it a bad reputation, but it's fairly common to get significant speedups using cmov compared to branches

A real-world example of the cost of extra branches are enabling integer overflow checks. When using bzip2 to compress a particular file, that increases the number of instructions by about 30% (with all of the increase coming from extra branch instructions), which results in a 1% performance hit.

Unpredictable branches are bad, but most branches are predictable. Ignoring the cost of branches until your profiler tells you that you have a hot spot is pretty reasonable nowadays. CPUs have gotten a lot better at executing poorly optimized code over the past decade, and compilers are getting better at optimizing code, which makes optimizing branches a poor use of time unless you're trying to squeeze out the absolute best possible performance out of some code.

If it turns out that's what you need to do, you're likely to be better off using profile-guided optimization than trying to screw with this stuff by hand.

If you really must do this by hand, there are compiler directives you can use to say whether a particular branch is likely to be taken or not. Modern CPUs ignore branch hint instructions, but they can help the compiler lay out code better.

Alignment

Old school conventional wisdom is that you should pad out structs and make sure things are aligned. But on a Haswell chip, the mis-alignment for almost any single-threaded thing you can think of that doesn't cross a page boundary is zero. There are some cases where it can make a difference, but in general, this is another type of optimization that's mostly irrelevant because CPUs have gotten so much better at executing bad code. It's also mildly harmful in cases where it increases the memory footprint for no benefit.

Also, don't make things page aligned or otherwise aligned to large boundaries or you'll destroy the performance of your caches.

Self-modifying code

Here's another optimization that doesn't really make sense anymore. Using self-modifying code to decrease code size or increase performance used to make sense, but because modern caches tend to split up their l1 instruction and data caches, modifying running code requires expensive communication between a chip's l1 caches.

The Future

Here are some possible changes, from least speculative to most speculative.

Partitioning

It's now obvious that more and more compute is moving into large datacenters. Sometimes this involves running on VMs, sometimes it involves running in some kind of container, and sometimes it involves running bare metal, but in any case, individual machines are often multiplexed to run a wide variety of workloads. Ideally, you'd be able to schedule best effort workloads to soak up stranded resources without effecting latency sensitive workloads with an SLA. It turns out that you can actually do this with some relatively straightforward hardware changes.

David Lo, et. al, were able to show that you can get about 90% machine utilization without impacting latency SLAs if caches can be partitioned such that best effort workloads don't impact latency sensitive workloads. The solid red line is the load on a normal Google web search cluster, and the dashed green line is what you get with the appropriate optimizations. From bar-room conversations, my impression is that the solid red line is actually already better (higher) than most of Google's competitors are able to do. If you compare the 90% optimized utilization to typical server utilization of 10% to 90%, that results in a massive difference in cost per unit of work compared to running a naive, unoptimized, setup. With substantial hardware effort, Google was able to avoid interference, but additional isolation features could allow this to be done at higher efficiency with less effort.

Transactional Memory and Hardware Lock Elision

IBM already has these features in their POWER chips. Intel made an attempt to add these to Haswell, but they're disabled because of a bug. In general, modern CPUs are quite complex and we should expect to see many more bugs than we used to.

Transactional memory support is what it sounds like: hardware support for transactions. This is through three new instructions, xbegin, xend, and xabort.

xbegin starts a new transaction. A conflict (or an xabort) causes the architectural state of the processor (including memory) to get rolled back to the state it was in just prior to the xbegin. If you're using transactional memory via library or language support, this should be transparent to you. If you're implementing the library support, you'll have to figure out how to convert this hardware support, with its limited hardware buffer sizes, to something that will handle arbitrary transactions.

I'm not going to discuss Hardware Lock Elision except to say that, under the hood, it's implemented with mechanisms that are really similar to the mechanisms used to implement transactional memory and that it's designed to speed up lock-based code. If you want to take advantage of HLE, see this doc.

Fast I/O

I/O bandwidth is going up and I/O latencies are going down, both for storage and for networking. The problem is that I/O is normally done via syscalls. As we've seen, the relative overhead of syscalls has been going up. For both storage and networking, the answer is to move to user mode I/O stacks (putting everything in kernel mode would work, too, but that's a harder sell). On the storage side, that's mostly still a weirdo research thing, but HPC and HFT folks have been doing that in networking for a while. And by a while, I don't mean a few months. Here's a paper from 2005 that talks about the networking stuff I'm going to discuss, as well as some stuff I'm not going to discuss (DCA).

This is finally trickling into the non-supercomputing world. MS has been advertising Azure with infiniband networking with virtualized RDMA for over a year, Cloudflare has talked about using Solarflare NICs to get the same capability, etc. Eventually, we're going to see SoCs with fast Ethernet onboard, and unless that's limited to Xeon-type devices, it's going to trick down into all devices. The competition between ARM devices will probably cause at least one ARM device maker to put fast Ethernet on their commodity SoCs, which may force Intel's hand.

That RDMA bit is significant; it lets you bypass the CPU completely and have the NIC respond to remote requests. A couple months ago, I worked through the Stanford/Coursera Mining Massive Data Sets class. During one of the first lectures, they provide an example of a “typical” datacenter setup with 1Gb top-of-rack switches. That's not unreasonable for processing “massive” data if you're doing kernel TCP through non-RDMA NICs, since you can floor an entire core trying to push 1Gb/s through linux's TCP stack. But with Azure, MS talks about getting 40Gb out of a single machine; that's one machine getting 40x the bandwidth of what you might expect out of an entire rack. They also mention sub 2 us latencies, which is multiple orders of magnitude lower than you can get out of kernel TCP. This isn't exactly a new idea. This paper from 2011 predicts everything that's happened on the network side so far, along with some things that are still a ways off.

This MS talk discusses how you can take advantage of this kind of bandwidth and latency for network storage. A concrete example that doesn't require clicking through to a link is Amazon's EBS. It lets you use an “elastic” disk of arbitrary size on any of your AWS nodes. Since a spinning metal disk seek has higher latency than an RPC over kernel TCP, you can get infinite storage pretty much transparently. For example, say you can get 100us (.1ms) latency out of your network, and your disk seek time is 8ms. That makes a remote disk access 8.1ms instead of 8ms, which isn't that much overhead. That doesn't work so well with SSDs, though, since you can get 20 us (.02ms) out of an SSD. But RDMA latency is low enough that a transparent EBS-like layer is possible for SSDs.

So that's networked I/O. The performance benefit might be even bigger on the disk side, if/when next generation storage technologies that are faster than flash start getting deployed. The performance delta is so large that Intel is adding new instructions to keep up with next generation low-latency storage technology. Depending on who you ask, that stuff has been a few years away for a decade or two; this is more iffy than the networking stuff. But even with flash, people are showing off devices that can get down into the single microsecond range for latency, which is a substantial improvement.

Hardware Acceleration

Like fast networked I/O, this is already here in some niches. DESRES has been doing ASICs to get 100x-1000x speedup in computational chemistry for years. Microsoft has talked about speeding up search with FPGAs. People have been looking into accelerating memcached and similar systems for a while, researchers from Toshiba and Stanford demonstrated a real implementation a while back, and I recently saw a pre-print out of Berkeley on the same thing. There are multiple companies making Bitcoin mining ASICs. That's also true for other application areas.

It seems like we should see more of this as it gets harder to get power/performance gains out of CPUs. You might consider this a dodge of your question, if you think of programming as being a software oriented endeavor, but another way to look at it is that what it means to program something will change. In the future, it might mean designing hardware like an FPGA or ASIC in combination with writing software.

Update

Now that it's 2016, one year after this post was originally published, we can see that companies are investing in hardware accelerators. In addition to its previous work on FPGA accelerated search, Microsoft has announced that it's using FPGAs to accelerate networking. Google has been closed mouthed about infrastructure, as is typical for them, but if you look at the initial release of Tensorflow, you can see snippets of code that clearly references FPGAs, such as:

enum class PlatformKind {
  kInvalid,
  kCuda,
  kOpenCL,
  kOpenCLAltera,  // Altera FPGA OpenCL platform.
                  // See documentation: go/fpgaopencl
                  // (StreamExecutor integration)
  kHost,
  kMock,
  kSize,
};

and

string PlatformKindString(PlatformKind kind) {
  switch (kind) {
    case PlatformKind::kCuda:
      return "CUDA";
    case PlatformKind::kOpenCL:
      return "OpenCL";
    case PlatformKind::kOpenCLAltera:
      return "OpenCL+Altera";
    case PlatformKind::kHost:
      return "Host";
    case PlatformKind::kMock:
      return "Mock";
    default:
      return port::StrCat("InvalidPlatformKind(", static_cast<int>(kind), ")");
  }
}

As of this writing, Google doesn't return any results for +google +kOpenClAltera, so it doesn't appear that this has been widely observed. If you're not familiar with Altera OpenCL and you work at google, you can try the internal go link suggested in the comment, go/fpgaopencl. If, like me, you don't work at Google, well, there's Altera's docs here. The basic idea is that you can take OpenCL code, the same kind of thing you might run on a GPU, and run it on an FPGA instead, and from the comment, it seems like Google has some kind of setup that lets you stream data in and out of nodes with FPGAs.

That FPGA-specific code was removed in ddd4aaf5286de24ba70402ee0ec8b836d3aed8c7, which has a commit message that starts with “TensorFlow: upstream changes to git.” and then has a list of internal google commits that are being upstreamed, along with a description of each internal commit. Curiously, there's nothing about removing FPGA support even though that seems like it's a major enough thing that you'd expect it to be described, unless it was purposely redacted. Amazon has also been quite secretive about their infrastructure plans, but you can make reasonable guesses there by looking at the hardware people they've been vacuuming up. A couple other companies are also betting pretty heavily on hardware accelerators, but since I learned about that through private conversations (as opposed to accidentally published public source code or other public information), I'll leave you to guess which companies.

Dark Silicon / SoCs

One funny side effect of the way transistor scaling has turned out is that we can pack a ton of transistors on a chip, but they generate so much heat that the average transistor can't switch most of the time if you don't want your chip to melt.

A result of this is that it makes more sense to include dedicated hardware that isn't used a lot of the time. For one thing, this means we get all sorts of specialized instructions like the PCMP and ADX instructions. But it also means that we're getting chips with entire devices integrated that would have previously lived off-chip. That includes things like GPUs and (for mobile devices) radios.

In combination with the hardware acceleration trend, it also means that it makes more sense for companies to design their own chips, or at least parts of their own chips. Apple has gotten a lot of mileage out of acquiring PA Semi. First, by adding little custom accelerators to bog standard ARM architectures, and then by adding custom accelerators to their own custom architecture. Due to a combination of the right custom hardware plus well thought out benchmarking and system design, the iPhone 4 is slightly more responsive than my flagship Android phone, which is multiple years newer and has a much faster processor as well as more RAM.

Amazon has picked up a decent chunk of the old Calxeda team and are hiring enough to create a good-sized hardware design team. Facebook has picked up a small handful of ARM SoC folks and is partnering with Qualcomm on something-or-other. Linus is on record as saying we're going to see more dedicated hardware all over the place. And so on and so forth.

Conclusion

x86 chips have picked up a lot of new features and whiz-bang gadgets. For the most part, you don't have to know what they are to take advantage of them. As a first-order approximation, making your code predictable and keeping memory locality in mind works pretty well. The really low-level stuff is usually hidden by libraries or drivers, and compilers will try to take care of the rest of it. The exceptions are if you're writing really low-level code, in which case the world has gotten a lot messier, or if you're trying to get the absolute best possible performance out of your code, in which case the world has gotten a lot weirder.

Also, things will happen in the future. But most predictions are wrong, so who knows?

Resources

This is a talk by Matt Godbolt that covers a lot of the implementation details that I don't get into. To down into one more level of detail, see Modern Processor Design, by Shen and Lipasti. Despite the date listed on Amazon (2013), the book is pretty old, but it's still the best book I've found on processor internals. It describes, in good detail, what you need to implement to make a P6-era high-performance CPU. It also derives theoretical performance limits given different sets of assumptions and talks about a lot of different engineering tradeoffs, with explanations of why for a lot of them.

For one level deeper of "why", you'll probably need to look at a VLSI text, which will explain how devices and interconnect scale and how that affects circuit design, which in turn affects architecture. I really like Weste & Harris because they have clear explanations and good exercises with solutions that you can find online, but if you're not going to work the problems pretty much any VLSI text will do. For one more level deeper of the "why" of things, you'll want a solid state devices text and something that explains how transmission lines and interconnect can work. For devices, I really like Pierret's books. I got introduced to the E-mag stuff through Ramo, Whinnery & Van Duzer, but Ida is a better intro text.

For specifics about current generation CPUs and specific optimization techniques, see Agner Fog's site. For something on optimization tools from the future, see this post. What Every Programmer Should Know About Memory is also good background knowledge. Those docs cover a lot of important material, but if you're writing in a higher level language there are a lot of other things you need to keep in mind. For more on Intel CPU history, Xao-Feng Li has a nice overview.

For something a bit off the wall, see this post on the possibility of CPU backdoors. For something less off the wall, see this post on how complexity we have in modern CPUs enables all sorts of exciting bugs.

For more benchmarks on locking, See this post by Aleksey Shipilev, this post by Paul Khuong, as well as their archives.

For general benchmarking, last year's Strange Loop benchmarking talk by Aysylu Greenberg is a nice intro to common gotchas. For something more advanced but more specific, Gil Tene's talk on latency is great.

For historical computing that predates everything I've mentioned by quite some time, see IBM's Early Computers and Design of a Computer, which describes the design of the CDC 6600. Readings in Computer Architecture is also good for seeing where a lot of these ideas originally came from.

Sorry, this list is pretty incomplete. Suggestions welcome!

Tiny Disclaimer

I have no doubt that I'm leaving stuff out. Let me know if I'm leaving out anything you think is important and I'll update this. I've tried to keep things as simple as possible while still capturing the flavor of what's going on, but I'm sure that there are some cases where I'm oversimplifying, and some things that I just completely forgot to mention. And of course basically every generalization I've made is wrong if you're being really precise. Even just picking at my first couple sentences, A20M isn't always and everywhere irrelevant (I've probably spent about 0.2% of my career dealing with it), x86-64 isn't strictly superior to x86 (on one workload I had to deal with, the performance benefit from the extra registers was more than canceled out by the cost of the longer instructions; it's pretty rare that the instruction stream and icache misses are the long pole for a workload, but it happens), etc. The biggest offender is probably in my NUMA explanation, since it is actually possible for P6 busses to respond with a defer or retry to a request. It's reasonable to avoid using a similar mechanism to enforce coherency but I couldn't think of a reasonable explanation of why that didn't involve multiple levels of explanations. I'm really not kidding when I say that pretty much every generalization falls apart if you dig deep enough. Every abstraction I'm talking about is leaky. I've tried to include links to docs that go at least one level deeper, but I'm probably missing some areas.

Acknowledgments

Thanks to Leah Hanson and Nathan Kurz for comments that results in major edits, and to Nicholas Radcliffe, Stefan Kanthak, Garret Reid, Matt Godbolt, Nikos Patsiouras, Aleksey Shipilev, and Oscar R Moll for comments that resulted in minor edits, and to David Albert for allowing me to quote him and also for some interesting follow-up questions when we talked about this a while back. Also, thanks for Cliff Burdick for writing the section on GPUs and for Hari Angepat for spotting the Google kOpenCLAltera code in TensorFlow.


show more
Blog monetization
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-01-24 00:00:00 | Created: 2026-07-23 05:18:40

Does it make sense for me to run ads on my blog? I've been thinking about this lately, since Carbon Ads contacted me about putting an ad up. What are the pros and cons? This isn't a rhetorical question. I'm genuinely interested in what you think.

Pros

Money

Hey, who couldn't use more money? And it's basically free money. Well, except for the all of the downsides.

Data

There's lots of studies on the impact of ads on site usage and behavior. But as with any sort of benchmarking, it's not really clear how or if that generalizes to other sites if you don't have a deep understanding of the domain, and I have almost no understanding of the domain. If I run some ads and do some A/B testing I'll get to see what the effect is on my site, which would be neat.

Cons

Money

It's not enough money to make a living off of, and it's never going to be. When Carbon contacted me, they asked me how much traffic I got in the past 30 days. At the time, Google Analytics showed 118k sessions, 94k users, 143k page views. Cloudflare tends to show about 20% higher traffic since 20% of people block Google Analytics, but those 20% plus more probably block ads, so the "real" numbers aren't helpful here. I told them that, but I also told them that those numbers were pretty unusual and that I'd expect to average much less traffic.

How much money is that worth? I don't know if the CPM (cost per thousand impressions) numbers they gave me are confidential, so I'll just use a current standard figure of $1 CPM. If my traffic continued at that rate, that would be $143/month, or $1,700/year. Ok, that's not too bad.

Distribution of traffic on this blog. About 500k hits total.

But let's look at the traffic since I started this blog. I didn't add analytics until after a post of mine got passed around on HN and reddit, so this isn't all of my traffic, but it's close.

For one thing, the 143k hits over a 30-day period seems like a fluke. I've never had a calendar month with that much traffic. I just happen to have a traffic distribution which turned up a bunch of traffic over a specific 30-day period.

Also, if I stop blogging, as I did from April to October, my traffic level drops to pretty much zero. And even if I keep blogging, it's not really clear what my “natural” traffic level is. Is the level before I paused my blogging the normal level or the level after? Either way, $143/month seems like a good guess for an upper bound. I might exceed that, but I doubt it.

For a hard upper bound, let's look at one of the most widely read programming blogs, Coding Horror. Jeff Atwood is nice enough to make his traffic stats available. Thanks Jeff!

Distribution of traffic on Coding Horror. 1.7M hits in a month at its peak.

He got 1.7M hits in his best month, and 1.25M wouldn't be a bad month for him, even when he was blogging regularly. With today's CPM rates, that's $1.7k/month at his peak and $1.25k/month for a normal month.

But Jeff Atwood blogs about general interest programming topics, like Markdown and Ruby and I blog about obscure stuff, like why Intel might want to add new instructions to speed up non-volatile storage with the occasional literature review for variety. There's no way I can get as much traffic as someone who blogs about more general interest topics; I'd be surprised if I could even get within a factor of 2, so $600/month seems like a hard and probably unreachable upper bound for sustainable income.

That's not bad. After taxes, that would have approximately covered my rent when I lived in Austin, and could have covered rent + utilities and other expenses if I'd had a roommate. But the wildly optimistic success rate is that you barely cover rent when the programming job market is hot enough that mid-level positions at big companies pay out total compensation that's 8x-9x the median income in the U.S. That's not good.

Worse yet, this is getting worse over time. CPM is down something like 5x since the 90s, and continues to decline. Meanwhile, the percentage of people using ad blockers continues to increase.

Premium ads can get well over an order of magnitude higher CPM and sponsorships can fetch an ever better return, so the picture might not be quite as bleak as I'm making it out to be. But to get premium ads you need to appeal to specific advertisers. What advertisers are interested in an audience that's mostly programmers with an interest in low-level shenanigans? I don't know, and I doubt it's worth the effort to find out unless I can get to Jeff Atwood levels of traffic, which I find unlikely.

A Tangent on Alexa Rankings

What's up with Alexa? Why do so many people use it as a gold standard? In theory, it's supposed to show how popular a site was over the past three months. According to Alexa, Coding Horror is ranked at 22k and I'm at 162k. My understanding is that traffic is more than linear in rank so you'd expect Coding Horror to have substantially more than 7x the traffic that I do. But if you compare Jeff's stats to mine over the past three months (Oct 21 - Jan 21), statcounter claims he's had 78k hits compared to my 298k hits. Even if you assume that traffic is merely linear in Alexa rank, that's a 28x difference in relative traffic between the direct measurement and Alexa's estimate.

I'm not claiming that my blog is more popular in any meaningful sense -- if Jeff posted as often as I did in the past three months, I'm sure he'd have at least 10x more traffic than me. But given that Jeff now spends most of his time on non-blogging activities and that his traffic is at the level it's at when he rarely blogs, the Alexa ranks for our sites seem way off.

Moreover, the Alexa sub-metrics are inconsistent and nonsensical. Take this graph on the relative proportion of users who use this site from home, school, or work.

Relatively below average, at everything!

It's below average in every category, which should be impossible for a relative ranking like this. But even mathematical impossibility doesn't stop Alexa!

Traffic

Ads reduce traffic. How much depends both on the site and the ads. I might do a literature review some other time, but for now I'm just going to link to this single result by Daniel G. Goldstein, Siddharth Suri, R. Preston McAfee, Matthew Ekstrand-Abueg, and Fernando Diaz that attempts to quantify the cost.

My point isn't that some specific study applies to adding a single ad to my site, but that it's well known that adding ads reduces traffic and has some effect on long-term user behavior, which has some cost.

It's relatively easy to quantify the cost if you're looking at something like the study above, which compares “annoying” ads to “good” ads to see what the cost of the “annoying” ads are. It's harder to quantify for a personal blog where the baseline benefit is non-monetary.

What do I get out of this blog, anyway? The main benefits I can see are that I've met and regularly correspond with some great people I wouldn't have otherwise met, that I often get good feedback on my ideas, and that every once in a while someone pings me about a job that sounds interesting because they saw a relevant post of mine.

I doubt I can effectively estimate the amount of traffic I'll lose, and even if I could, I doubt I could figure out the relationship between that and the value I get out of blogging. My gut says that the value is “a lot” and that the monetary payoff is probably “not a lot”, but it's not clear what that means at the margin.

Incentives

People are influenced by money, even when they don't notice it. I'm people. I might do something to get more revenue, even though the dollar amount is small and I wouldn't consciously spend a lot of effort of optimizing things to get an extra $5/month.

What would that mean here? Maybe I'd write more blog posts? When I experimented with blurting out blog posts more frequently, with less editing, I got uniformly positive feedback, so maybe being incentivized to write more wouldn't be so bad. But I always worry about unconscious bias and I wonder what other effects running ads might have on me.

Privacy

Ad networks can track people through ads. My impression is that people are mostly concerned with really big companies that have enough information that they could deanonymize people if they were so inclined, like Google and Facebook, but some people are probably also concerned about smaller ad networks like Carbon. Just as an aside, I'm curious if companies that attempt to do lots of tracking, like Tapad and MediaMath actually have more data on people than better known companies like Yahoo and eBay. I doubt that kind of data is publicly available, though.

Paypal

This is specific to Carbon, but they pay out through PayPal, which is notorious for freezing funds for six months if you get enough money that you'd actually want the money, and for pseudo-randomly draining your bank account due to clerical errors. I've managed to avoid hooking my PayPal account up to my bank account so far, but I'll have to either do that or get money out through an intermediary if I end up making enough money that I want to withdraw it.

Conclusion

Is running ads worth it? I don't know. If I had to guess, I'd say no. I'm going to try it anyway because I'm curious what the data looks like, and I'm not going to get to see any data if I don't try something, but it's not like that data will tell me whether or not it was worth it.

At best, I'll be able to see a difference in click-through rates on my blog with and without ads. This blog mostly spreads through word of mouth, so what I really want to see is the difference in the rate at which the blog gets shared with other people, but I don't see a good way to do that. I could try globally enabling or disabling ads for months at a time, but the variance between months is so high that I don't know that I'd get good data out of that even if I did it for years.

Thanks to Anja Boskovic for comments/corrections/discussion.

Update

After running an ads for a while, it looks like about 40% of my traffic uses an ad blocker (whereas about 17% of my traffic blocks Google Analytics). I'm not sure if I should be surprised that the number is so high or that it's so low. On the one hand, 40% is a lot! On the other hand, despite complaints that ad blockers slow down browsers, my experience has been that web pages load a lot faster when I'm blocking ads using the right ad blocker and I don't see any reason not to use an blocker. I'd expect that most of my traffic comes from programmers, who all know that ad blocking is possible.

There's the argument that ad blocking is piracy and/or stealing, but I've never heard a convincing case made. If anything, I think that some of the people who make that argument step over the line, as when ars technica blocked people who used ad blockers, and then backed off and merely exhorted people to disable ad blocking for their site. I think most people would agree that directly exhorting people to click on ads and commit click fraud is unethical; asking people to disable ad blocking is a difference in degree, not in kind. People who use ad blockers are much less likely to click on ads, so having them disable ad blockers to generate impressions that are unlikely to convert strikes me as pretty similar to having people who aren't interested in the product generate clicks.

Anyway, I ended up removing this ad after they failed to send a payment after the first payment. AdSense is rumored to wait until just before payment before cutting people off, to get as many impressions as possible for free, but AdSense at least notifies you about it. Carbon just stopped paying without saying anything, while still running the ad. I could probably ask someone at Carbon or BuySellAds about it, but considering how little the ad is worth, it's not really worth the hassle of doing that.

Update 2

It's been almost two years since I said that I'd never get enough traffic for blogging to be able to cover my living expenses. It turns out that's not true! My reasoning was that I mostly tend to blog about low-level technical topics, which can't possibly generate enough traffic to generate "real" ad revenue. That reason is still as valid as ever, but my blogging is now approximately half low-level technical stuff, and half general-interest topics for programmers.

Traffic for one month on this blog in 2016. Roughly 3.1M hits.

Here's a graph of my traffic for the past 30 days (as of October 25th, 2016). Since this is Cloudflare's graph of requests, this would wildly overestimate traffic for most sites, because each image and CSS file is one request. However, since the vast majority of my traffic goes to pages with no external CSS and no images, this is pretty close to my actual level of traffic. 15% of the requests are images, and 10% is RSS (which I won't count because the rate of RSS hits is hard to correlation to the rate of actual people reading). But that means that 75% of the traffic appears to be "real", which puts the traffic into this site at roughly 2.3M hits per month. At a typical $1 ad CPM, that's $2.3k/month, which could cover my share of household expenses.

Additionally, when I look at blogs that really try to monetize their traffic, they tend to monetize at a much better rate. For example, Slate Star Codex charges $1250 for 6 months of ads and appears to be running 8 ads, for a total of $20k/yr. The author claims to get "10,000 to 20,000 impressions per day", or roughly 450k hits per month. I get about 5x that much traffic. If we scale that linearly, that might be $100k/yr instead of $20k/yr. One thing that I find interesting is that the ads on Slate Star Codex don't get blocked by my ad blocker. It seems like that's because the author isn't part of some giant advertising program, and ad blockers don't go out of their way to block every set of single-site custom ads out there. I'm using Slate Star Codex as an example because I think it's not super ad optimized because I doubt I would optimize my ads much if I ran ads.

This is getting to the point where it seems a bit unreasonable not to run ads (I doubt the non-direct value I get out of this blog can consistently exceed $100k/yr). I probably "should" run ads, but I don't think the revenue I get from something like AdSense or Carbon is really worth it, and it seems like a hassle to run my own ad program the way Slate Star Codex does. It seems totally irrational to leave $90k/yr on the table because "it seems like a hassle", but here we are. I went back and added affiliate code to all of my Amazon links, but if I'm estimating Amazon's payouts correctly, that will amount to less than $100/month.

I don't think it's necessarily more irrational than behavior I see from other people -- I regularly talk to people who leave $200k/yr or more on the table by working for startups instead of large companies, and that seems like a reasonable preference to me. They make "enough" money and like things the way they are. What's wrong with that? So why can't not running ads be a reasonable preference? It still feels pretty unreasonable to me, though! A few people have suggested crowdfunding, but the top earning programmers have at least an order of magnitude more exposure than I do and make an order of magnitude less than I could on ads (folks like Casey Muratori, ESR, and eevee are pulling in around $1000/month).

Update 3

I'm now trying donations via Patreon. I suspect this won't work, but I'd be happy to be wrong!

show more
CPU backdoors
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-02-03 00:00:00 | Created: 2026-07-23 05:18:40

It's generally accepted that any piece of software could be compromised with a backdoor. Prominent examples include the Sony/BMG installer, which had a backdoor built-in to allow Sony to keep users from copying the CD, which also allowed malicious third-parties to take over any machine with the software installed; the Samsung Galaxy, which has a backdoor that allowed the modem to access the device's filesystem, which also allows anyone running a fake base station to access files on the device; Lotus Notes, which had a backdoor which allowed encryption to be defeated; and Lenovo laptops, which pushed all web traffic through a proxy (including HTTPS, via a trusted root certificate) in order to push ads, which allowed anyone with the correct key (which was distributed on every laptop) to intercept HTTPS traffic.

Despite sightings of backdoors in FPGAs and networking gear, whenever someone brings up the possibility of CPU backdoors, it's still common for people to claim that it's impossible. I'm not going to claim that CPU backdoors exist, but I will claim that the implementation is easy, if you've got the right access.

Let's say you wanted to make a backdoor. How would you do it? There are three parts to this: what could a backdoored CPU do, how could the backdoor be accessed, and what kind of compromise would be required to install the backdoor?

Starting with the first item, what does the backdoor do? There are a lot of possibilities. The simplest is to allow privilege escalation: make the CPU to transition from ring3 to ring0 or SMM, giving the running process kernel-level privileges. Since it's the CPU that's doing it, this can punch through both hardware and software virtualization. There are a lot of subtler or more invasive things you could do, but privilege escalation is both simple enough and powerful enough that I'm not going to discuss the other options.

Now that you know what you want the backdoor to do, how should it get triggered? Ideally, it will be something that no one will run across by accident, or even by brute force, while looking for backdoors. Even with that limitation, the state space of possible triggers is huge.

Let's look at a particular instruction, fyl2x1. Under normal operation, it takes two floating point registers as input, giving you 2*80=160 bits to hide a trigger in. If you trigger the backdoor off of a specific pair of values, that's probably safe against random discovery. If you're really worried about someone stumbling across the backdoor by accident, or brute forcing a suspected backdoor, you can check more than the two normal input registers (after all, you've got control of the CPU).

This trigger is nice and simple, but the downside is that hitting the trigger probably requires executing native code since you're unlikely to get chrome or Firefox to emit an fyl2x instruction. You could try to work around that by triggering off an instruction you can easily get a JavaScript engine to emit (like an fadd). The problem with that is that if you patch an add instruction and add some checks to it, it will become noticeably slower (although, if you can edit the hardware, you should be able to do it with no overhead). It might be possible to create something hard to detect that's triggerable through JavaScript by patching a rep string instruction and doing some stuff to set up the appropriate “key” followed by a block copy, or maybe idiv. Alternately, if you've managed to get a copy of the design, you can probably figure out a way to use debug logic triggers2 or performance counters to set off a backdoor when some arbitrary JavaScript gets run.

Alright, now you've got a backdoor. How do you insert the backdoor? In software, you'd either edit the source or the binary. In hardware, if you have access to the source, you can edit it as easily as you can in software. The hardware equivalent of recompiling the source, creating physical chips, has tremendously high fixed costs; if you're trying to get your changes into the source, you'll want to either compromise the design3 and insert your edits before everything is sent off to get manufactured, or compromise the manufacturing process and sneak in your edits at the last second4.

If that sounds too hard, you could try compromising the patch mechanism. Most modern CPUs come with a built-in patch mechanism to allow bug fixes after the fact. It's likely that the CPU you're using has been patched, possibly from day one, and possibly as part of a firmware update. The details of the patch mechanism for your CPU are a closely guarded secret. It's likely that the CPU has a public key etched into it, and that it will only accept a patch that's been signed by the right private key.

Is this actually happening? I have no idea. Could it be happening? Absolutely. What are the odds? Well, the primary challenge is non-technical, so I'm not the right person to ask about that. If I had to guess, I'd say no, if for no other reason than the ease of subverting other equipment.

I haven't discussed how to make a backdoor that's hard to detect even if someone has access to software you've used to trigger a backdoor. That's harder, but it should be possible once chips start coming with built-in TPMs.

If you liked this post, you'll probably enjoy this post on CPU bugs and might be interested in this post about new CPU features over the past 35 years.

Updates

See this twitter thread for much more discussion, some of which is summarized below.

I'm not going to provide individual attributions because there are too many comments, but here's a summary of comments from @hackerfantastic, Arrigo Triulzi, David Kanter, @solardiz, @4Dgifts, Alfredo Ortega, Marsh Ray, and Russ Cox. Mistakes are my own, of course.

AMD's K7 and K8 had their microcode patch mechanisms compromised, allowing for the sort of attacks mentioned in this post. Turns out, AMD didn't encrypt updates or validate them with a checksum, which lets you easily modify updates until you get one that does what you want.

Here's an example of a backdoor that was created for demonstration purposes, by Alfredo Ortega.

For folks without a hardware background, this talk on how to implement a CPU in VHDL is nice, and it has a section on how to implement a backdoor.

Is it possible to backdoor RDRAND by providing bad random results? Yes. I mentioned that in my first draft of this post, but I got rid of it since my impression was that people don't trust RDRAND and mix the results other sources of entropy. That doesn't make a backdoor useless, but it significantly reduces the value.

Would it be possible to store and dump AES-NI keys? It's probably infeasible to sneak flash memory onto a chip without anyone noticing, but modern chips have logic analyzer facilities that let you store and dump data. However, access to those is through some secret mechanism and it's not clear how you'd even get access to binaries that would let you reverse engineer their operation. That's in stark contrast to the K8 reverse engineering, which was possible because microcode patches get included in firmware updates.

It would be possible to check instruction prefixes for the trigger. x86 lets you put redundant (and contradictory) instruction prefixes on instructions. Which prefixes get used are well defined, so you can add as many prefixes as you want without causing problems (up to the prefix length limit). The issues with this are that it's probably hard to do without sacrificing performance with a microcode patch, the limited number of prefixes and the length limit mean that your effective key size is relatively small if you don't track state across multiple instructions, and that you can only generate the trigger with native code.

As far as anyone knows, this is all speculative, and no one has seen an actual CPU backdoor being used in the wild.

Acknowledgments

Thanks to Leah Hanson for extensive comments, to Aleksey Shipilev and Joe Wilder for suggestions/corrections, and to the many participants in the twitter discussion linked to above. Also, thanks to Markus Siemens for noticing that a bug in some RSS readers was causing problems, and for providing the workaround. That's not really specific to this post, but it happened to come up here.


  1. This choice of instruction is somewhat, but not completely, arbitrary. You'll probably want an instruction that's both slow and microcoded, to make it easy to patch with a microcode patch without causing a huge performance hit. The rest of this footnote is about what it means for an instruction to be microcoded. It's quite long and not in the critical path of this post, so you might want to skip it.

    The distinction between a microcoded instruction and one that's implemented in hardware is, itself, somewhat arbitrary. CPUs have an instruction set they implement, which you can think of as a public API. Internally, they can execute a different instruction set, which you can think of as a private API.

    On modern Intel chips, instructions that turn into four (or fewer) uops (private API calls) are translated into uops directly by the decoder. Instructions that result in more uops (anywhere from five to hundreds or possibly thousands) are decoded via a microcode engine that reads uops out of a small ROM or RAM on the CPU. Why four and not five? That's a result of some tradeoffs, not some fundamental truth. The terminology for this isn't standardized, but the folks I know would say that an instruction is “microcoded” if its decode is handled by the microcode engine and that it's “implemented in hardware” if its decode is handled by the standard decoder. The microcode engine is sort of its own CPU, since it has to be able to handle things like reading and writing from temporary registers that aren't architecturally visible, reading and writing from internal RAM for instructions that need more than just a few registers of scratch space, conditional microcode branches that change which microcode the microcode engine fetches and decodes, etc.

    Implementation details vary (and tend to be secret). But whatever the implementation, you can think of the microcode engine as something that loads a RAM with microcode when the CPU starts up, which then fetches and decodes microcoded instructions out of that RAM. It's easy to modify what microcode gets executed by changing what gets loaded on boot via a microcode patch.

    For quicker turnaround while debugging, it's somewhere between plausible and likely that Intel also has a mechanism that lets them force non-microcoded instructions to execute out of the microcode RAM in order to allow them to be patched with a microcode patch. But even if that's not the case, compromising the microcode patch mechanism and modifying a single microcoded instruction should be sufficient to install a backdoor.

    [return]
  2. For the most part, these aren't publicly documented, but you can get a high-level overview of what kind of debug triggers Intel was building into their chips a couple generators ago starting at page 128 of Intel Technology Journal, Volume 4, Issue 3. [return]
  3. For the past couple years, there's been a debate over whether or not major corporations have been compromised and whether such a thing is even possible. During the cold war, government agencies on all sides were compromised at various levels for extended periods of time, despite having access to countermeasures not available to any corporations today (not hiring citizens of foreign countries, "enhanced interrogation techniques", etc.). I'm not sure that we'll ever know if companies are being compromised, but it would certainly be easier to compromise a present-day corporation than it was to compromise government agencies during the cold war, and that was eminently doable. Compromising a company enough to get the key to the microcode patch is trivial compared to what was done during the cold war. [return]
  4. This is another really long footnote about minutia! In particular, it's about the manufacturing process. You might want to skip it! If you don't, don't say I didn't warn you.

    It turns out that editing chips before manufacturing is fully complete is relatively easy, by design. To explain why, we'll have to look at how chips are made.

    Cross section of Intel chip, 22nm process

    When you look at a cross-section of a chip, you see that silicon gates are at the bottom, forming logical primitives like nand gates, with a series of metal layers above (labeled M1 through M8), forming wires that connect different gates. A cartoon model of the manufacturing process is that chips are built from the bottom up, one layer a time, where each layer is created by depositing some material and then etching part of it away using a mask, in a process that's analogous to lithographic printing. The non-cartoon version involves a lot of complexity -- Todd Fernendez estimates that it takes about 500 steps to create the layers below “M1”. Additionally, the level of precision needed is high enough that the light used to etch causes enough wear in the equipment that it wears out. You probably don't normally think about lenses wearing out due to light passing through them, but at the level of precision required for each of the hundreds of steps required to make a transistor, it's a serious problem. If that sounds surprising to you, you're not alone. An ITRS roadmap from the 90s predicted that by 2016, we'd be at almost 30GHz (higher is better) on a 9nm process (smaller is better), with chips consuming almost 300 watts. Instead, 5 GHz is considered pretty fast, and anyone who isn't Intel will be lucky to get high-yield production on a 14nm process by the start of 2016. Making chips is harder than anyone guessed it would be.

    A modern chip has enough layers that it takes about three months to make one, from start to finish. This makes bugs very bad news since a bug fix that requires a change to one of the bottom layers takes three months to manufacture. In order to reduce the turnaround time on bug fixes, it's typical to scatter unused logic gates around the silicon, to allow small bug fixes to be done with an edit to a few layers that are near the top. Since chips are made in a manufacturing line process, at any point in time, there are batches of partially complete chips. If you only need to edit one of the top metal layers, you can apply the edit to a partially finished chip, cutting the turnaround time down from months to weeks.

    Since chips are designed to allow easy edits, someone with access to the design before the chip is manufactured (such as the manufacturer) can make major changes with relatively small edits. I suspect that if you were to make this comment to anyone at a major CPU company, they'd tell you it's impossible to do this without them noticing because it would get caught in characterization or when they were trying to find speed paths or something similar. One would hope, but actual hardware devices have shipped with backdoors, and either no one noticed, or they were complicit.

    [return]
show more
AI doesn't have to be very good to displace humans
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-02-15 00:00:00 | Created: 2026-07-23 05:18:40

There's an ongoing debate over whether "AI" will ever be good enough to displace humans and, if so, when it will happen. In this debate, the optimists tend to focus on how much AI is improving and the pessimists point to all the ways AI isn't as good as an ideal human being. I think this misses two very important factors.

One, is that jobs that are on the potential chopping block, such as first-line customer service, customer service for industries that are either low margin or don't care about the customer, etc., tend to be filled by apathetic humans in a poorly designed system, and humans aren't even very good at simple tasks they care a lot about. When we're apathetic, we're absolutely terrible; it's not going to take a nearly-omniscient sci-fi level AI to perform at least somewhat comparably.

Two, companies are going to replace humans with AI in many roles even if AI is significantly worse as if the AI is much cheaper. One place this has already happened (though perhaps this software is too basic to be considered an AI) is with phone trees. Phone trees are absolutely terrible compared to the humans they replaced, but they're also orders of magnitude cheaper. Although there are high-margin high-touch companies that won't put you through a phone tree, at most companies, for a customer looking for customer service, a huge number of work hours have been replaced by phone trees, and were then replaced again by phone trees with poor AI voice recognition that I find worse than old school touch pad phone trees. It's not a great experience, and may get much worse when AI automates even more of the process.

But on the other hand, here's a not-too-atypical customer service interaction I had last week with a human who was significantly worse than a mediocre AI. I scheduled an appointment for an MRI. The MRI is for a jaw problem which makes it painful to talk. I was hoping that the scheduling would be easy, so I wouldn't have to spend a lot of time talking on the phone. But, as is often the case when dealing with bureaucracy, it wasn't easy.

Here are the steps it took.

  1. Have jaw pain.
  2. See dentist. Get referral for MRI when dentist determines that it's likely to be a joint problem.
  3. Dentist gets referral form from UW Health, faxes it to them according to the instructions on the form, and emails me a copy of the referral.
  4. Call UW Health.
  5. UW Health tries to schedule me for an MRI of my pituitary.
  6. Ask them to make sure there isn't an error.
  7. UW Health looks again and realizes that's a referral for something else. They can't find anything for me.
  8. Ask UW Health to call dentist to work it out. UW Health claims they cannot make phone calls.
  9. Talk to dentist again. Ask dentist to fax form again.
  10. Call UW Health again. Ask them to check again.
  11. UW Health says form is illegally filled out.
  12. Ask them to call dentist to work it out, again.
  13. UW Health says that's impossible.
  14. Ask why.
  15. UW Health says, “for legal reasons”.
  16. Realize that's probably a vague and unfounded fear of HIPAA regulations. Try asking again nicely for them to call my dentist, using different phrasing.
  17. UW Health agrees to call dentist. Hangs up.
  18. Look at referral, realize that it's actually impossible for someone outside of UW Health (like my dentist) to fill out the form legally given the instructions on the form.
  19. Talk to dentist again.
  20. Dentist agrees form is impossible, talks to UW Health to figure things out.
  21. Call UW Health to see if they got the form.
  22. UW Health acknowledges receipt of valid referral.
  23. Ask to schedule earliest possible appointment.
  24. UW Health isn't sure they can accept referrals from dentists. Goes to check.
  25. UW Health determines it is possible to accept a referral from a dentist.
  26. UW Health suggests a time on 2/17.
  27. I point out that I probably can't make it because of a conflicting appointment, also with UW Health, which I know about because I can see it on my profile with I log into the UW Health online system.
  28. UW Health suggests a time on 2/18.
  29. I point out another conflict that is in the UW Health system.
  30. UW Health starts looking for times on later dates.
  31. I ask if there are any other times available on 2/17.
  32. UW Health notices that there are other times available on 2/17 and schedules me later on 2/17.

I present this not because it's a bad case, but because it's a representative one1. In this case, my dentist's office was happy to do whatever was necessary to resolve things, but UW Health refused to talk to them without repeated suggestions that talking to my dentist would be the easiest way to resolve things. Even then, I'm not sure it helped much. This isn't even all that bad, since I was able to convince the intransigent party to cooperate. The bad cases are when both parties refuse to talk to each other and both claim that the situation can only be resolved when the other party contacts them, resulting in a deadlock. The good cases are when both parties are willing to talk to each other and work out whatever problems are necessary. Having a non-AI phone tree or web app that exposes simple scheduling would be far superior to the human customer service experience here. An AI chatbot that's a light wrapper around the API a web app would use would be worse than being able to use a normal website, but still better than human customer service. An AI chatbot that's more than a just a light wrapper would blow away the humans who do this job for UW Health.

The case against using computers instead of humans is that computers are bad at handling error conditions, can't adapt to unusual situations, and behave according to mechanical rules, which can often generate ridiculous outcomes, but that's precisely the situation we're in right now with humans. It already feels like dealing with a computer program. Not a modern computer program, but a compiler from the 80s that tells you that there's at least one error, with no other diagnostic information.

UW Health sent a form with impossible instructions to my dentist. That's not great, but it's understandable; mistakes happen. However, when they got the form back and it wasn't correctly filled out, instead of contacting my dentist they just threw it away. Just like an 80s compiler. Error! The second time around, they told me that the form was incorrectly filled out. Error! There was a human on the other end who could have noted that the form was impossible to fill out. But like an 80s compiler, they stopped at the first error and gave it no further thought. This eventually got resolved, but the error messages I got along the way were much worse than I'd expect from a modern program. Clang (and even gcc) give me much better error messages than I got here.

Of course, as we saw with healthcare.gov, outsourcing interaction to computers doesn't guarantee good results. There are some claims that market solutions will automatically fix any problem, but those claims don't always work out.

That's an ad someone was running for a few months on Facebook in order to try to find a human at Google to help them because every conventional technique they had at their disposal failed. Google has perhaps the most advanced ML in the world, they're as market driven as any other public company, and they've mostly tried to automate away service jobs like first-level support because support doesn't scale. As a result, the most reliable methods of getting support at Google are

  1. Be famous enough that a blog post or tweet will get enough attention to garner a response.
  2. Work at Google or know someone who works at Google and is willing to not only file an internal bug, but to drive it to make sure it gets handled.

If you don't have direct access to one of these methods, running an ad is actually a pretty reasonable solution. (1) and (2) don't always work, but they're more effective than not being famous and hoping a blog post will hit HN, or being a paying customer. The point here isn't to rag on Google, it's just that automated customer service solutions aren't infallible, even when you've got an AI that can beat the strongest go player in the world and multiple buildings full of people applying that same technology to practical problems.

While replacing humans with computers doesn't always create a great experience, good computer based systems for things like scheduling and referrals can already be much better than the average human at a bureaucratic institution2. With the right setup, a computer-based system can be better at escalating thorny problems to someone who's capable of solving them than a human-based system. And computers will only get better at this. There will be bugs. And there will be bad systems. But there are already bugs in human systems. And there are already bad human systems.

I'm not sure if, in my lifetime, technology will advance to the point where computers can be as good as helpful humans in a well designed system. But we're already at the point where computers can be as helpful as apathetic humans in a poorly designed system, which describes a significant fraction of service jobs.

2023 update

When ChatGPT was released in 2022, the debate described above in 2015 happened again, with the same arguments on both sides. People are once again saying that AI (this time, ChatGPT and LLMs) can't replace humans because a great human is better than ChatGPT. They'll often pick a couple examples of ChatGPT saying something extremely silly, "hallucinating", but if you ask a human to explain something, even a world-class expert, they often hallucinate a totally fake explanation as well.

Many people on the pessimist side argued that it would be decades before LLMs can replace humans for the exact reasons we noted were false in 2015. Everyone made this argument after multiple industries had massive cuts in the number of humans they need to employ due to pre-LLM "AI" automation and many of these people even made this argument after companies had already laid people off and replaced people with LLMs. I commented on this at the time, using the same reasoning I used in this 2015 post before realizing that I'd already written down this line of reasoning in 2015. But, cut me some slack; I'm just a human, not a computer, so I have a fallible memory.

Now that it's been a year ChatGPT was released, the AI pessimists who argued that LLMs would displace human jobs for a very long time have been proven even more wrong by layoff after layoff where customer service orgs were cut to the bone and mostly replaced by AI, AI customer service seems quite poor, just like human customer service. But human customer service isn't improving, while AI customer service is. For example, here are some recent customer service interactions I had as a result of bringing my car in to get the oil changed, rotate the tires, and do a third thing (long story).

  1. I call my local tire shop and oil change place3 and ask if they can do the three things I want with my car
  2. They say yes
  3. I ask if I can just drop by or if I need to make an appointment
  4. They say yes, I can just drop by to get the world done
  5. I ask if I can talk to the service manager directly to get some more info
  6. After being transferred to the service manager, I describe what I want again and ask when I can come in
  7. They say that will take a lot of time and I'll need to make an appointment. They can get me in next week. If I listened to the first guy, I would've had a completely pointless one-hour round trip drive since they couldn't, in fact, do the work I wanted as a drop-in
  8. A week later, I bring the car in and talk to someone at the desk, who asks me what I need done
  9. I describe what I need and notice that he only writes down about 1/3 of what I said, so I follow up and
  10. I ask what oil they're going to use
  11. The guy says "we'll use the right oil"
  12. I tell him that I want 0W-20 synthetic because my car has a service bulletin indicating that this is recommended, which is different from the label on the car, so could they please note this.
  13. The guy repeats "we'll use the right oil".
  14. (12) again, with slightly different phrasing
  15. (13) again, with slightly different phrasing
  16. (12) again, with slightly different phrasing
  17. The guy says, "it's all in the computer, the computer has the right oil".
  18. I ask him what oil the computer says to use
  19. Annoyed, the guy walks over to the computer and pull up my car, telling me that my car should use 5W-30
  20. I tell him that's not right for my vehicle due to the service bulletin and I want 0W-20 synthetic
  21. The guy, looking shocked, says "Oh", and then looks at the computer and says "oh, it says we can also use 0W-20"
  22. The guy writes down 0W-20 on the sheet for my car
  23. I leave, expecting that the third thing I asked for won't be done or won't completely be done since it wasn't really written down
  24. The next day, I pick up my car and they fully didn't do the third thing.

Overall, how does an LLM compare? It's probably significantly better than this guy, who acted like an archetypical stoner who doesn't want to be there and doesn't want to do anything, and the LLM will be cheaper as well. However, the LLM will be worse than a web interface that lets me book the exact work I want and write a note to the tech who's doing the work. For better or for worse, I don't think my local tire / oil change place is going to give me a nice web interface that lets me book the exact work I want any time soon, so this guy is going to be replaced by an LLM and not a simple web app.

Elsewhere

Thanks to Leah Hanson and Josiah Irwin for comments/corrections/discussion.


  1. Representative of my experience in Madison, anyway. The absolute worst case of this I encountered in Austin isn't even as bad as the median case I've seen in Madison. YMMV. [return]
  2. I wonder if a deranged version of the law of one price applies, the law of one level of customer service. However good or bad an organization is at customer service, they will create or purchase automated solutions that are equally good or bad.

    At Costco, the checkout clerks move fast and are helpful, so you don't have much reason to use the automated checkout. But then the self-checkout machines tend to be well-designed; they're physically laid out to reduce the time it takes to feed a large volume of stuff through them, and they rarely get confused and deadlock, so there's not much reason not to use them. At a number of other grocery chains, the checkout clerks are apathetic and move slowly, and will make mistakes unless you remind them of what's happening. It makes sense to use self-checkout at those places, except that the self-checkout machines aren't designed particularly well and are often configured so that they often get confused and require intervention from an overloaded checkout clerk.

    The same thing seems to happen with automated phone trees, as well as both of the examples above. Local Health has an online system to automate customer service, but they went with Epic as the provider, and as a result it's even worse than dealing with their phone support. And it's possible to get a human on the line if you're a customer on some Google products, but that human is often no more helpful than the automated system you'd otherwise deal with.

    [return]
  3. BTW, this isn't a knock against my local tire shop. I used my local tire shop because they're actually above average! I've also tried the local dealership, which is fine but super expensive, and a widely recommended independent Volvo specialist, which was much worse — they did sloppy work and missed important issues and were sloppy elsewhere as well; they literally forgot to order parts for the work they were going to do (a mistake an AI probably wouldn't have made), so I had to come back another day to finish the work on my car! [return]
show more
Goodhearting IQ, cholesterol, and tail latency
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-03-05 00:00:00 | Created: 2026-07-23 05:18:40

Most real-world problems are big enough that you can't just head for the end goal, you have to break them down into smaller parts and set up intermediate goals. For that matter, most games are that way too. “Win” is too big a goal in chess, so you might have a subgoal like don't get forked. While creating subgoals makes intractable problems tractable, it also creates the problem of determining the relative priority of different subgoals and whether or not a subgoal is relevant to the ultimate goal at all. In chess, there are libraries worth of books written on just that.

And chess is really simple compared to a lot of real world problems. 64 squares. 32 pieces. Pretty much any analog problem you can think of contains more state than chess, and so do a lot of discrete problems. Chess is also relatively simple because you can directly measure whether or not you succeeded (won). Many real-world problems have the additional problem of not being able to measure your goal directly.

IQ & Early Childhood Education

In 1962, what's now known as the Perry Preschool Study started in Ypsilanti, a blue-collar town near Detroit. It was a randomized trial, resulting in students getting either no preschool or two years of free preschool. After two years, students in the preschool group showed a 15 point bump in IQ scores; other early education studies showed similar results.

In the 60s, these promising early results spurred the creation of Head Start, a large scale preschool program designed to help economically disadvantaged children. Initial results from Head Start were also promising; children in the program got a 10 point IQ boost.

The next set of results was disappointing. By age 10, the difference in test scores and IQ between the trial and control groups wasn't statistically significant. The much larger scale Head Start study showed similar results; the authors of the first major analysis of Head Start concluded that

(1) Summer programs are ineffective in producing lasting gains in affective and cognitive development, (2) full-year programs are ineffective in aiding affective development and only marginally effective in producing lasting cognitive gains, (3) all Head Start children are still considerably below national norms on tests of language development and scholastic achievement, while school readiness at grade one approaches the national norm, and (4) parents of Head Start children voiced strong approval of the program. Thus, while full-year Head Start is somewhat superior to summer Head Start, neither could be described as satisfactory.

Education in the U.S. isn't cheap, and these early negative results caused calls for reductions in funding and even the abolishment of the program. Turns out, it's quite difficult to cut funding for a program designed to help disadvantaged children, and the program lives on despite repeated calls to cripple or kill the program.

Well after the initial calls to shut down Head Start, long-term results started coming in from the Perry preschool study. As adults, people in the experimental (preschool) group were less likely to have been arrested, less likely to have spent time in prison, and more likely to have graduated from high school. Unfortunately, due to methodological problems in the study design, it's not 100% clear where these effects come from. Although the goal was to do a randomized trial, the experimental design necessitated home visits for the experimental group. As a result, children in the experimental group whose mothers were employed swapped groups with children in the control group whose mothers were unemployed. The positive effects on the preschool group could have been caused by having at-home mothers. Since the Head Start studies weren't randomized and using instrumental variables (IVs) to tease out causation in “natural experiments” didn't become trendy until relatively recently, it took a long time to get plausible causal results from Head Start.

The goal of analyses with an instrumental variable is to extract causation, the same way you'd be able to in a randomized trial. A classic example is determining the effect of putting kids into school a year earlier or later. Some kids naturally start school a year earlier or later, but there are all sorts of factors that can cause that happen, which means that a correlation between an increased likelihood of playing college sports in kids who started school a year later could just as easily be from the other factors that caused kids to start a year later as it could be from actually starting school a year later.

However, date of birth can be used as an instrumental variable that isn't correlated with those other factors. For each school district, there's an arbitrary cutoff that causes kids on one side of the cutoff to start school a year later than kids on the other side. With the not-unreasonable assumption that being born one day later doesn't cause kids to be better athletes in college, you can see if starting school a year later seems to have a causal effect on the probability of playing sports in college.

Now, back to Head Start. One IV analysis used a funding discontinuity across counties to generate a quasi experiment. The idea is that there are discrete jumps in the level of Head Start funding across regions that are caused by variations in a continuous variable, which gives you something like a randomized trial. Moving 20 feet across the county line doesn't change much about kids or families, but it moves kids into an area with a significant change in Head Start funding.

The results of other IV analyses on Head Start are similar. Improvements in test scores faded out over time, but there were significant long-term effects on graduation rate (high school and college), crime rate, health outcomes, and other variables that are more important than test scores.

There's no single piece of incredibly convincing evidence. The randomized trial has methodological problems, and IV analyses nearly always leave some lingering questions, but the weight of the evidence indicates that even though scores on standardized tests, including IQ tests, aren't improved by early education programs, people's lives are substantially improved by early education programs. However, if you look at the early commentary on programs like Head Start, there's no acknowledgment that intermediate targets like IQ scores might not perfectly correlate with life outcomes. Instead you see declarations like “poor children have been so badly damaged in infancy by their lower-class environment that Head Start cannot make much difference”.

The funny thing about all this is that it's well known that IQ doesn't correlate perfectly to outcomes. In the range of environments that you see in typical U.S. families, to correlation to outcomes you might actually care about has an r value in the range of .3 to .4. That's incredibly strong for something in the social sciences, but even that incredibly strong statement is a statement IQ isn't responsible for "most" of the effect on real outcomes, even ignoring possible confounding factors.

Cholesterol & Myocardial Infarction

There's a long history of population studies showing a correlation between cholesterol levels and an increased risk of heart attack. A number of early studies found that lifestyle interventions that made cholesterol levels more favorable also decreased heart attack risk. And then statins were invented. Compared to older drugs, statins make cholesterol levels dramatically better and have a large effect on risk of heart attack.

Prior to the invention of statins, the standard intervention was a combination of diet and pre-statin drugs. There's a lot of literature on this; here's one typical review that finds, in randomized trials, a combination of dietary changes and drugs has a modest effect on both cholesterol levels and heart attack risk.

Given that narrative, it certainly sounds reasonable to try to develop new drugs that improve cholesterol levels, but when Pfizer spent $800 million doing exactly that, developing torcetrapib, they found that they created a drug which substantially increased heart attack risk despite improving cholesterol levels. Hoffman-La Roche's attempt fared a bit better because it improved cholesterol without killing anyone, but it still failed to decrease heart attack risk. Merck and Tricor have also had the same problem.

What happened? Some interventions that affected cholesterol levels also affected real health outcomes, prompting people to develop drugs that affect cholesterol. But it turns out that improving cholesterol isn't an inherent good, and like many intermediate targets, it's possible to improve without affecting the end goal.

99%-ile Latency & Latency

It's pretty common to see latency measurements and benchmarks nowadays. It's well understood that poor latency in applications costs you money, as it causes people to stop using the application. It's also well understood that average latency (mean, median, or mode), by itself, isn't a great metric. It's common to use 99%-ile, 99.9%-ile, 99.99%-ile, etc., in order to capture some information about the distribution and make sure that bad cases aren't too bad.

What happens when you use the 99%-iles as intermediate targets? If you require 99%-ile latency to be under 0.5 millisec and 99.99% to be under 5 millisecond you might get a latency distribution that looks something like this.

This is a graph of an actual application that Gil Tene has been showing off in his talks about latency. If you specify goals in terms of 99%-ile, 99.9%-ile, and 99.99%-ile, you'll optimize your system to barely hit those goals. Those optimizations will often push other latencies around, resulting in a funny looking distribution that has kinks at those points, with latency that's often nearly as bad as possible everywhere else.

It's is a bit odd, but there's nothing sinister about this. If you try a series of optimizations while doing nothing but looking at three numbers, you'll choose optimizations that improve those three numbers, even if they make the rest of the distribution much worse. In this case, latency rapidly degrades above the 99.99%-ile because the people optimizing literally had no idea how much worse they were making the 99.991%-ile when making changes. It's like the video game solving AI that presses pause before its character is about to get killed, because pausing the game prevents its health from decreasing. If you have very narrow optimization goals, and your measurements don't give you any visibility into anything else, everything but your optimization goals is going to get thrown out the window.

Since the end goal is usually to improve the user experience and not just optimize three specific points on the distribution, targeting a few points instead of using some kind of weighted integral can easily cause anti-optimizations that degrade the actual user experience, while producing great slideware.

In addition to the problem of optimizing just the 99%-ile to the detriment of everything else, there's the question of how to measure the 99%-ile. One method of measuring latency, used by multiple commonly used benchmarking frameworks, is to do something equivalent to

for (int i = 0; i < NUM; ++i) {
  auto a = get_time();
  do_operation();
  auto b = get_time();
  measurements[i] = b - a;
}

If you optimize the 99%-ile of that measurement, you're optimizing the 99%-ile for when all of your users get together and decide to use your app sequentially, coordinating so that no one requests anything until the previous user is finished.

Consider a contrived case where you measure for 20 seconds. For the first 10 seconds, each response takes 1ms. For the 2nd 10 seconds, the system is stalled, so the last request takes 10 seconds, resulting in 10,000 measurements of 1ms and 1 measurement of 10s. With these measurements, the 99%-ile is 1ms, as is the 99.9%-ile, for the matter. Everything looks great!

But if you consider a “real” system where users just submit requests, uniformly at random, the 75%-ile latency should be >= 5 seconds because if any query comes during the 2nd half, it will get jammed up, for an average of 5 seconds and as much as 10 seconds, in addition to whatever queuing happens because requests get stuck behind other requests.

If this example sounds contrived, it is; if you'd prefer a real world example, see this post by Nitsan Wakart, which finds shows how YCSB (Yahoo Cloud Serving Benchmark) has this problem, and how different the distributions look before and after the fix.

Order of magnitude latency differences between YCSB's measurement and the truth

The red line is YCSB's claimed latency. The blue line is what the latency looks like after Wakart fixed the coordination problem. There's more than an order of magnitude difference between the original YCSB measurement and Wakart's corrected version.

It's important to not only consider the whole distribution, to make make sure you're measuring a distribution that's relevant. Real users, which can be anything from a human clicking something on a web app, to an app that's waiting for an RPC, aren't going to coordinate to make sure they don't submit overlapping requests; they're not even going to obey a uniform random distribution.

Conclusion

This is the point in a blog post where you're supposed to get the one weird trick that solves your problem. But the only trick is that there is no trick, that you have to constantly check that your map is somehow connected to the territory1.

Resources

1990 HHS Report on Head Start. 2012 Review of Evidence on Head Start.

A short article on instrumental variables. A book on econometrics and instrumental variables.

Aysylu Greenberg video on benchmarking pitfalls; it's not latency specific, but it covers a wide variety of common errors. Gil Tene video on latency; covers many more topics than this post. Nitsan Wakart on measuring latency; has code examples and links to libraries.

Acknowledgments

Thanks to Leah Hanson for extensive comments on this, and to Scott Feeney and Kyle Littler for comments that resulted in minor edits.


  1. Unless you're in school and your professor likes to give problems where the answers are nice, simple, numbers, maybe the weird trick is that you know you're off track if you get an intermediate answer with a 170/23 in front of it. [return]
show more
What happens when you load a URL?
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-03-07 00:00:00 | Created: 2026-07-23 05:18:40

I've been hearing this question a lot lately, and when I do, it reminds me how much I don't know. Here are some questions this question brings to mind.

  1. How does a keyboard work? Why can’t you press an arbitrary combination of three keys at once, except on fancy gaming keyboards? That implies something about how key presses are detected/encoded.
  2. How are keys debounced? Is there some analog logic, or is there a microcontroller in the keyboard that does this, or what? How do membrane switches work?
  3. How is the OS notified of the keypress? I could probably answer this for a 286, but nowadays it's somehow done through x2APIC, right? How does that work?
  4. Also, USB, PS/2, and AT keyboards are different, somehow? How does USB work? And what about laptop keyboards? Is that just a USB connection?
  5. How does a USB connector work? You have this connection that can handle 10Gb/s. That surely won't work if there's any gap at all between the physical doodads that are being connected. How do people design connectors that can withstand tens of thousands of insertions and still maintain their tolerances?
  6. How does the OS tell the program something happened? How does it know which program to talk to?
  7. How does the browser know to try to load a webpage? I guess it sees an "http://" or just assumes that anything with no prefix is a URL?
  8. Assume we don't have the webpage cached, so we have to do DNS queries and stuff.
  9. How does DNS work? How does DNS caching work? Let's assume it isn't cached at anywhere nearby and we have to go find some far away DNS server.
  10. TCP? We establish a connection? Do we do that for DNS or does it have to be UDP?
  11. How does the OS decide if an outgoing connection should be allowed? What if there's a software firewall? How does that work?
  12. For TCP, without TLS/SSL, we can just do slow-start followed by some standard congestion protocol, right? Is there some deeper complexity there?
  13. One level down, how does a network card work?
  14. For what matter, how does the network card know what to do? Is there a memory region we write to that the network card can see or does it just monitor bus transactions directly?
  15. Ok, say there's a memory region. How does that work? How do we write memory?
  16. Some things happen in the CPU/SoC! This is one of the few areas where I know something, so, I'll skip over that. A signal eventually comes out on some pins. What's that signal? Nowadays, people use DDR3, but we didn't always use that protocol. Presumably DDR3 lets us go faster than DDR2, which was faster than DDR, and so on, but why?
  17. And then the signal eventually goes into a DRAM module. As with the CPU, I'm going to mostly ignore what's going on inside, but I'm curious if DRAM modules still either trench capacitors or stacked capacitors, or has this technology moved on?
  18. Going back to our network card, what happens when the signal goes out on the wire? Why do you need a cat5 and not a cat3 cable for 100Mb Ethernet? Is that purely a signal integrity thing or do the cables actually have different wiring?
  19. One level below that the wires are surely long enough that they can act like transmission lines / waveguides. How is termination handled? Is twisted pair sufficient to prevent inductive coupling or is there more fancy stuff going on?
  20. Say we have a local Ethernet connection to a cable modem. How do cable modems work? Isn't cable somehow multiplexed between different customers? How is it possible to get so much bandwidth through a single coax cable?
  21. Going back up a level, the cable connection eventually gets to the ISP. How does the ISP know where to route things? How does internet routing work? Some bits in the header decide the route? How do routing tables get adjusted?
  22. Also, the 8.8.8.8 DNS stuff is anycast, right? How is that different from routing "normal" traffic? Ditto for anything served from a Cloudflare CDN. What do they need to do to prevent route flapping and other badness?
  23. What makes anycast hard enough to do that very few companies use it?
  24. IIRC, the Stanford/Coursera algorithms course mentioned that it's basically a distributed Bellman-Ford calculation. But what prevents someone from putting bogus routes up?
  25. If we can figure out where to go our packets go from our ISP through some edge router, some core routers, another edge router, and then go through their network to get into the “meat” of a datacenter.
  26. What's the difference between core and edge routers?
  27. At some point, our connection ends up going into fiber. How does that happen?
  28. There must be some kind of laser. What kind? How is the signal modulated? Is it WDM or TDM? Is it single-mode or multi-mode fiber?
  29. If it's WDM, how is it muxed/demuxed? It would be pretty weird to have a prism in free space, right? This is the kind of thing an AWG could do. Is that what's actually used?
  30. There must be repeaters between links. How do repeaters work? Do they just boost the signal or do they decode it first to avoid propagating noise? If the latter, there must be DCF between repeaters.
  31. Something that just boosts the signal is the simplest case. How does an EDFA work? Is it basically just running current through doped fiber, or is there something deeper going on there?
  32. Below that level, there's the question of how standard single mode fiber and DCF work.
  33. Why do we need DCF, anyway? I guess it's cheaper to have a combination of standard fiber and DCF than to have fiber with very low dispersion. Why is that?
  34. How does fiber even work? I mean, ok, it's probably a waveguide that uses different dielectrics to keep the light contained, but what's the difference between good fiber and bad fiber?
  35. For example, hasn't fiber changed over the past couple decades to severely reduce PMD? How is that possible? Is that just more precise manufacturing, or is there something else involved?
  36. Before PMD became a problem and was solved, there was decades of work that went into increasing fiber bandwidth, vaugely analogous to the way there was decades of work that went into increasing processor performance but also completely different. What was that work and what were the blockers that work was clearing? You'd have to actually know a good deal about fiber engineering to answer this, and I don't.
  37. Going back up a few levels, we go into a datacenter. What's up there? Our packets go through a switching network to TOR to machine? What's a likely switch topology? Facebook's isn't quite something straight out of Dally and Towles, but it's the kind of thing you could imagine building with that kind of knowledge. It hasn't been long enough since FB published their topology for people to copy them, but is the idea obvious enough that you'd expect it to be independently "copied"?
  38. Wait, is that even right? Should we expect a DNS server to sit somewhere in some datacenter?
  39. In any case, after all this our DNS resolves query to an IP. We establish a connection, and then what?
  40. HTTP GET? How are HTTP 1.0 and 1.1 different? 2.0?
  41. And then we get some files back and the browser has to render them somehow. There's a request for the HTML and also for the CSS and js, and separate requests for images? This must be complicated, since browsers are complicated. I don't have any idea of the complexity of this, so there must be a lot I'm missing.
  42. After the browser renders something, how does it get to the GPU and what does the GPU do?
  43. For 2d graphics, we probably just notify the OS of... something. How does that work?
  44. And how does the OS talk to the GPU? Is there some memory mapped region where you can just paint pixels, or is it more complicated than that?
  45. How does an LCD display work? How does the connection between the monitor and the GPU work?
  46. VGA is probably the simplest possibility. How does that work?
  47. If it's a static site, I guess we're done?
  48. But if the site has ads, isn't that stuff pretty complicated? How do targeted ads and ad auctions work? A bunch of stuff somehow happens in maybe 200ms?

Where I can get answers to this stuff1? That's not a rhetorical question! I'm really interested in hearing about other resources!

Alex Gaynor set up a GitHub repo that attempts to answer this entire question. It answers some of the questions, and has answers to some questions it didn't even occur to me to ask, but it's missing answers to the vast majority of these questions.

For high-level answers, here's Tali Garsiel and Paul Irish on how a browser works and Jessica McKellar how the Internet Works. For how a simple OS does things, Xv6 has good explanations. For how Linux works, Gustavo Duarte has a series of explanations hereFor TTYs, this article by Linus Akesson is a nice supplement to Duarte's blog.

One level down from that, James Marshall has a concise explanation of HTTP 1.0 and 1.1, and SANS has an old but readable guide on SSL and TLS. This isn't exactly smooth prose, but this spec for URLs explains in great detail what a URL is.

Going down another level, MS TechNet has an explanation of TCP, which also includes a short explanation of UDP.

One more level down, Kyle Cassidy has a quick primer on Ethernet, Iljitsch van Beijnum has a lengthier explanation with more history, and Matthew J Castelli has an explanation of LAN switches. And then we have DOCSIS and cable modems. This gives a quick sketch of how long haul fiber is set up, but there must be a better explanation out there somewhere. And here's a quick sketch of modern CPUs. For an answer to the keyboard specific questions, Simon Inns explains keypress decoding and why you can't press an arbitrary combination of keys on a keyboard.

Down one more level, this explains how wires work, Richard A. Steenbergen explains fiber, and Pierret explains transistors.

P.S. As an interview question, this is pretty much the antithesis of the tptacek strategy. From what I've seen, my guess is that tptacek-style interviews are much better filters than open ended questions like this.

Thanks to Marek Majkowski, Allison Kaptur, Mindy Preston, Julia Evans, Marie Clemessy, and Gordon P. Hemsley for providing answers and links to resources with answers! Also, thanks to Julia Evans and Sumana Harihareswara for convincing me to turn these questions into a blog post.


  1. I mostly don't have questions about stuff that happens inside a PC listed, but I'm pretty curious about how modern high-speed busses work and how high-speed chips deal with the massive inductance they must have to deal with getting signals to and from the chip. [return]
show more
Given that we spend little effort on testing, how should we test software?
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-03-10 00:00:00 | Created: 2026-07-23 05:18:40

I've been reading a lot about software testing, lately. Coming from a hardware background (CPUs and hardware accelerators), it's interesting how different software testing is. Bugs in software are much easier to fix, so it makes sense to spend a lot less effort spent on testing. Because less effort is spent on testing, methodologies differ; software testing is biased away from methods with high fixed costs, towards methods with high variable costs. But that doesn't explain all of the differences, or even most of the differences. Most of the differences come from a cultural path dependence, which shows how non-optimally test effort is allocated in both hardware and software.

I don't really know anything about software testing, but here are some notes from what I've seen at Google, on a few open source projects, and in a handful of papers and demos. Since I'm looking at software, I'm going to avoid talking about how hardware testing isn't optimal, but I find that interesting, too.

Manual Test Generation

From what I've seen, most test effort on most software projects comes from handwritten tests. On the hardware projects I know of, writing tests by hand consumed somewhere between 1% and 25% of the test effort and was responsible for a much smaller percentage of the actual bugs found. Manual testing is considered ok for sanity checking, and sometimes ok for really dirty corner cases, but it's not scalable and too inefficient to rely on.

It's true that there's some software that's difficult to do automated testing on, but the software projects I've worked on have relied pretty much totally on manual testing despite being in areas that are among the easiest to test with automated testing. As far as I can tell, that's not because someone did a calculation of the tradeoffs and decided that manual testing was the way to go, it's because it didn't occur to people that there were alternatives to manual testing.

At the hardware company I worked for, we called what programmers call tests, "hand tests" or "hand jobs" because they were written by hand (the latter isn't innuedo, it's becasue we launched tests into "jobs" in our job system). What people in the software world call "fuzzing", "property based testing", "randomized testing", etc., we just called testing because that was the default. How else would you test? Sure, you might have 1% of test writing time go to hand tests (whch would mean some tiny fraction of actual tests were written by hand, surely less than 0.00000001%), but no one would actually spend a signifiant amount of time writing tests by hand, would they?

So, what do you do?

Random Test Generation

The good news is that random testing is easy to implement. You can spend an hour implementing a random test generator and find tens of bugs, or you can spend more time and find thousands of bugs.

You can start with something that's almost totally random and generates incredibly dumb tests. As you spend more time on it, you can add constraints and generate smarter random tests that find more complex bugs. Some good examples of this are jsfunfuzz, which started out relatively simple and gained smarts as time went out, and Jepsen, which originally checked some relatively simple constraints and can now check linearizability.

While you can generate random tests pretty easily, it still takes some time to write a powerful framework or collection of functions. Luckily, this space is well covered by existing frameworks.

[2026 update: I'm writing this long after I originally wrote this post. Since writing this post, I've sat down with a few people and wrote a fuzzer with them. This has worked very well every time I've tried it and people carried this skill away with them and use it all the time after learning how to do it. The trick is, it's extremely easy to do and I wasn't really providing any knowledge at all, other than the fact that it can be done. If you take a piece of software and use any knowledge you have whatsoever to start sending randomized inputs to it, this tends to work pretty well.

I'll also add that I'm less positive about frameworks, etc., than I used to be. I've tried a number of them out at this point and, while I see the advantages they have, the value add is much less than I would've expected back when I wrote this post and didn't have almost any software experience. There are a variety of reasons for this that are fairly long and should probably be their own post, but I think the two top ones are that almost every test framework I've tried is very slow compared to what I'd write by hand, so you lose a lot of actual test capability, and the other part is that I haven't found that the frameworks really save time on the most time consuming aspects of testing nor do I find that they're generally better at finding bugs once you account for the execution speed slowdown you get from using the framework.

Looking back on this post, I consider it a fairly bad failure in that I have a 100% success rate a converting people to being quite good at testing by sitting down with them for an hour or less and the blog post had an epsilon success rate at doing the same thing. Due to the scale of readership a blog post gets, it surely helped more people learn how to test well than I've personally helped, but given how easy it is for me to do this in person, I definitely failed to convey the key insights in this post.]

Random Test Generation, Framework

Here's an example of how simple it is to write a JavaScript tests using Scott Feeney's gentest, taken from the gentest readme.

You want to test something like

function add(x, y) {
  return x + y;
}

To check that addition commutes, so you'd write

var t = gentest.types;

forAll([t.int, t.int], 'addition is commutative', function(x, y) {
  return add(x, y) === add(y, x);
});

Instead of checking the values by hand, or writing the code to generate the values, the framework handles that and generates tests for after you when you specify the constraints. QuickCheck-like generative test frameworks tend to be simple enough that they're no harder to learn how to use than any other unit test or mocking framework.

You'll sometimes hear objections about how random testing can only find shallow bugs because random tests are too dumb to find really complex bugs. For one thing, that assumes that you don't specify constraints that allow the random generator to generate intricate test cases. But even then, this paper analyzed production failures in distributed systems, looking for "critical" bugs, bugs that either took down the entire cluster or caused data corruption, and found that 58% could be caught with very simple tests. Turns out, generating “shallow” random tests is enough to catch most production bugs. And that's on projects that are unusually serious about testing and static analysis, projects that have much better test coverage than the average project.

A specific examples of the effective of naive random testing this is the story John Hughes tells in this talk. It starts out when some people came to him with a problem.

We know there is a lurking bug somewhere in the dets code. We have got 'bad object' and 'premature eof' every other month the last year. We have not been able to track the bug down since the dets files is repaired automatically next time it is opened.

An application that ran on top of Mnesia, a distributed database, was somehow causing errors a layer below the database. There were some guesses as to the cause. Based on when they'd seen the failures, maybe something to do with rehashing something or other in files that are bigger than 1GB? But after more than a month of effort, no one was really sure what was going on.

In less than a day, with QuickCheck, they found five bugs. After fixing those bugs, they never saw the problem again. Each of the five bugs was reproducible on a database with one record, with at most five function calls. It is very common for bugs that have complex real-world manifestations to be reproducible with really simple test cases, if you know where to look.

In terms of developer time, using some kind of framework that generates random tests is a huge win over manually writing tests in a lot of circumstances, and it's so trivially easy to try out that there's basically no reason not to do it. The ROI of using more advanced techniques may or may not be worth the extra investment to learn how to implement and use them.

While dumb random testing works really well in a lot of cases, it has limits. Not all bugs are shallow. I know of a hardware company that's very good at finding deep bugs by having people with years or decades of domain knowledge write custom test generators, which then run on N-thousand machines. That works pretty well, but it requires a lot of test effort, much more than makes sense for almost any software.

The other option is to build more smarts into the program doing the test generation. There are a ridiculously large number of papers on how to do that, but very few of those papers have turned into practical, robust, software tools. The sort of simple coverage-based test generation used in AFL doesn't have that many papers on it, but it seems to be effective.

Random Test Generation, Coverage Based

If you're using an existing framework, coverage-based testing isn't much harder than using any other sort of random testing. In theory, at least. There are often a lot of knobs you can turn to adjust different settings, as well other complexity.

If you're writing a framework, there are a lot of decisions. Chief among them are what coverage metric to use and how to use that coverage metric to drive test generation.

For the first choice, which coverage metric, there are coverage metrics that are tractable, but too simplistic, like function coverage, or line coverage (a.k.a. basic block coverage). It's easy to track those, but it's also easy to get 100% coverage while missing very serious bugs. And then there are metrics that are great, but intractable, like state coverage or path coverage. Without some kind of magic to collapse equivalent paths or states together, it's impossible to track those for non-trivial programs.

For now, let's assume we're not going to use magic, and use some kind of approximation instead. Coming up with good approximations that work in practice often takes a lot of trial and error. Luckily, Michal Zalewski has experimented with a wide variety of different strategies for AFL, a testing tool that instruments code with some coverage metrics that allow the tool to generate smart tests.

AFL does the following. Each branch gets something like the following injected, which approximates tracking edges between basic blocks, i.e., which branches are taken and how many times:

cur_location = <UNIQUE_COMPILE_TIME_RANDOM_CONSTANT>;
shared_mem[prev_location ^ cur_location]++;
prev_location = cur_location >> 1;

shared_mem happens to be a 64kB array in AFL, but the size is arbitrary.

The non-lossy version of this would be to have shared_mem be a map of (prev_location, cur_location) -> int, and increment that. That would track how often each edge (prev_location, cur_location) is taken in the basic block graph.

Using a fixed sized array and xor'ing prev_location and cur_location provides lossy compression. To keep from getting too much noise out of trivial changes, for example, running a loop 1200 times vs. 1201 times, AFL only considers a bucket to have changed when it crosses one of the following boundaries: 1, 2, 3, 4, 8, 16, 32, or 128. That's one of the two things that AFL tracks to determine coverage.

The other is a global set of all (prev_location, cur_location) tuples, which makes it easy to quickly determine if a tuple/transition is new.

Roughly speaking, AFL keeps a queue of “interesting” test cases it's found and generates mutations of things in the queue to test. If something changes the coverage stat, it gets added to the queue. There's also some logic to avoid adding test cases that are too slow, and to remove test cases that are relatively uninteresting.

AFL is about 13k lines of code, so there's clearly a lot more to it than that, but, conceptually, it's pretty simple. Zalewksi explains why he's kept AFL so simple here. His comments are short enough that they're worth reading in their entirety if you're at all interested, but I'll excerpt a few bits anyway.

In the past six years or so, I've also seen a fair number of academic papers that dealt with smart fuzzing (focusing chiefly on symbolic execution) and a couple papers that discussed proof-of-concept application of genetic algorithms. I'm unconvinced how practical most of these experiments were … Effortlessly getting comparable results [from AFL] with state-of-the-art symbolic execution in equally complex software still seems fairly unlikely, and hasn't been demonstrated in practice so far.

Test Generation, Other Smarts

While Zalewski is right that it's hard to write a robust and generalizable tool that uses more intelligence, it's possible to get a lot of mileage out of domain specific tools. For example, BloomUnit, a test framework for distributed systems, helps you test non-deterministic systems by generating a subset of valid orderings, which uses a SAT solver to avoid generating equivalent re-orderings. The authors don't provide benchmark results the same way Zalewksi does with AFL, but even without benchmarks it's at least plausible that a SAT solver can be productively applied to test case generation. If nothing else, distributed system tests are often slow enough that you can do a lot of work without severely impacting test throughput.

Zalewski says “If your instrumentation makes it 10x more likely to find a bug, but runs 100x slower, your users [are] getting a bad deal.“, which is a great point -- gains in test smartness have to be balanced against losses in test throughput, but if you're testing with something like Jepsen, where your program under test actually runs on multiple machines that have to communicate with each other, the test is going to be slow enough that you can spend a lot of computation generating smarter tests before getting a 10x or 100x slowdown.

This same effect makes it difficult to port smart hardware test frameworks to software. It's not unusual for a “short” hardware test to take minutes, and for a long test to take hours or days. As a result, spending a massive amount of computation to generate more efficient tests is worth it, but naively porting a smart hardware test framework1 to software is a recipe for overly clever inefficiency.

Why Not Coverage-Based Unit Testing?

QuickCheck and the tens or hundreds of QuickCheck clones are pretty effective for random unit testing, and AFL is really amazing at coverage-based pseudo-random end-to-end test generation to find crashes and security holes. How come there isn't a tool that does coverage-based unit testing?

I often assume that if there isn't an implementation of a straightforward idea, there must be some reason, like maybe it's much harder than it sounds, but Mindy convinced me that there's often no reason something hasn't been done before, so I tried making the simplest possible toy implementation.

Before I looked at AFL's internals, I created this really dumb function to test. The function takes an array of arbitrary length as input and is supposed to return a non-zero int.

// Checks that a number has its bottom bits set
func some_filter(x int) bool {
	for i := 0; i < 16; i = i + 1 {
		if !(x&1 == 1) {
			return false
		}
		x >>= 1
	}
	return true
}

// Takes an array and returns a non-zero int
func dut(a []int) int {
	if len(a) != 4 {
		return 1
	}

	if some_filter(a[0]) {
		if some_filter(a[1]) {
			if some_filter(a[2]) {
				if some_filter(a[3]) {
					return 0 // A bug! We failed to return non-zero!
				}
				return 2
			}
			return 3
		}
		return 4
	}
	return 5
}

dut stands for device under test, a commonly used term in the hardware world. This code is deliberately contrived to make it easy for a coverage based test generator to make progress. Since the code does little work as possible per branch and per loop iteration, the coverage metric changes every time we do a bit of additional work2. It turns out, that a lot of software acts like this, despite not being deliberately built this way.

Random testing is going to have a hard time finding cases where dut incorrectly returns 0. Even if you set the correct array length, a total of 64 bits have to be set to particular values, so there's a 1 in 2^64 chance of any particular random input hitting the failure.

But a test generator that uses something like AFL's fuzzing algorithm hits this case almost immediately. Turns out, with reasonable initial inputs, it even finds a failing test case before it really does any coverage-guided test generation because the heuristics AFL uses for generating random tests generate an input that covers this case.

That brings up the question of why QuickCheck and most of its clones don't use heuristics to generate random numbers. The QuickCheck paper mentions that it uses random testing because it's nearly as good as partition testing and much easier to implement. That may be true, but it doesn't mean that generating some values using simple heuristics can't generate better results with the same amount of effort. Since Zalewski has already done the work of figuring out, empirically, what heuristics are likely to exercise more code paths, it seems like a waste to ignore that and just generate totally random values.

Whether or not it's worth it to use coverage guided generation is a bit iffier; it doesn't prove anything that a toy coverage-based unit testing prototype can find a bug in a contrived function that's amenable to coverage based testing. But that wasn't the point. The point was to see if there was some huge barrier that should prevent people from doing coverage-driven unit testing. As far as I can tell, there isn't.

It helps that the implementation of the golang is very well commented and has good facilities for manipulating go code, which makes it really easy to modify its coverage tools to generate whatever coverage metrics you want, but most languages have some kind of coverage tools that can be hacked up to provide the appropriate coverage metrics so it shouldn't be too painful for any mature language. And once you've got the coverage numbers, generating coverage-guided tests isn't much harder than generating random QuickCheck like tests. There are some cases where it's pretty difficult to generate good coverage-guided tests, like when generating functions to test a function that uses higher-order functions, but even in those cases you're no worse off than you would be with a QuickCheck clone3.

Test Time

It's possible to run software tests much more quickly than hardware tests. One side effect of that is that it's common to see people proclaim that all tests should run in time bound X, and you're doing it wrong if they don't. I've heard various values of X from 100ms to 5 minutes. Regardless of the validity of those kinds of statements, a side effect of that attitude is that people often think that running a test generator for a few hours is A LOT OF TESTING. I overheard one comment about how a particular random test tool had found basically all the bugs it could find because, after a bunch of bug fixes, it had been run for a few hours without finding any additional bugs.

And then you have hardware companies, which will dedicate thousands of machines to generating and running tests. That probably doesn't make sense for a software company, but considering the relative cost of a single machine compared to the cost of a developer, it's almost certainly worth dedicating at least one machine to generating and running tests. And for companies with their own machines, or dedicated cloud instances, generating tests on idle machines is pretty much free.

Attitude

In "Lessons Learned in Software Testing", the authors mention that QA shouldn't be expected to find all bugs and that QA shouldn't have veto power over releases because it's impossible to catch most important bugs, and thinking that QA will do so leads to sloppiness. That's a pretty common attitude on the software teams I've seen. But on hardware teams, it's expected that all “bad” bugs will be caught before the final release and QA will shoot down a release if it's been inadequately tested. Despite that, devs are pretty serious about making things testable by avoiding unnecessary complexity. If a bad bug ever escapes (e.g., the Pentium FDIV bug or the Haswell STM bug), there's a post-mortem to figure out how the test process could have gone so wrong that a significant bug escaped.

It's hard to say how much of the difference in bug count between hardware and software is attitude, and how much is due to the difference in the amount of effort expended on testing, but I think attitude is a significant factor, in addition to the difference in resources.

It affects everything4, down to what level of tests people write. There's a lot of focus on unit testing in software. In hardware, people use the term unit testing, but it usually refers to what would be called an integration test in software. It's considered too hard to thoroughly test every unit; it's much less total effort to test “units” that lie on clean API boundaries (which can be internal or external), so that's where test effort is concentrated.

This also drives test generation. If you accept that bad bugs will occur frequently, manually writing tests is ok. But if your goal is to never release a chip with a bad bug, there's no way to do that when writing tests by hand, so you'll rely on some combination of random testing, manual testing for tricky edge cases, and formal methods. If you then decide that you don't have the resources to avoid bad bugs all the time, and you have to scale things back, you'll be left with the most efficient bug finding methods, which isn't going to leave a lot of room for writing tests by hand.

Conclusion

A lot of projects could benefit from more automated testing. Basically every language has a QuickCheck-like framework available, but most projects that are amenable to QuickCheck still rely on manual tests. For all but the tiniest companies, dedicating at least one machine for that kind of testing is probably worth it.

I think QuickCheck-like frameworks could benefit from using a coverage driven approach. It's certainly easy to implement for functions that take arrays of ints, but that's also pretty much the easiest possible case for something that uses AFL-like test generation (other than, maybe, an array of bytes). It's possible that this is much harder than I think, but if so, I don't see why.

My background is primarily in hardware, so I could be totally wrong! If you have a software testing background, I'd be really interested in hearing what you think. Also, I haven't talked about the vast majority of the topics that testing covers. For example, figuring out what should be tested is really important! So is figuring out how where nasty bugs might be hiding, and having a good regression test setup. But those are pretty similar between hardware and software, so there's not much to compare and contrast.

Resources

Brian Marick on code coverage, and how it can be misused.

If a part of your test suite is weak in a way that coverage can detect, it's likely also weak in a way coverage can't detect.

I'm used to bugs being thought of in the same way -- if a test generator takes a month to catch a bug in an area, there are probably other subtle bugs in the same area, and more work needs to be done on the generator to flush them out.

Lessons Learned in Software Testing: A Context-Driven Approach, by Kaner, Bach, & Pettichord. This book is too long to excerpt, but I find it interesting because it reflects a lot of conventional wisdom.

AFL whitepaper, AFL historical notes, and AFL code tarball. All of it is really readable. One of the reasons I spent so much time looking at AFL is because of how nicely documented it is. Another reason is, of course, that it's been very effective at finding bugs on a wide variety of projects.

Update: Dmitry Vyukov's Go-fuzz, which looks like it was started a month after this post was written, uses the approach from the proof of concept in this post of combining the sort of logic seen in AFL with a QuickCheck-like framework, and has been shown to be quite effective. I believe David R. MacIver is also planning to use this approach in the next version of hypothesis.

And here's some testing related stuff of mine: everything is broken, builds are broken, julia is broken, and automated bug finding using analytics.

Terminology

I use the term random testing a lot, in a way that I'm used to using it among hardware folks. I probably mean something broader than what most software folks mean when they say random testing. For example, here's how sqlite describes their testing. There's one section on fuzz (random) testing, but it's much smaller than the sections on, say, I/O error testing or OOM testing. But as a hardware person, I'd also put I/O error testing or OOM testing under random testing because I'd expect to use randomly generated tests to test those.

Acknowledgments

I've gotten great feedback from a lot of software folks! Thanks to Leah Hanson, Mindy Preston, Allison Kaptur, Lindsey Kuper, Jamie Brandon, John Regehr, David Wragg, and Scott Feeney for providing comments/discussion/feedback.


  1. This footnote is a total tangent about a particular hardware test framework! You may want to skip this!

    SixthSense does a really good job of generating smart tests. It takes as input, some unit or collection of units (with assertions), some checks on the outputs, and some constraints on the inputs. If you don't give it any constraints, it assumes that any input is legal.

    Then it runs for a while. For units without “too much” state, it will either find a bug or tell you that it formally proved that there are no bugs. For units with “too much” state, it's still pretty good at finding bugs, using some combination of random simulation and exhaustive search.

    Combination of exhaustive search and random execution

    It can issue formal proofs for units with way too much state to brute force. How does it reduce the state space and determine what it's covered?

    I basically don't know. There are at least thirty-seven papers on SixthSense. Apparently, it uses a combination of combinational rewriting, sequential redundancy removal, min-area retiming, sequential rewriting, input reparameterization, localization, target enlargement, state-transition folding, isomoprhic property decomposition, unfolding, semi-formal search, symbolic simulation, SAT solving with BDDs, induction, interpolation, etc..

    My understanding is that SixthSense has had a multi-person team working on it for over a decade. Considering the amount of effort IBM puts into finding hardware bugs, investing tens or hundreds of person years to create a tool like SixthSense is an obvious win for them, but it's not really clear that it makes sense for any software company to make the same investment.

    Furthermore, SixthSense is really slow by software test standards. Because of the massive overhead involved in simulating hardware, SixthSense actually runs faster than a lot of simple hardware tests normally would, but running SixthSense on a single unit can easily take longer than it takes to run all of the tests on most software projects.

    [return]
  2. Among other things, it uses nested if statements instead of && because go's coverage tool doesn't create separate coverage points for && and ||. [return]
  3. Ok, you're slightly worse off due to the overhead of generating and looking at coverage stats, but that's pretty small for most non-trivial programs. [return]
  4. This is another long, skippable, footnote. This difference in attitude also changes how people try to write correct software. I've had "testing is hopelessly inadequate….(it) can be used very effectively to show the presence of bugs but never to show their absence." quoted at me tens of times by software folks, along with an argument that we have to reason our way out of having bugs. But the attitude of most hardware folks is that while the back half of that statement is true, testing (and, to some extent, formal verification) is the least bad way to assure yourself that something is probably free of bad bugs.

    This is even true not just on a macro level, but on a micro level. When I interned at Micron in 2003, I worked on flash memory. I read "the green book", and the handful of papers that were new enough that they weren't in the green book. After all that reading, it was pretty obvious that we (humans) didn't understand all of the mechanisms behind the operation and failure modes of flash memory. There were plausible theories about the details of the exact mechanisms, but proving all of them was still an open problem. Even one single bit of flash memory was beyond human understanding. And yet, we still managed to build reliable flash devices, despite building them out of incompletely understood bits, each of which would eventually fail due to some kind of random (in the quantum sense) mechanism.

    It's pretty common for engineering to advance faster than human understanding of the underlying physics. When you work with devices that aren't understood and assembly them to create products that are too complex for any human to understand or for any known technique to formally verify, there's no choice but to rely on testing. With software, people often have the impression that it's possible to avoid relying on testing because it's possible to just understand the whole thing.

    [return]
show more
Reading citations is easier than most people think
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-03-29 00:00:00 | Created: 2026-07-23 05:18:40

It's really common to see claims that some meme is backed by “studies” or “science”. But when I look at the actual studies, it usually turns out that the data are opposed to the claim. Here are the last few instances of this that I've run across.

Dunning-Kruger

A pop-sci version of Dunning-Kruger, the most common one I see cited, is that, the less someone knows about a subject, the more they think they know. Another pop-sci version is that people who know little about something overestimate their expertise because their lack of knowledge fools them into thinking that they know more than they do. The actual claim Dunning and Kruger make is much weaker than the first pop-sci claim and, IMO, the evidence is weaker than the second claim. The original paper isn't much longer than most of the incorrect pop-sci treatments of the paper, and we can get pretty good idea of the claims by looking at the four figures included in the paper. In the graphs below, “perceived ability” is a subjective self rating, and “actual ability” is the result of a test.

Dunning-Kruger graph Dunning-Kruger graph Dunning-Kruger graph Dunning-Kruger graph

In two of the four cases, there's an obvious positive correlation between perceived skill and actual skill, which is the opposite of the first pop-sci conception of Dunning-Kruger that we discussed. As for the second, we can see that people at the top end also don't rate themselves correctly, so the explanation for Dunning-Kruger's results is that people who don't know much about a subject (an easy interpretation to have of the study, given its title, Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments) is insufficient because that doesn't explain why people at the top of the charts have what appears to be, at least under the conditions of the study, a symmetrically incorrect guess about their skill level. One could argue that there's a completely different effect that just happens to cause the same, roughly linear, slope in perceived ability that people who are "unskilled and unaware of it" have. But, if there's any plausible simpler explanation, then that explanation seems overly complicated without additional evidence (which, if any exists, is not provided in the paper)1.

A plausible explanation of why perceived skill is compressed, especially at the low end, is that few people want to rate themselves as below average or as the absolute best, shrinking the scale but keeping a roughly linear fit. The crossing point of the scales is above the median, indicating that people, on average, overestimate themselves, but that's not surprising given the population tested (more on this later). In the other two cases, the correlation is very close to zero. It could be that the effect is different for different tasks, or it could be just that the sample size is small and that the differences between the different tasks is noise. It could also be that the effect comes from the specific population sampled (students at Cornell, who are probably actually above average in many respects). If you look up Dunning-Kruger on Wikipedia, it claims that a replication of Dunning-Kruger on East Asians shows the opposite result (perceived skill is lower than actual skill, and the greater the skill, the greater the difference), and that the effect is possibly just an artifact of American culture, but the citation is actually a link to an editorial which mentions a meta analysis on East Asian confidence, so that might be another example of a false citation. Or maybe it's just a link to the wrong source. In any case, the effect certainly isn't that the more people know, the less they think they know.

Income & Happiness

It's become common knowledge that money doesn't make people happy. As of this writing, a Google search for happiness income returns a knowledge card that making more than $75k/year has no impact on happiness. Other top search results claim the happiness ceiling occurs at $10k/year, $30k/year, $40k/year and $75k/year.

Not only is that wrong, the wrongness is robust across every country studied, too.

People with more income are happier

That happiness is correlated with income doesn't come from cherry picking one study. That result holds across five iterations of the World Values Survey (1981-1984, 1989-1993, 1994-1999, 2000-2004, and 2005-2009), three iterations of the Pew Global Attitudes Survey (2002, 2007, 2010), five iterations of the International Social Survey Program (1991, 1998, 2001, 2007, 2008), and a large scale Gallup survey.

The graph above has income on a log scale, if you pick a country and graph the results on a linear scale, you get something like this.

Best fit log to happiness vs. income

As with all graphs of a log function, it looks like the graph is about to level off, which results in interpretations like the following:

Distorted log graph

That's an actual graph from an article that claims that income doesn't make people happy. These vaguely log-like graphs that level off are really common. If you want to see more of these, try an image search for “happiness income”. My favorite is the one where people who make enough money literally hit the top of the scale. Apparently, there's a dollar value which not only makes you happy, it makes you as happy as it is possible for humans to be.

As with Dunning-Kruger, you can look at the graphs in the papers to see what's going on. It's a little easier to see why people would pass along the wrong story here, since it's easy to misinterpret the data when it's plotted against a linear scale, but it's still pretty easy to see what's going on by taking a peek at the actual studies.

Hedonic Adaptation & Happiness

The idea that people bounce back from setbacks (as well as positive events) and return to a fixed level of happiness entered the popular consciousness after Daniel Gilbert wrote about it in a popular book.

But even without looking at the literature on adaptation to adverse events, the previous section on wealth should cast some doubt on this. If people rebound from both bad events and good, how is it that making more money causes people to be happier?

Turns out, the idea that people adapt to negative events and return to their previous set-point is a myth. Although the exact effects vary depending on the bad event, disability2, divorce3, loss of a partner4, and unemployment5 all have long-term negative effects on happiness. Unemployment is the one event that can be undone relatively easily, but the effects persist even after people become reemployed. I'm only citing four studies here, but a meta analysis of the literature shows that the results are robust across existing studies.

The same thing applies to positive events. While it's “common knowledge” that winning the lottery doesn't make people happier, it turns out that isn't true, either.

In both cases, early cross-sectional results indicated that it's plausible that extreme events, like winning the lottery or becoming disabled, don't have long term effects on happiness. But the longitudinal studies that follow individuals and measure the happiness of the same person over time as events happen show the opposite result -- events do, in fact, affect happiness. For the most part, these aren't new results (some of the initial results predate Daniel Gilbert's book), but the older results based on less rigorous studies continue to propagate faster than the corrections.

Chess position memorization

I frequently see citations claiming that, while experts can memorize chess positions better than non-experts, the advantage completely goes away when positions are randomized. When people refer to a specific citation, it's generally Chase and Simon's 1973 paper Perception in Chess, a "classic" which has been cited a whopping 7449 times in the literature, which says:

De Groat did, however, find an intriguing difference between masters and weaker players in his short-term memory experiments. Masters showed a remarkable ability to reconstruct a chess position almost perfectly after viewing it for only 5 sec. There was a sharp dropoff in this ability for players below the master level. This result could not be attributed to the masters’ generally superior memory ability, for when chess positions were constructed by placing the same numbers of pieces randomly on the board, the masters could then do no better in reconstructing them than weaker players, Hence, the masters appear to be constrained by the same severe short-term memory limits as everyone else ( Miller, 1956), and their superior performance with "meaningful" positions must lie in their ability to perceive structure in such positions and encode them in chunks. Specifically, if a chess master can remember the location of 20 or more pieces on the board, but has space for only about five chunks in short-term memory, then each chunk must be composed of four or five pieces, organized in a single relational structure.

The paper then runs an experiment which "proves" that master-level players actually do worse than beginners when memorizing random mid-game positions even though they do much better memorizing real mid-game positions (and, in end-game positions, they do the about the same as beginners when positions are randomized). Unfortunately, the paper used an absurdly small sample size of one chess player at each skill level.

A quick search indicates that this result does not reproduce with larger sample sizes, e.g., Gobet and Simon, in "Recall of rapidly presented random chess positions is a function of skill", say

A widely cited result asserts that experts’ superiority over novices in recalling meaningful material from their domain of expertise vanishes when they are confronted with random material. A review of recent chess experiments in which random positions served as control material (presentation time between 3 and 10 sec) shows, however, that strong players generally maintain some superiority over weak players even with random positions, although the relative difference between skill levels is much smaller than with game positions. The implications of this finding for expertise in chess are discussed and the question of the recall of random material in other domains is raised.

They find this scales with skill level and, e.g., for "real" positions, 2350+ ELO players memorized ~2.2x the number of correct pieces that 1600-2000 ELO players did, but the difference was ~1.6x for random positions (these ratios are from eyeballing a graph and may be a bit off). 1.6x is smaller than 2.2x, but it's certainly not the claimed 1.0.

I've also seen this result cited to claim that it applies to other fields, but in a quick search of applying this result to other fields, results either show something similar (a smaller but still observable difference on randomized positions) or don't reproduce, e.g., McKeithen did this for programmers and found that, on trying to memorize programs, on "normal" program experts were ~2.5x better than beginners on the first trial and 3x better by the 6th trial, whereas on the "scrambled" program, experts were 3x better on the first trial and progressed to being only ~1.5x better by the 6th trial. Despite this result contradicting Chase and Simon, I've seen people cite this result to claim the same thing as Chase and Simon, presumably from people who didn't read what McKeithen actually wrote.

Type Systems

Unfortunately, false claims about studies and evidence aren't limited to pop-sci memes; they're everywhere in both software and hardware development. For example, see this comment from a Scala/FP "thought leader":

Tweet claiming that any doubt that type systems are helpful is equivalent to being an anti-vaxxer

I see something like this at least once a week. I'm picking this example not because it's particularly egregious, but because it's typical. If you follow a few of the big time FP proponents on twitter, you'll see regularly claims that there's very strong empirical evidence and extensive studies backing up the effectiveness of type systems.

However, a review of the empirical evidence shows that the evidence is mostly incomplete, and that it's equivocal where it's not incomplete. Of all the false memes, I find this one to be the hardest to understand. In the other cases, I can see a plausible mechanism by which results could be misinterpreted. “Relationship is weaker than expected” can turn into “relationship is opposite of expected”, log can look a lot like an asymptotic function, and preliminary results using inferior methods can spread faster than better conducted follow-up studies. But I'm not sure what the connection between the evidence and beliefs are in this case.

Is this preventable?

I can see why false memes might spread quickly, even when they directly contradict reliable sources. Reading papers sounds like a lot of work. It sometimes is. But it's often not. Reading a pure math paper is usually a lot of work. Reading an empirical paper to determine if the methodology is sound can be a lot of work. For example, biostatistics and econometrics papers tend to apply completely different methods, and it's a lot of work to get familiar enough with the set of methods used in any particular field to understand precisely when they're applicable and what holes they have. But reading empirical papers just to see what claims they make is usually pretty easy.

If you read the abstract and conclusion, and then skim the paper for interesting bits (graphs, tables, telling flaws in the methodology, etc.), that's enough to see if popular claims about the paper are true in most cases. In my ideal world, you could get that out of just reading the abstract, but it's not uncommon for papers to make claims in the abstract that are much stronger than the claims made in the body of the paper, so you need to at least skim the paper.

Maybe I'm being naive here, but I think a major reason behind false memes is that checking sources sounds much harder and more intimidating than it actually is. A striking example of this is when Quartz published its article on how there isn't a gender gap in tech salaries, which cited multiple sources that showed the exact opposite. Twitter was abuzz with people proclaiming that the gender gap has disappeared. When I published a post which did nothing but quote the actual cited studies, many of the same people then proclaimed that their original proclamation was mistaken. It's great that they were willing to tweet a correction6, but as far as I can tell no one actually went and read the source data, even though the graphs and tables make it immediately obvious that the author of the original Quartz article was pushing an agenda, not even with cherry picked citations, but citations that showed the opposite of their thesis.

Unfortunately, it's in the best interests of non-altruistic people who do read studies to make it seem like reading studies is difficult. For example, when I talked to the founder of a widely used pay-walled site that reviews evidence on supplements and nutrition, he claimed that it was ridiculous to think that "normal people" could interpret studies correctly and that experts are needed to read and summarize studies for the masses. But he's just a serial entrepreneur who realized that you can make a lot of money by reading studies and summarizing the results! A more general example is how people sometimes try to maintain an authoritative air by saying that you need certain credentials or markers of prestige to really read or interpret studies.

There are certainly fields where you need some background to properly interpret a study, but even then, the amount of knowledge that a degree contains is quite small and can be picked up by anyone. For example, excluding lab work (none of which contained critical knowledge for interpreting results), I was within a small constact factor of spending one hour of time per credit hour in school. At the conversion rate, an engineering degree from my alma mater costs a bit more than 100 hours and almost all non-engineering degrees land at less than 40 hours, with a large amount of overlap between them because a lot of degrees will require the same classes (e.g., calculus). Gatekeeping reading and interpreting a study on whether or not someone has a credential like a degree is absurd when someone can spend a week's worth of time gaining the knowledge that a degree offers.

If you liked this post, you'll probably enjoy this post on odd discontinuities, this post how the effect of markets on discrimination is more nuanced than it's usually made out to be and this other post discussing some common misconceptions.

2021 update

In retrospect, I think the mystery of the "type systems" example is simple: it's a different kind of fake citation than the others. In the first three examples, a clever, contrarian, but actually wrong idea got passed around. This makes sense because people love clever, contrarian, ideas and don't care very much if they're wrong, so clever, contarian, relatively frequently become viral relative to their correctness.

For the type systems example, it's just that people commonly fabricate evidence and then appeal to authority to support their position. In the post, I was confused because I couldn't see how anyone could look at the evidence and then make the claims that type system advocates do but, after reading thousands of discussions from people advocating for their pet tool/language/practice, I can see that it was naive of me to think that these advocates would even consider looking for evidence as opposed to just pretending that evidence exists without ever having looked.

Thanks to Leah Hanson, Lindsey Kuper, Jay Weisskopf, Joe Wilder, Scott Feeney, Noah Ennis, Myk Pono, Heath Borders, Nate Clark, and Mateusz Konieczny for comments/corrections/discussion.

BTW, if you're going to send me a note to tell me that I'm obviously wrong, please make sure that I'm actually wrong. In general, I get great feedback and I've learned a lot from the feedback that I've gotten, but the feedback I've gotten on this post has been unusually poor. Many people have suggested that the studies I've referenced have been debunked by some other study I clearly haven't read, but in every case so far, I've already read the other study.


  1. Dunning and Kruger claim, without what I'd consider strong evidence, that this is because people who perform well overestimate how well other people perform. While that may be true, one could also say that the explanation for people who are "unskilled" is that they underestimate how well other people perform. "Phase 2" attempts to establish that's not the case, but I don't find the argument convincing for a number of reasons. To pick one example, at the end of the section, they say "Despite seeing the superior performances of their peers, bottom-quartile participants continued to hold the mistaken impression that they had performed just fine.", but we don't know that the participants believed that they performed fine, we just know what their perceived percentile is. It's possible to believe that you're peforming poorly while also being in a high percentile (and I frequently have this belief for activties I haven't seriously practiced or studied, which seems likely to be the case for the participants of the Dunning-Kruger study who scored poorly on tasks with respect to those tassks). [return]
  2. Long-term disability is associated with lasting changes in subjective well-being: evidence from two nationally representative longitudinal studies.

    Hedonic adaptation refers to the process by which individuals return to baseline levels of happiness following a change in life circumstances. Two nationally representative panel studies (Study 1: N = 39,987; Study 2: N = 27,406) were used to investigate the extent of adaptation that occurs following the onset of a long-term disability. In Study 1, 679 participants who acquired a disability were followed for an average of 7.18 years before and 7.39 years after onset of the disability. In Study 2, 272 participants were followed for an average of 3.48 years before and 5.31 years after onset. Disability was associated with moderate to large drops in happiness (effect sizes ranged from 0.40 to 1.27 standard deviations), followed by little adaptation over time.

    [return]
  3. Time does not heal all wounds

    Cross-sectional studies show that divorced people report lower levels of life satisfaction than do married people. However, such studies cannot determine whether satisfaction actually changes following divorce. In the current study, data from an 18-year panel study of more than 30,000 Germans were used to examine reaction and adaptation to divorce. Results show that satisfaction drops as one approaches divorce and then gradually rebounds over time. However, the return to baseline is not complete. In addition, prospective analyses show that people who will divorce are less happy than those who stay married, even before either group gets married. Thus, the association between divorce and life satisfaction is due to both preexisting differences and lasting changes following the event.

    [return]
  4. Reexamining adaptation and the set point model of happiness: Reactions to changes in marital status.

    According to adaptation theory, individuals react to events but quickly adapt back to baseline levels of subjective well-being. To test this idea, the authors used data from a 15-year longitudinal study of over 24,000 individuals to examine the effects of marital transitions on life satisfaction. On average, individuals reacted to events and then adapted back toward baseline levels. However, there were substantial individual differences in this tendency. Individuals who initially reacted strongly were still far from baseline years later, and many people exhibited trajectories that were in the opposite direction to that predicted by adaptation theory. Thus, marital transitions can be associated with long-lasting changes in satisfaction, but these changes can be overlooked when only average trends are examined.

    [return]
  5. Unemployment Alters the Set-Point for Life Satisfaction

    According to set-point theories of subjective well-being, people react to events but then return to baseline levels of happiness and satisfaction over time. We tested this idea by examining reaction and adaptation to unemployment in a 15-year longitudinal study of more than 24,000 individuals living in Germany. In accordance with set-point theories, individuals reacted strongly to unemployment and then shifted back toward their baseline levels of life satisfaction. However, on average, individuals did not completely return to their former levels of satisfaction, even after they became reemployed. Furthermore, contrary to expectations from adaptation theories, people who had experienced unemployment in the past did not react any less negatively to a new bout of unemployment than did people who had not been previously unemployed. These results suggest that although life satisfaction is moderately stable over time, life events can have a strong influence on long-term levels of subjective well-being.

    [return]
  6. One thing I think it's interesting to look at is how you can see the opinions of people who are cagey about revealing their true opinions in which links they share. For example, Scott Alexander and Tyler Cowen both linked to the bogus gender gap article as something interesting to read and tend to link to things that have the same view.

    If you naively read their writing, it appears as if they're impartially looking at evidence about how the world works, which they then share with people. But when you observe that they regularly share evidence that supports one narrative, regardless of quality, and don't share evidence that supports the opposite narrative, it would appear that they have a strong opinion on the issue that they reveal via what they link to.

    [return]
show more
We used to build steel mills near cheap power. Now that's where we build datacenters
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-05-04 00:00:00 | Created: 2026-07-23 05:18:40

Why are people so concerned with hardware power consumption nowadays? Some common answers to this question are that power is critically important for phones, tablets, and laptops and that we can put more silicon on a modern chip than we can effectively use. In 2001 Patrick Gelsinger observed that if scaling continued at then-current rates, chips would have the power density of a nuclear reactor by 2005, a rocket nozzle by 2010, and the surface of the sun by 2015, implying that power density couldn't continue on its then-current path. Although this was already fairly obvious at the time, now that it's 2015, we can be extra sure that power density didn't continue to grow at unbounded rates. Anyway, the importance of portables and scaling limits are both valid and important reasons, but since they're widely discussed, I'm going to talk about an underrated reason.

People often focus on the portable market because it's cannibalizing desktop market, but that's not the only growth market -- servers are also becoming more important than desktops, and power is really important for servers. To see why power is important for servers, let's look at some calculations about how what it costs to run a datacenter from Hennessy & Patterson.

One of the issues is that you pay for power multiple times. Some power is lost at the substation, although we might not have to pay for that directly. Then we lose more storing energy in a UPS. This figure below states 6%, but smaller scale datacenters can easily lose twice that. After that, we lose more power stepping down the power to a voltage that a server can accept. That's over a 10% loss for a setup that's pretty efficient.

After that, we lose more power in the server's power supply, stepping down the voltage to levels that are useful inside a computer, which is often about another 10% loss (not pictured in the figure below).

And then once we get the power into servers, it gets turned into waste heat. To keep the servers from melting, we have to pay for power to cool them. Barroso and Holzle estimated that 30%-50% of the power drawn by a datacenter is used for chillers, and that an additional 10%-20% is for the CRAC (air circulation). That means for every watt of power used in the server, we pay for another 1-2 watts of support power.

And to actually get all this power, we have to pay for the infrastructure required to get the power into and throughout the datacenter. Hennessy & Patterson estimate that of the $90M cost of an example datacenter (just the facilities -- not the servers), 82% is associated with power and cooling1. The servers in the datacenter are estimated to only cost $70M. It's not fair to compare those numbers directly since servers need to get replaced more often than datacenters; once you take into account the cost over the entire lifetime of the datacenter, the amortized cost of power and cooling comes out to be 33% of the total cost, when servers have a 3 year lifetime and infrastructure has a 10-15 year lifetime.

If we look at all the costs, the breakdown is:

category%
server machines53
power & cooling infra20
power use13
networking8
other infra4
humans2

Power use and people are the cost of operating the datacenter (OPEX), whereas server machines, networking, power & cooling infra, and other infra are capital expenditures that are amortized across the lifetime of the datacenter (CAPEX).

Computation uses a lot of power. We used to build steel mills near cheap sources of power, but now that's where we build datacenters. As companies start considering the full cost of applications, we're seeing a lot more power optimized solutions2. Unfortunately, this is really hard. On the software side, with the exceptions of toy microbenchmark examples, best practices for writing power efficient code still aren't well understood. On the hardware side, Intel recently released a new generation of chips with significantly improved performance per watt that doesn't have much better absolute performance than the previous generation. On the hardware accelerator front, some large companies are building dedicated power-efficient hardware for specific computations. But with existing tools, hardware accelerators are costly enough that dedicated hardware only makes sense for the largest companies. There isn't an easy answer to this problem.

If you liked this post, you'd probably like chapter 6 of Hennessy & Patterson, which walks through not only the cost of power, but a number of related back of the envelope calculations relating to datacenter performance and cost.

Apologies for the quickly scribbled down post. I jotted this down shortly before signing an NDA for an interview where I expected to learn some related information and I wanted to make sure I had my thoughts written down before there was any possibility of being contaminated with information that's under NDA.

Thanks to Justin Blank for comments/corrections/discussion.


  1. Although this figure is widely cited, I'm unsure about the original source. This is probably the most suspicious figure in this entire post. Hennessy & Patterson cite “Hamilton 2010”, which appears to be a reference to this presentation. That presentation doesn't make the source of the number obvious, although this post by Hamilton does cite a reference for that figure, but the citation points to this post, which seems to be about putting datacenters in tents, not the fraction of infrastructure that's dedicated to power and cooling.

    Some other works, such as this one cite this article. However, that article doesn't directly state 82% anywhere, and it makes a number of estimates that the authors acknowledge are very rough, with qualifiers like “While, admittedly, the authors state that there is a large error band around this equation, it is very useful in capturing the magnitude of infrastructure cost.”

    [return]
  2. That being said, power isn't everything -- Reddi et al. looked at replacing conventional chips with low-power chips for a real workload (MS Bing) and found that while they got an improvement in power use per query, tail latency increased significantly, especially when servers were heavily loaded. Since Bing has a mechanism that causes query-related computations to terminate early if latency thresholds are hit, the result is both higher latency and degraded search quality. [return]
show more
Advantages of monorepos
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-05-17 00:00:00 | Created: 2026-07-23 05:18:40

Here's a conversation I keep having:

Someone: Did you hear that Facebook/Google uses a giant monorepo? WTF!
Me: Yeah! It's really convenient, don't you think?
Someone: That's THE MOST RIDICULOUS THING I've ever heard. Don't FB and Google know what a terrible idea it is to put all your code in a single repo?
Me: I think engineers at FB and Google are probably familiar with using smaller repos (doesn't Junio Hamano work at Google?), and they still prefer a single huge repo for [reasons].
Someone: Oh that does sound pretty nice. I still think it's weird but I could see why someone would want that.

“[reasons]” is pretty long, so I'm writing this down in order to avoid repeating the same conversation over and over again.

Simplified organization

With multiple repos, you typically either have one project per repo, or an umbrella of related projects per repo, but that forces you to define what a “project” is for your particular team or company, and it sometimes forces you to split and merge repos for reasons that are pure overhead. For example, having to split a project because it's too big or has too much history for your VCS is not optimal.

With a monorepo, projects can be organized and grouped together in whatever way you find to be most logically consistent, and not just because your version control system forces you to organize things in a particular way. Using a single repo also reduces overhead from managing dependencies.

A side effect of the simplified organization is that it's easier to navigate projects. The monorepos I've used let you essentially navigate as if everything is on a networked file system, re-using the idiom that's used to navigate within projects. Multi repo setups usually have two separate levels of navigation -- the filesystem idiom that's used inside projects, and then a meta-level for navigating between projects.

A side effect of that side effect is that, with monorepos, it's often the case that it's very easy to get a dev environment set up to run builds and tests. If you expect to be able to navigate between projects with the equivalent of cd, you also expect to be able to do cd; make. Since it seems weird for that to not work, it usually works, and whatever tooling effort is necessary to make it work gets done1. While it's technically possible to get that kind of ease in multiple repos, it's not as natural, which means that the necessary work isn't done as often.

Simplified dependencies

This probably goes without saying, but with multiple repos, you need to have some way of specifying and versioning dependencies between them. That sounds like it ought to be straightforward, but in practice, most solutions are cumbersome and involve a lot of overhead.

With a monorepo, it's easy to have one universal version number for all projects. Since atomic cross-project commits are possible (though these tend to split into many parts for practical reasons at large companies), the repository can always be in a consistent state -- at commit #X, all project builds should work. Dependencies still need to be specified in the build system, but whether that's a make Makefiles or bazel BUILD files, those can be checked into version control like everything else. And since there's just one version number, the Makefiles or BUILD files or whatever you choose don't need to specify version numbers.

Tooling

The simplification of navigation and dependencies makes it much easier to write tools. Instead of having tools that must understand relationships between repositories, as well as the nature of files within repositories, tools basically just need to be able to read files (including some file format that specifies dependencies between units within the repo).

This sounds like a trivial thing but, take this example by Christopher Van Arsdale on how easy builds can become:

The build system inside of Google makes it incredibly easy to build software using large modular blocks of code. You want a crawler? Add a few lines here. You need an RSS parser? Add a few more lines. A large distributed, fault tolerant datastore? Sure, add a few more lines. These are building blocks and services that are shared by many projects, and easy to integrate. … This sort of Lego-like development process does not happen as cleanly in the open source world. … As a result of this state of affairs (more speculation), there is a complexity barrier in open source that has not changed significantly in the last few years. This creates a gap between what is easily obtainable at a company like Google versus a[n] open sourced project.

The system that Arsdale is referring to is so convenient that, before it was open sourced, ex-Google engineers at Facebook and Twitter wrote their own versions of bazel in order to get the same benefits.

It's theoretically possible to create a build system that makes building anything, with any dependencies, simple without having a monorepo, but it's more effort, enough effort that I've never seen a system that does it seamlessly. Maven and sbt are pretty nice, in a way, but it's not uncommon to lose a lot of time tracking down and fixing version dependency issues. Systems like rbenv and virtualenv try to sidestep the problem, but they result in a proliferation of development environments. Using a monorepo where HEAD always points to a consistent and valid version removes the problem of tracking multiple repo versions entirely2.

Build systems aren't the only thing that benefit from running on a mono repo. Just for example, static analysis can run across project boundaries without any extra work. Many other things, like cross-project integration testing and code search are also greatly simplified.

Cross-project changes

With lots of repos, making cross-repo changes is painful. It typically involves tedious manual coordination across each repo or hack-y scripts. And even if the scripts work, there's the overhead of correctly updating cross-repo version dependencies. Refactoring an API that's used across tens of active internal projects will probably a good chunk of a day. Refactoring an API that's used across thousands of active internal projects is hopeless.

With a monorepo, you just refactor the API and all of its callers in one commit. That's not always trivial, but it's much easier than it would be with lots of small repos. I've seen APIs with thousands of usages across hundreds of projects get refactored and with a monorepo setup it's so easy that it's no one even thinks twice.

Most people now consider it absurd to use a version control system like CVS, RCS, or ClearCase, where it's impossible to do a single atomic commit across multiple files, forcing people to either manually look at timestamps and commit messages or keep meta information around to determine if some particular set of cross-file changes are “really” atomic. SVN, hg, git, etc solve the problem of atomic cross-file changes; monorepos solve the same problem across projects.

This isn't just useful for large-scale API refactorings. David Turner, who worked on twitter's migration from many repos to a monorepo gives this example of a small cross-cutting change and the overhead of having to do releases for those:

I needed to update [Project A], but to do that, I needed my colleague to fix one of its dependencies, [Project B]. The colleague, in turn, needed to fix [Project C]. If I had had to wait for C to do a release, and then B, before I could fix and deploy A, I might still be waiting. But since everything's in one repo, my colleague could make his change and commit, and then I could immediately make my change.

I guess I could do that if everything were linked by git versions, but my colleague would still have had to do two commits. And there's always the temptation to just pick a version and "stabilize" (meaning, stagnate). That's fine if you just have one project, but when you have a web of projects with interdependencies, it's not so good.

[In the other direction,] Forcing dependees to update is actually another benefit of a monorepo.

It's not just that making cross-project changes is easier, tracking them is easier, too. To do the equivalent of git bisect across multiple repos, you must be disciplined about using another tool to track meta information, and most projects simply don't do that. Even if they do, you now have two really different tools where one would have sufficed.

Ironically, there's a sense in which this benefit decreases as the company gets larger. At Twitter, which isn't exactly small, David Turner got a lot of value out of being able to ship cross-project changes. But at a Google-sized company, large commits can be large enough that it makes sense to split them into many smaller commits for a variety of reasons, which necessitates tooling that can effectively split up large conceptually atomic changes into many non-atomic commits.

Mercurial and git are awesome; it's true

The most common response I've gotten to these points is that switching to either git or hg from either CVS or SVN is a huge productivity win. That's true. But a lot of that is because git and hg are superior in multiple respects (e.g., better merging), not because having small repos is better per se.

In fact, Twitter has been patching git and Facebook has been patching Mercurial in order to support giant monorepos.

Downsides

Of course, there are downsides to using a monorepo. I'm not going to discuss them because the downsides are already widely discussed. Monorepos aren't strictly superior to manyrepos. They're not strictly worse, either. My point isn't that you should definitely switch to a monorepo; it's merely that using a monorepo isn't totally unreasonable, that folks at places like Google, Facebook, Twitter, Digital Ocean, and Etsy might have good reasons for preferring a monorepo over hundreds or thousands or tens of thousands of smaller repos.

Other discussion

Gregory Szorc. Facebook. Benjamin Pollack (one of the co-creators of Kiln). Benjamin Eberlei. Simon Stewart. Digital Ocean. Google. Twitter. thedufer. Paul Hammant.

Thanks to Kamal Marhubi, David Turner, Leah Hanson, Mindy Preston, Chris Ball, Daniel Espeset, Joe Wilder, Nicolas Grilly, Giovanni Gherdovich, Paul Hammant, Juho Snellman, and Simon Thulbourn for comments/corrections/discussion.


  1. This was even true at a hardware company I worked at which created a monorepo by versioning things in RCS over NFS. Of course you can't let people live edit files in the central repository so someone wrote a number of scripts that basically turned this into perforce. I don't recommend this system, but even with an incredibly hacktastic monorepo, you still get a lot of the upsides of a monorepo. [return]
  2. At least as long as you have some mechanism for vendoring upstream dependencies. While this works great for Google because Google writes a large fraction of the code it relies on, and has enough employees that tossing all external dependencies into the monorepo has a low cost amortized across all employees, I could imagine this advantage being too expensive to take advantage of for smaller companies. [return]
show more
A defense of boring languages
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-05-25 00:00:00 | Created: 2026-07-23 05:18:40

Boring languages are underrated. Many appear to be rated quite highly, at least if you look at market share. But even so, they're underrated. Despite the popularity of Dan McKinley's "choose boring technology" essay, boring languages are widely panned. People who use them are too (e.g., they're a target of essays by Paul Graham and Joel Spolsky, and other people have picked up a similar attitude).

A commonly used pitch for interesting languages goes something like "Sure, you can get by with writing blub for boring work, which almost all programmers do, but if you did interesting work, then you'd want to use an interesting language". My feeling is that this has it backwards. When I'm doing boring work that's basically bottlenecked on the speed at which I can write boilerplate, it feels much nicer to use an interesting language (like F#), which lets me cut down on the amount of time spent writing boilerplate. But when I'm doing interesting work, the boilerplate is a rounding error and I don't mind using a boring language like Java, even if that means a huge fraction of the code I'm writing is boilerplate.

Another common pitch, similar to the above, is that learning interesting languages will teach you new ways to think that will make you a much more effective programmer1. I can't speak for anyone else, but I found that line of reasoning compelling when I was early in my career and learned ACL2 (a Lisp), Forth, F#, etc.; enough of it stuck that I still love F#. But, despite taking the advice that "learning a wide variety of languages that support different programming paradigms will change how you think" seriously, my experience has been that the things I've learned mostly let me crank through boilerplate more efficiently. While that's pretty great when I have a boilerplate-constrained problem, when I have a hard problem, I spend so little time on that kind of stuff that the skills I learned from writing a wide variety of languages don't really help me; instead, what helps me is having domain knowledge that gives me a good lever with which I can solve the hard problem. This explains something I'd wondered about when I finished grad school and arrived in the real world: why is it that the programmers who build the systems I find most impressive typically have deep domain knowledge rather than interesting language knowledge?

Another perspective on this is Sutton's response when asked why he robbed banks, "because that's where the money is". Why do I work in boring languages? Because that's what the people I want to work with use, and what the systems I want to work on are written in. The vast majority of the systems I'm interested in are writing in boring languages. Although that technically doesn't imply that the vast majority of people I want to work with primarily use and have their language expertise in boring languages, that also turns out to be the case in practice. That means that, for greenfield work, it's also likely that the best choice will be a boring language. I think F# is great, but I wouldn't choose it over working with the people I want to work with on the problems that I want to work on.

If I look at the list of things I'm personally impressed with (things like Spanner, BigTable, Colossus, etc.), it's basically all C++, with almost all of the knockoffs in Java. When I think for a minute, the list of software written in C, C++, and Java is really pretty long. Among the transitive closure of things I use and the libraries and infrastructure used by those things, those three languages are ahead by a country mile, with PHP, Ruby, and Python rounding out the top 6. Javascript should be in there somewhere if I throw in front-end stuff, but it's so ubiquitous that making a list seems a bit pointless.

Below are some lists of software written in boring languages. These lists are long enough that I’m going to break them down into some arbitrary sublists. As is often the case, these aren’t really nice orthogonal categories and should be tags, but here we are. In the lists below, apps are categorized under “Backend” based on the main language used on the backend of a webapp. The other categories are pretty straightforward, even if their definitions a bit idiosyncratic and perhaps overly broad.

C

Operating Systems

Linux, including variants like KindleOS
BSD
Darwin (with C++)
Plan 9
Windows (kernel in C, with some C++ elsewhere)

Platforms/Infrastructure

Memcached
SQLite
nginx
Apache
DB2
PostgreSQL
Redis
Varnish
HAProxy AWS Lambda workers (with most of the surrounding infrastructure written in Java), according to @jayachdee

Desktop Apps

git
Gimp (with perl)
VLC
Qemu
OpenGL
FFmpeg
Most GNU userland tools
Most BSD userland tools
AFL
Emacs
Vim

C++

Operating Systems

BeOS/Haiku

Platforms/Infrastructure

GFS
Colossus
Ceph
Dremel
Chubby
BigTable
Spanner
MySQL
ZeroMQ
ScyllaDB
MongoDB
Mesos
JVM
.NET

Backend Apps

Google Search
PayPal
Figma (front-end written in C++ and cross-compiled to JS)

Desktop Apps

Chrome
MS Office
LibreOffice (with Java)
Evernote (originally in C#, converted to C++)
Firefox
Opera
Visual Studio (with C#)
Photoshop, Illustrator, InDesign, etc.
gcc
llvm/clang
Winamp
Z3
Most AAA games
Most pro audio and video production apps

Elsewhere

Also see this list and some of the links here.

Java

Platforms/Infrastructure

Hadoop
HDFS
Zookeeper
Presto
Cassandra
Elasticsearch
Lucene
Tomcat
Jetty

Backend Apps

Gmail
LinkedIn
Ebay
Most of Netflix
A large fraction of Amazon services

Desktop Apps

Eclipse
JetBrains IDEs
SmartGit
Minecraft

VHDL/Verilog

I'm not even going to make a list because basically every major microprocessor, NIC, switch, etc. is made in either VHDL or Verilog. For existing projects, you might say that this is because you have a large team that's familiar with some boring language, but I've worked on greenfield hardware/software co-design for deep learning and networking virtualization, both with teams that are hired from scratch for the project, and we still used Verilog, despite one of the teams having one of the larger collections of bluespec proficient hardware engineers anywhere outside of Arvind's group at MIT.

Please suggest other software that you think belongs on this list; it doesn't have to be software that I personally use. Also, does anyone know what EC2, S3, and Redshift are written in? I suspect C++, but I couldn't find a solid citation for that. This post was last updated 2021-08.

Appendix: meta

One thing I find interesting is that, in personal conversations with people, the vast majority of experienced developers I know think that most mainstream languages are basically fine, modulo performance constraints, and this is even more true among people who've built systems that are really impressive to me. Online discussion of what someone might want to learn is very different, with learning interesting/fancy languages being generally high up on people's lists. When I talk to new programmers, they're often pretty influenced by this (e.g., at Recurse Center, before ML became trendy, learning fancy languages was the most popular way people tried to become better as a programmer, and I'd say that's now #2 behind ML). While I think learning a fancy language does work for some people, I'd say that's overrated in that there are many other techniques that seem to click with at least the same proportion of people who try it that are much less popular.

A question I have is, why is online discussion about this topic so one-sided while the discussions I've had in real life are so oppositely one-sided. Of course, neither people who are loud on the internet nor people I personally know are representative samples of programmers, but I still find it interesting.

Thanks to Leah Hanson, James Porter, Waldemar Q, Nat Welch, Arjun Sreedharan, Rafa Escalante, @matt_dz, Bartlomiej Filipek, Josiah Irwin, @jayachdee, Larry Ogrondek, Miodrag Milic, Presto, Matt Godbolt, Leah Hanson, Noah Haasis, Lifan Zeng, @chozu@fedi.absturztau.be, and Josiah Irwin for comments/corrections/discussion.


  1. a variant of this argument goes beyond teaching you techniques and says that the languages you know determine what you think via the Sapir-Whorf hypothesis. I don't personally find this compelling since, when I'm solving hard problems, I don't think about things in a programming language. YMMV if you think in a programming language, but I think of an abstract solution and then translate the solution to a language, so having another language in my toolbox can, at most, help me think of better translations and save on translation. [return]
show more
The googlebot monopoly
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-05-27 00:00:00 | Created: 2026-07-23 05:18:40

TIL that Bell Labs and a whole lot of other websites block archive.org, not to mention most search engines. Turns out I have a broken website link in a GitHub repo, caused by the deletion of an old webpage. When I tried to pull the original from archive.org, I found that it's not available because Bell Labs blocks the archive.org crawler in their robots.txt:

User-agent: Googlebot
User-agent: msnbot
User-agent: LSgsa-crawler
Disallow: /RealAudio/
Disallow: /bl-traces/
Disallow: /fast-os/
Disallow: /hidden/
Disallow: /historic/
Disallow: /incoming/
Disallow: /inferno/
Disallow: /magic/
Disallow: /netlib.depend/
Disallow: /netlib/
Disallow: /p9trace/
Disallow: /plan9/sources/
Disallow: /sources/
Disallow: /tmp/
Disallow: /tripwire/
Visit-time: 0700-1200
Request-rate: 1/5
Crawl-delay: 5

User-agent: *
Disallow: /

In fact, Bell Labs not only blocks the Internet Archiver bot, it blocks all bots except for Googlebot, msnbot, and their own corporate bot. And msnbot was superseded by bingbot five years ago!

A quick search using a term that's only found at Bell Labs1, e.g., “This is a start at making available some of the material from the Tenth Edition Research Unix manual.”, reveals that bing indexes the page; either bingbot follows some msnbot rules, or that msnbot still runs independently and indexes sites like Bell Labs, which ban bingbot but not msnbot. Luckily, in this case, a lot of search engines (like Yahoo and DDG) use Bing results, so Bell Labs hasn't disappeared from the non-Google internet, but you're out of luck if you're one of the 55% of Russians who use yandex.

And all that is a relatively good case, where one non-Google crawler is allowed to operate. It's not uncommon to see robots.txt files that ban everything but Googlebot. Running a competing search engine and preventing a Google monopoly is hard enough without having sites ban non-Google bots. We don't need to make it even harder, nor do we need to accidentally2 ban the Internet Archive bot.

P.S. While you're checking that your robots.txt doesn't ban everyone but Google, consider looking at your CPUID checks to make sure that you're using feature flags instead of banning everyone but Intel and AMD.

BTW, I do think there can be legitimate reasons to block crawlers, including archive.org, but I don't think that the common default many web devs have, of blocking everything but googlebot, is really intended to block competing search engines as well as archive.org.

2021 Update: since this post was first published, archive.org started ignoring being blocked in robots.txt and archives posts where they are blocked in robots.txt. I've heard that some competing search engines do the same thing, so this mis-use of robots.txt, where sites ban everything but googlebot, is slowly making robots.txt effectively useless, much like browsers identify themselves as every browser in user-agent strings to work around sites that incorrectly block browsers they don't think are compatible.

A related thing is that sites will sometimes ban competing search engines, like Bing, in a fit of pique, which they wouldn't do to Google since Google provides too much traffic for them to be able get away with that, e.g., Discourse banned Bing because they were upset that Bing was crawling discourse at 0.46 QPS.


  1. At least until this page gets indexed. Google has a turnaround time of minutes to hours on updates to this page, which I find pretty amazing. I actually find that more impressive than seeing stuff on CNN reflected in seconds to minutes. Of course search engines are going to want to update CNN in real time. But a blog like mine? If they're crawling a niche site like mine every hour, they must also be crawling millions or tens of millions of other sites on an hourly basis and updating their index appropriately. Either that or they pull updates off of RSS, but even that requires millions or tens of millions of index updates per hour for sites with my level of traffic. [return]
  2. I don't object, in principle, to a robots.txt that prevents archive.org from archiving sites -- although the common opinion among programmers seems to be that it's a sin to block archive.org, I believe it's fine to do that if you don't want old versions of your site floating around. But it should be an informed decision, not an accident. [return]
show more
Slashdot and Sourceforge
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-05-31 00:00:00 | Created: 2026-07-23 05:18:40

If you've followed any tech news aggregator in the past week (the week of the 24th of May, 2015), you've probably seen the story about how SourceForge is taking over admin accounts for existing projects and injecting adware in installers for packages like GIMP. For anyone not following the story, SourceForge has a long history of adware laden installers, but they used to be opt-in. It appears that the process is now mandatory for many projects.

People have been wary of SourceForge ever since they added a feature to allow projects to opt-in to adware bundling, but you could at least claim that projects are doing it by choice. But now that SourceForge is clearly being malicious, they've wiped out all of the user trust that was built up over sixteen years of operating. No clueful person is going to ever download something from SourceForge again. If search engines start penalizing SourceForge for distributing adware, they won't even get traffic from people who haven't seen this story, wiping out basically all of their value.

Whenever I hear about a story like this, I'm amazed at how quickly it's possible to destroy user trust, and how much easier it is to destroy a brand than to create one. In that vein, it's funny to see Slashdot (which is owned by the same company as SourceForge) also attempting to destroy their own brand. They're the only major tech news aggregator which hasn't had a story on this, and that's because they've buried every story that someone submits. This has prompted people to start submitting comments about this on other stories.

A highly upvoted comment about SourceForge on Slashdot

I find this to be pretty incredible. How is it possible that someone, somewhere, thinks that censoring SourceForge's adware bundling on Slashdot is a net positive for Slashdot Media, the holding company that owns Slashdot and SourceForge? A quick search on either Google or Google News shows that the story has already made it to a number of major tech publications, making the value of suppressing the story nearly zero in the best case. And in the worst case, this censorship will create another Digg moment1, where readers stop trusting the moderators and move on to sites that aren't as heavily censored. There's basically no upside here and a substantial downside risk.

I can see why DHI, the holding company that owns Slashdot Media, would want to do something. Their last earnings report indicated that Slashdot Media isn't doing well, and the last thing they need is bad publicity driving people away from Slashdot:

Corporate & Other segment revenues decreased 6% to $4.5 million for the quarter ended March 31, 2015, reflecting a decline in certain revenue streams at Slashdot Media.

Compare that to their post-acquisition revenue from Q4 2012, which is the first quarter after DHI purchased Slashdot Media:

Revenues totaled $52.7 . . . including $4.7 million from the Slashdot Media acquisition

“Corporate & Other” seems to encompass more than just Slashdot Media. And despite that, as well as milking SourceForge for all of the short-term revenue they can get, all of “Corporate & Other” is doing worse than Slashdot Media alone in 20122. Their original stated plan for SourceForge and Slashdot was "to keep them pretty much the same as they are [because we] are very sensitive to not disrupting how users use them . . .", but it didn't take long for them realize that wasn't working; here's a snippet from their 2013 earnings report:

advertising revenue has declined over the past year and there is no improvement expected in the future financial performance of Slashdot Media's underlying advertising business. Therefore, $7.2 million of intangible assets and $6.3 million of goodwill related to Slashdot Media were reduced to zero.

I believe it was shortly afterwards that SourceForge started experimenting with adware/malware bundlers for projects that opted in, which somehow led us to where we are today.

I can understand the desire to do something to help Slashdot Media, but it's hard to see how permanently damaging Slashdot's reputation is going to help. As far as I can tell, they've fallen back to this classic syllogism: “We must do something. This is something. We must do this.”

Update: The Sourceforge/GIMP story is now on Slashdot, the week after it appeared everywhere else and a day after this was written, with a note about how the editor just got back from the weekend to people "freaking out that we're 'burying' this story", playing things down to make it sound like this would have been posted if it wasn't the weekend. That's not a very convincing excuse when tens of stories were posted by various editors, including the one who ended up making the Sourceforge/GIMP post, since the Sourceforge/GIMP story broke last Wednesday. The "weekend" excuse seems especially flimsy since when the Sourceforge/nmap story broke on the next Wednesday and Slashdot was under strict scrutiny for the previously delay, they were able to publish that story almost immediately on the same day, despite it having been the start of the "weekend" the last time a story broke on a Wednesday. Moreover, the Slashdot story is very careful to use terms like "modified binary" and "present third party offers" instead of "malware" or "adware".

Of course this could all just be an innocent misunderstanding, and I doubt we'll ever have enough information to know for sure either way. But Slashdot's posted excuse certainly isn't very confidence inspiring.


  1. Ironically, if you follow the link, you'll see that Slashdot's founder, CmdrTaco, is against “content getting removed for being critical of sponsors”. It's not that Slashdot wasn't biased back then; Slashdot used to be notorious for their pro-Linux pro-open source anti-MS anti-commercial bias. If you read through the comments in that link, you'll see that a lot of people lost their voting abilities after upvoting a viewpoint that runs against Slashdot's inherent bias. But it's Slashdot's bias that makes the omission of this story so remarkable. This is exactly the kind of thing Slashdot readers and moderators normally make hay about. But CmdrTaco has been gone for years, as has the old Slashdot. [return]
  2. If you want to compare YoY results, Slashdot Media pulled in $4M in Q1 2013. [return]
show more
Reading postmortems
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-08-20 00:00:00 | Created: 2026-07-23 05:18:40

I love reading postmortems. They're educational, but unlike most educational docs, they tell an entertaining story. I've spent a decent chunk of time reading postmortems at both Google and Microsoft. I haven't done any kind of formal analysis on the most common causes of bad failures (yet), but there are a handful of postmortem patterns that I keep seeing over and over again.

Error Handling

Proper error handling code is hard. Bugs in error handling code are a major cause of bad problems. This means that the probability of having sequential bugs, where an error causes buggy error handling code to run, isn't just the independent probabilities of the individual errors multiplied. It's common to have cascading failures cause a serious outage. There's a sense in which this is obvious -- error handling is generally regarded as being hard. If I mention this to people they'll tell me how obvious it is that a disproportionate number of serious postmortems come out of bad error handling and cascading failures where errors are repeatedly not handled correctly. But despite this being “obvious”, it's not so obvious that sufficient test and static analysis effort are devoted to making sure that error handling works.

For more on this, Ding Yuan et al. have a great paper and talk: Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems. The paper is basically what it says on the tin. The authors define a critical failure as something that can take down a whole cluster or cause data corruption, and then look at a couple hundred bugs in Cassandra, HBase, HDFS, MapReduce, and Redis, to find 48 critical failures. They then look at the causes of those failures and find that most bugs were due to bad error handling. 92% of those failures are actually from errors that are handled incorrectly.

Graphic of previous paragraph

Drilling down further, 25% of bugs are from simply ignoring an error, 8% are from catching the wrong exception, 2% are from incomplete TODOs, and another 23% are "easily detectable", which are defined as cases where “the error handling logic of a non-fatal error was so wrong that any statement coverage testing or more careful code reviews by the developers would have caught the bugs”. By the way, this is one reason I don't mind Go style error handling, despite the common complaint that the error checking code is cluttering up the main code path. If you care about building robust systems, the error checking code is the main code!

The full paper has a lot of gems that that I mostly won't describe here. For example, they explain the unreasonable effectiveness of Jepsen (98% of critical failures can be reproduced in a 3 node cluster). They also dig into what percentage of failures are non-deterministic (26% of their sample), as well as the causes of non-determinism, and create a static analysis tool that can catch many common error-caused failures.

Configuration

Configuration bugs, not code bugs, are the most common cause I've seen of really bad outages. When I looked at publicly available postmortems, searching for “global outage postmortem” returned about 50% outages caused by configuration changes. Publicly available postmortems aren't a representative sample of all outages, but a random sampling of postmortem databases also reveals that config changes are responsible for a disproportionate fraction of extremely bad outages. As with error handling, I'm often told that it's obvious that config changes are scary, but it's not so obvious that most companies test and stage config changes like they do code changes.

Except in extreme emergencies, risky code changes are basically never simultaneously pushed out to all machines because of the risk of taking down a service company-wide. But it seems that every company has to learn the hard way that seemingly benign config changes can also cause a company-wide service outage. For example, this was the cause of the infamous November 2014 Azure outage. I don't mean to pick on MS here; their major competitors have also had serious outages for similar reasons, and they've all put processes into place to reduce the risk of that sort of outage happening again.

I don't mean to pick on large cloud companies, either. If anything, the situation there is better than at most startups, even very well funded ones. Most of the “unicorn” startups that I know of don't have a proper testing/staging environment that lets them test risky config changes. I can understand why -- it's often hard to set up a good QA environment that mirrors prod well enough that config changes can get tested, and like driving without a seatbelt, nothing bad happens the vast majority of the time. If I had to make my own seatbelt before driving my car, I might not drive with a seatbelt either. Then again, if driving without a seatbelt were as scary as making config changes, I might consider it.

Back in 1985, Jim Gray observed that "operator actions, system configuration, and system maintenance was the main source of failures -- 42%". Since then, there have been a variety of studies that have found similar results. For example, Rabkin and Katz found the following causes for failures:

Hardware

Basically every part of a machine can fail. Many components can also cause data corruption, often at rates that are much higher than advertised. For example, Schroeder, Pinherio, and Weber found DRAM error rates were more than an order of magnitude worse than advertised. The number of silent errors is staggering, and this actually caused problems for Google back before they switched to ECC RAM. Even with error detecting hardware, things can go wrong; relying on Ethernet checksums to protect against errors is unsafe and I've personally seen malformed packets get passed through as valid packets. At scale, you can run into more undetected errors than you expect, if you expect hardware checks to catch hardware data corruption.

Failover from bad components can also fail. This AWS failure tells a typical story. Despite taking reasonable sounding measures to regularly test the generator power failover process, a substantial fraction of AWS East went down when a storm took out power and a set of backup generators failed to correctly provide power when loaded.

Humans

This section should probably be called process error and not human error since I consider having humans in a position where they can accidentally cause a catastrophic failure to be a process bug. It's generally accepted that, if you're running large scale systems, you have to have systems that are robust to hardware failures. If you do the math on how often machines die, it's obvious that systems that aren't robust to hardware failure cannot be reliable. But humans are even more error prone than machines. Don't get me wrong, I like humans. Some of my best friends are human. But if you repeatedly put a human in a position where they can cause a catastrophic failure, you'll eventually get a catastrophe. And yet, the following pattern is still quite common:

Oh, we're about to do a risky thing! Ok, let's have humans be VERY CAREFUL about executing the risky operation. Oops! We now have a global outage.

Postmortems that start with “Because this was a high risk operation, foobar high risk protocol was used” are ubiquitous enough that I now think of extra human-operated steps that are done to mitigate human risk as an ops smell. Some common protocols are having multiple people watch or confirm the operation, or having ops people standing by in case of disaster. Those are reasonable things to do, and they mitigate risk to some extent, but in many postmortems I've read, automation could have reduced the risk a lot more or removed it entirely. There are a lot of cases where the outage happened because a human was expected to flawlessly execute a series of instructions and failed to do so. That's exactly the kind of thing that programs are good at! In other cases, a human is expected to perform manual error checking. That's sometimes harder to automate, and a less obvious win (since a human might catch an error case that the program misses), but in most cases I've seen it's still a net win to automate that sort of thing.

In an IDC survey, respondents voted human error as the most troublesome cause of problems in the datacenter.

One thing I find interesting is how underrepresented human error seems to be in public postmortems. As far as I can tell, Google and MS both have substantially more automation than most companies, so I'd expect their postmortem databases to contain proportionally fewer human error caused outages than I see in public postmortems, but in fact it's the opposite. My guess is that's because companies are less likely to write up public postmortems when the root cause was human error enabled by risky manual procedures. A prima facie plausible alternate reason is that improved technology actually increases the fraction of problems caused by humans, which is true in some industries, like flying. I suspect that's not the case here due to the sheer number of manual operations done at a lot of companies, but there's no way to tell for sure without getting access to the postmortem databases at multiple companies. If any company wants to enable this analysis (and others) to be done (possibly anonymized), please get in touch.

Monitoring / Alerting

The lack of proper monitoring is never the sole cause of a problem, but it's often a serious contributing factor. As is the case for human errors, these seem underrepresented in public postmortems. When I talk to folks at other companies about their worst near disasters, a large fraction of them come from not having the right sort of alerting set up. They're often saved having a disaster bad enough to require a public postmortem by some sort of ops heroism, but heroism isn't a scalable solution.

Sometimes, those near disasters are caused by subtle coding bugs, which is understandable. But more often, it's due to blatant process bugs, like not having a clear escalation path for an entire class of failures, causing the wrong team to debug an issue for half a day, or not having a backup on-call, causing a system to lose or corrupt data for hours before anyone notices when (inevitably) the on-call person doesn't notice that something's going wrong.

The Northeast blackout of 2003 is a great example of this. It could have been a minor outage, or even just a minor service degradation, but (among other things) a series of missed alerts caused it to become one of the worst power outages ever.

Not a Conclusion

This is where the conclusion's supposed to be, but I'd really like to do some serious data analysis before writing some kind of conclusion or call to action. What should I look for? What other major classes of common errors should I consider? These aren't rhetorical questions and I'm genuinely interested in hearing about other categories I should think about. Feel free to ping me here. I'm also trying to collect public postmortems here.

One day, I'll get around to the serious analysis, but even without going through and classifying thousands of postmortems, I'll probably do a few things differently as a result of having read a bunch of these. I'll spend relatively more time during my code reviews on errors and error handling code, and relatively less time on the happy path. I'll also spend more time checking for and trying to convince people to fix “obvious” process bugs.

One of the things I find to be curious about these failure modes is that when I talked about what I found with other folks, at least one person told me that each process issue I found was obvious. But these “obvious” things still cause a lot of failures. In one case, someone told me that what I was telling them was obvious at pretty much the same time their company was having a global outage of a multi-billion dollar service, caused by the exact thing we were talking about. Just because something is obvious doesn't mean it's being done.

Elsewhere

Richard Cook's How Complex Systems Fail takes a more general approach; his work inspired The Checklist Manifesto, which has saved lives.

Allspaw and Robbin's Web Operations: Keeping the Data on Time talks about this sort of thing in the context of web apps. Allspaw also has a nice post about some related literature from other fields.

In areas that are a bit closer to what I'm used to, there's a long history of studying the causes of failures. Some highlights include Jim Gray's Why Do Computers Stop and What Can Be Done About It? (1985), Oppenheimer et. al's Why Do Internet Services Fail, and What Can Be Done About It? (2003), Nagaraja et. al's Understanding and Dealing with Operator Mistakes in Internet Services (2004), part of Barroso et. al's The Datacenter as a Computer (2009), and Rabkin and Katz's How Hadoop Clusters Break (2013), and Xu et. al's Do Not Blame Users for Misconfigurations.

There's also a long history of trying to understand aircraft reliability, and the story of how processes have changed over the decades is fascinating, although I'm not sure how to generalize those lessons.

Just as an aside, I find it interesting how hard it's been to eke out extra uptime and reliability. In 1974, Ritchie and Thompson wrote about a system "costing as little as $40,000" with 98% uptime. A decade later, Jim Gray uses 99.6% uptime as a reasonably good benchmark. We can do much better than that now, but the level of complexity required to do it is staggering.

Acknowledgments

Thanks to Leah Hanson, Anonymous, Marek Majkowski, Nat Welch, Joe Wilder, and Julia Hansbrough for providing comments on a draft of this. Anonymous, if you prefer to not be anonymous, send me a message on Zulip. For anyone keeping score, that's three folks from Google, one person from Cloudflare, and one anonymous commenter. I'm always open to comments/criticism, but I'd be especially interested in comments from folks who work at companies with less scale. Do my impressions generalize?

Thanks to gwern and Dan Reif for taking me up on this and finding some bugs in this post.

show more
Steve Yegge's prediction record
Feed: https://danluu.com/atom.xml (https://danluu.com/atom.xml)
Published: 2015-08-31 00:00:00 | Created: 2026-07-23 05:18:40

I try to avoid making predictions1. It's a no-win proposition: if you're right, hindsight bias makes it look like you're pointing out the obvious. And most predictions are wrong. Every once in a while when someone does a review of predictions from pundits, they're almost always wrong at least as much as you'd expect from random chance, and then hindsight bias makes each prediction look hilariously bad.

But, occasionally, you run into someone who makes pretty solid non-obvious predictions. I was re-reading some of Steve Yegge's old stuff and it turns out that he's one of those people.

His most famous prediction is probably the rise of JavaScript. This now seems incredibly obvious in hindsight, so much so that the future laid out in Gary Bernhardt's Birth and Death of JavaScript seems at least a little plausible. But you can see how non-obvious Steve's prediction was at the time by reading both the comments on his blog, and comments from HN, reddit, and the other usual suspects.

Steve was also crazy-brave enough to post ten predictions about the future in 2004. He says “Most of them are probably wrong. The point of the exercise is the exercise itself, not in what results.”, but the predictions are actually pretty reasonable.

Prediction #1: XML databases will surpass relational databases in popularity by 2011

2011 might have been slightly too early and JSON isn't exactly XML, but NoSQL databases have done really well for pretty much the reason given in the prediction, “Nobody likes to do O/R mapping; everyone just wants a solution.”. Sure, Mongo may lose your data, but it's easy to set up and use.

Prediction #2: Someone will make a lot of money by hosting open-source web applications

This depends on what you mean by “a lot”, but this seems basically correct.

We're rapidly entering the age of hosted web services, and big companies are taking advantage of their scalable infrastructure to host data and computing for companies without that expertise.

For reasons that seem baffling in retrospect, Amazon understood this long before any of its major competitors and was able to get a huge head start on everybody else. Azure didn't get started until 2009, and Google didn't get serious about public cloud hosting until even later.

Now that everyone's realized what Steve predicted in 2004, it seems like every company is trying to spin up a public cloud offering, but the market is really competitive and hiring has become extremely difficult. Despite giving out a large number of offers an integer multiple above market rates, Alibaba still hasn't managed to put together a team that's been able to assemble a competitive public cloud, and companies that are trying to get into the game now without as much cash to burn as Alibaba are having an even harder time.

For both bug databases and source-control systems, the obstacle to outsourcing them is trust. I think most companies would love it if they didn't have to pay someone to administer Bugzilla, Subversion, Twiki, etc. Heck, they'd probably like someone to outsource their email, too.

A lot of companies have moved both issue tracking and source-control to GitHub or one of its competitors, and even more have moved if you just count source-control. Hosting your own email is also a thing of the past for all but the most paranoid (or most bogged down in legal compliance issues).

Prediction #3: Multi-threaded programming will fall out of favor by 2012

Hard to say if this is right or not. Depends on who you ask. This seems basically right for applications that don't need the absolute best levels for performance, though.

In the past, oh, 20 years since they invented threads, lots of new, safer models have arrived on the scene. Since 98% of programmers consider safety to be unmanly, the alternative models (e.g. CSP, fork/join tasks and lightweight threads, coroutines, Erlang-style message-passing, and other event-based programming models) have largely been ignored by the masses, including me.

Shared memory concurrency is still where it's at for really high performance programs, but Go has popularized CSP; actors and futures are both “popular” on the JVM; etc.

Prediction #4: Java's "market share" on the JVM will drop below 50% by 2010

I don't think this was right in 2010, or even now, although we're moving in the right direction. There's a massive amount of dark matter -- programmers who do business logic and don't blog or give talks -- that makes this prediction unlikely to come true in the near future.

It's impossible to accurately measure market share, but basically every language ranking you can find will put Java in the top 3, with Scala and Clojure not even in the top 10. Given the near power-law distribution of measured language usage, Java must still be above 90% share (and that's probably a gross underestimate).

Not even close. Depending on how you measure this, Clojure might be in the top 20 (it is if you believe the Redmonk rankings), but it's hard to see it making it into the top 10 in this decade. As with the previous prediction, there's just way too much inertia here. Breaking into the top 10 means joining the ranks of Java, JS, PHP, Python, Ruby, C, C++, and C#. Clojure just isn't boring enough. C# was able to sneak in by pretending to boring, but Clojure's got no hope of doing that and there isn't really another Dylan on the horizon.

Prediction #6: A new internet community-hangout will appear. One that you and I will frequent

This seems basically right, at least for most values of “you”.

Wikis, newsgroups, mailing lists, bulletin boards, forums, commentable blogs — they're all bullshit. Home pages are bullshit. People want to socialize, and create content, and compete lightly with each other at different things, and learn things, and be entertained: all in the same place, all from their couch. Whoever solves this — i.e. whoever creates AOL for real people, or whatever the heck this thing turns out to be — is going to be really, really rich.

Facebook was founded the year that was written. Zuckerberg is indeed really, really, rich.

Prediction #7: The mobile/wireless/handheld market is still at least 5 years out

Five years from Steve's prediction would have been 2009. Although the iPhone was released in 2007, it was a while before sales really took off. In 2009, the majority of phones were feature phones, and Android was barely off the ground.

Symbian is in the lead until Q4 2010!

Note that this graph only runs until 2013; if you graph things up to 2015 on a linear scale, sales are so low in 2009 that you basically can't even see what's going on.

Prediction #8: Someday I will voluntarily pay Google for one of their services

It's hard to tell if this is correct (Steve, feel free to let me know), but it seems true in spirit. Google has more and more services that they charge for, and they're even experimenting with letting people pay to avoid seeing ads.

Prediction #9: Apple's laptop sales will exceed those of HP/Compaq, IBM, Dell and Gateway combined by 2010

If you include tablets, Apple hit #1 in the market by 2010, but I don't think they do better than all of the old workhorses combined. Again, this seems to underestimate the effect of dark matter, in this case, people buying laptops for boring reasons, e.g., corporate buyers and normal folks who want something under Apple's price range.

Prediction #10: In five years' time, most programmers will still be average

More of a throwaway witticism than a prediction, but sure.

That's a pretty good set of predictions for 2004. With the exception of the bit about Lisp, all of the predictions seem directionally correct; the misses are mostly caused by underestimating the sheer about of inertia it takes for a young/new solution to take over.

Steve also has a number of posts that aren't explicitly about predictions that, nevertheless, make pretty solid predictions about how things are today, written way back in 2004. There's It's Not Software, which was years ahead of its time about how people write “software”, how writing server apps is really different from writing shrinkwrap software in a way that obsoletes a lot of previously solid advice, like Joel's dictum against rewrites, as well as how service oriented architectures look; the Google at Delphi (again from 2004) correctly predicts the importance of ML and AI as well as Google's very heavy investment in ML; an old interview where he predicts "web application programming is gradually going to become the most important client-side programming out there. I think it will mostly obsolete all other client-side toolkits: GTK, Java Swing/SWT, Qt, and of course all the platform-specific ones like Cocoa and Win32/MFC/"; etc. A number of Steve's internal Google blog posts also make interesting predictions, but AFAIK those are confidential. Of course these all these things seem obvious in retrospect, but that's just part of Steve's plan to pass as a normal human being.

In a relatively recent post, Steve throws Jeff Bezos under the bus, exposing him as one of a number of “hyper-intelligent aliens with a tangential interest in human affairs”. While the crowd focuses on Jeff, Steve is able to sneak out the back. But we're onto you, Steve.

Thanks to Leah Hanson, Chris Ball, Mindy Preston, and Paul Gross for comments/corrections.


  1. When asked about a past prediction of his, Peter Thiel commented that writing is dangerous and mentioned that a professor once told him that writing a book is more dangerous than having a child -- you can always disown a child, but there's nothing you can do to disown a book.

    The only prediction I can recall publicly making is that I've been on the record for at least five years saying that, despite the hype, ARM isn't going completely crush Intel in the near future, but that seems so obvious that it's not even worth calling it a prediction. Then again, this was a minority opinion up until pretty recently, so maybe it's not that obvious.

    I've also correctly predicted the failure of a number of chip startups, but since the vast majority of startups fail, that's expected. Predicting successes is much more interesting, and my record there is decidedly mixed. Based purely on who was involved, I thought that SiByte, Alchemy, and PA Semi were good bets. Of those, SiByte was a solid success, Alchemy didn't work out, and PA Semi was maybe break-even.

    [return]
show more
Page 652 of 1014 (50668 total items)