The Persistent Myth of Market Efficiency
There’s a popular cocktail party version of the efficient markets hypothesis that goes something like: “Markets enforce efficiency, so it’s not possible for a company to have a major inefficiency and survive.” We’ve previously discussed this idea in the context of tech hiring, where Marc Andreessen argued that Silicon Valley companies simply can’t be discriminatory because they are “dying for talent.” The logic is that if a company were doing something obviously wrong, a competitor would have already outcompeted them by doing it right.
That line of reasoning has a vague plausibility, which is why it keeps coming up in casual debate. One person points out a glaring corporate inefficiency or product flaw, and another responds that if it were so obvious, someone inside the company would have fixed it, or an outside firm would have swooped in to win on the merits. Talking purely abstractly, it’s hard to settle. But looking at specific cases—like the hiring examples we’ve covered before—shows that inefficiencies can persist for decades despite abstract arguments to the contrary.
The same pattern holds when buying products and services at a personal level. Most people who have audited the work of hired professionals—whether for home renovation or accounting—have found grievous errors. It’s possible to find people who don’t do shoddy work, but it’s generally difficult for a non-expert to determine who those people are in advance. Paying more doesn’t reliably solve the problem. My friends and colleagues who’ve gone with large, brand-name accounting firms have paid much more than those using small local accountants and gotten a higher error rate. Trying expensive local accountants hasn’t fared much better. The good accountants are typically somewhat expensive, but they’re generally not charging the highest rates, and only a small percentage of mid-priced accountants are good.
In many markets, consumers are uninformed, and it’s fairly difficult to figure out which products are even half decent. When people happen to choose a product that fits them, it’s often for the wrong reasons. In my social circles, there have been two waves of people migrating from iPhones to Android phones over the past few years. Both waves happened due to Apple PR snafus, which caused a lot of people to think iPhones were terrible at something when, in fact, they were better at that thing than Android phones. Many switchers did end up with a device better for them—not because their reasoning was sound, but because their prior iPhone decision had been influenced by good Apple PR, and their errors cancelled out.
In capital markets, the efficient market hypothesis doesn’t require everyone to be informed. It only takes enough informed participants to exploit an inefficiency until it disappears. It’s a truism that published results about market inefficiencies stop being true the moment they’re published. But with labor markets, even when firms can take advantage of mispriced talent, inefficiencies can persist. Alan Greenspan famously hired women economists because they were less expensive than men, noting that other employers’ discrimination made it “great business sense.” But individual firms exploiting mispriced labor have limited demand. Inefficiencies persisted for decades because the firms acting on “all available information” didn’t buy enough labor to move the price to where it would be if most firms were acting rationally.
With products and services, there’s even less of a mechanism to exploit inefficiency directly. If you observe that people are switching from iPhones to Android based on false beliefs about planned obsolescence—when Android devices generally become obsolete more quickly due to shorter update support and slower processors—you can’t really make money off that observation. This is unlike a mispriced asset where you could buy derivatives to profit. There’s no direct arbitrage, and sometimes there isn’t even a mechanism to make any money at all.
The Expert Problem
A common suggestion for solving the problem of not knowing what product is good is to ask an expert. But this often fails as well. A friend of mine had trouble sleeping because his window air conditioner was loud and would wake him up when it turned on. When he asked a trusted friend who works on air conditioners whether a newer model would be quieter, his friend said, “No; air conditioners are basically all the same.”
Anyone who has compared consumer products with motors over time knows this is false—engineers have gotten much better at producing quieter devices while holding power and cost constant. My friend eventually bought a newer, quieter air conditioner, which solved his problem. But he suffered longer than he needed to because he assumed someone whose job it is to work on air conditioners would give him non-terrible advice about them. The catch is that if he’d had the expertise to know his friend was wrong, he wouldn’t have needed advice in the first place.
This problem exists at the firm level too, and it’s often worse because B2B markets tend to be thinner, with fewer products available and opaque “call us” pricing. A common piece of advice is that firms should focus on their “core competencies” and outsource everything else. But mid-sized tech companies often need in-house expertise far outside their apparent core competency. Every social media company, for example, needs kernel expertise. In principle, firms can outsource this kind of work, but people I know who’ve relied on consultants or vendor support contracts have been very unhappy with the results—both in absolute terms and for the money. Support frequently doesn’t come up with a satisfactory resolution in weeks or months, even for problems a good engineer could solve in days. And despite engineers being expensive, large support contracts can cost more than an engineer while delivering worse service.
Why “Buy” So Often Means “Broken”
Ben Kuhn, the CTO of Wave, has documented some of the issues his company ran into with vendors, and he now believes one of his big mistakes as CTO was not putting much more effort into vendor selection—even when the decision appeared to be a slam dunk—and not considering moving systems to custom in-house versions sooner.
We tried “buy” instead of “build” for a product that syncs data from Postgres to Snowflake. Syncing from Postgres is the main offering from a leading data sync company, and we found that it would lose data, duplicate data, and corrupt data. Digging in, it turned out the product has a design that relies on the data source being able to seek backwards on its changelog. But Postgres throws changelogs away once they’re consumed, so the data source can’t support that operation. When the product attempts it and fails, the sync gets stuck, needing manual intervention from the vendor’s operator, or causes data loss.
Recovery is possible via a full resync, but the sync product tops out at 5MB/s for reasons that appear unknown to the vendor, so a full resync can take days even on modest databases. Resyncs also silently drop and corrupt data, so multiple cycles of resync followed by integrity checks are sometimes necessary, taking weeks. Despite being widely recommended and the leading product in the space, this product has design flaws that mean it literally cannot work.
This isn’t so different from Mongo or other products with fundamental design flaws that caused severe data loss. The main difference is that, in most areas, there isn’t a Kyle Kingsbury who spends years publishing tests and responding to bogus correctness claims until PR pressure forces companies to take correctness seriously. Without that pressure, most software products basically don’t work. At our scale, there are many things we won’t build any time soon—like CPUs—but for many things where the received wisdom is to “buy,” “build” seems like a reasonable option.
Even CPUs aren’t as outlandish as they used to be. Fifteen years ago, high-performance CPU design was a canonical example of something it would be absurd for even the largest software company to attempt in-house. But Apple and Amazon have been able to produce best-in-class chips on the dimensions they optimize for, for predictable reasons. Even when there’s an obvious opportunity, factors like network effects, high fixed costs, and the ability of incumbents to suppress competitors often prevent anyone from taking it. The teams Apple and Amazon bought were from failed startups that couldn’t survive independently.
Shipping as a Cautionary Tale
This isn’t just a tech company issue. Any company that wants to mail items to customers has to either implement shipping themselves or deal with the fallout of unreliable shipping services. As a user, whether packages get delivered depends a lot on where you live and what kind of building you live in.
When I’ve lived in a house, packages have usually arrived regardless of the shipper, though they’ve often been late. In apartment buildings, some just don’t get deliveries from certain services. UPS and FedEx often won’t even attempt delivery—they’ll put notices on the building door falsely indicating the person wasn’t home, forcing residents to pick packages up at a remote location.
For a while, I lived in a city where Amazon used third-party couriers for same-day delivery. These services were famous for marking things as delivered without actually delivering them for days, making “same-day” shipping slower than next-day. When I contacted Amazon support about a package marked delivered but not received, support told me to wait three days because couriers often mark packages as delivered without delivering them, but often deliver within a few days. Amazon knew their courier service wasn’t actually attempting deliveries, and their only short-term mitigation was a script telling customers not to expect packages when they were marked as delivered.
Amazon eventually solved the problem by building their own delivery service. My local grocery store tried outsourcing to DoorDash—I received groceries 2 out of 3 times, which is well below an acceptable hit rate, and the successful orders sometimes contained items I didn’t order or pay for, presumably meant for other customers. At scale, there’s no commercial service you can reliably pay for delivery. If you want a service that works, you’re generally on the hook for building it yourself.
The Cost of Going In-House
Building instead of buying is a huge drag on productivity, especially for smaller companies. But it’s often worth it. At a small chip startup I worked at, we had in-house capability for end-to-end chip processing, excluding having our own fabs. When the first wafer of a new design came back from a fab, we’d fly it in and use our own wafer saw to cut it into individual chips so testing could start immediately.
This was often considered absurd—after all, the saw and expertise would be idle over 99% of the time. Having full-time equipment and expertise you rarely use is a textbook case for outsourcing. But when you price out competent people plus equipment availability, it’s cheaper in-house even at fairly low volumes, with the equipment idle 99% of the time. More importantly, you get much better service with faster turnaround, letting you ship at a higher cadence. Companies that contract this kind of work out get slower, less reliable service at a higher cost.
Likewise with chip software tooling. Despite it being standard to outsource to large EDA vendors, we got a lot of mileage from custom tools generally maintained by one person. Most simulator cycles ran on a custom simulator maintained by a single engineer, saving millions a year in licensing fees. You might think that if one person can create a tool worth millions a year, our competitors would do the same. But they mostly didn’t.
Joel Spolsky has an old post capturing the attitude: “Find the dependencies—and eliminate them. When you’re working on a really good team, everybody else’s code is bug-infested garbage.” We had a similar attitude, though more humble—we assumed we could produce something comparable to what we could buy for a tenth of the cost. From talking to our competitors, there was a cultural difference. It simply didn’t occur to them that they didn’t have to buy into the standard business logic of focusing on core competencies, that they could reason through each case on its merits rather than outsourcing their thinking to a pithy saying.
I once watched a company undergo this cultural shift from the inside. Leadership decided to focus on core competencies, abandoning custom software for infrastructure. This triggered large migrations from custom internal systems to SaaS and open source software. Most people bought the party line and pushed for migrations regardless of specifics. A few “unusually unreasonable people” tried to reason through particular cases on their merits, but the migrations were driven by cultural conviction rather than technical reasoning.
One team moved to an open source system “to save money,” though the new system was quite obviously less efficient and required much higher capital and operating expenses. The expected cost savings from shrinking the team were dominated by increased operational costs, and the system’s complexity meant the team grew instead of shrank. There were cases where migration genuinely made sense, but the stated reasons were usually unrelated to the actual justifications.
The pervasiveness of such decisions—technical decisions made without serious technical consideration—is a major reason that selection pressure on companies to make good products is so weak. There is some pressure, but it’s noisy enough that successful companies often route around making products that work. Mongo’s decision to loudly repeat demonstrably bogus performance and correctness claims was, from a business standpoint, superior to focusing on actual correctness—they outcompeted companies that made the mistake of devoting serious resources to those things.
The Volvo Problem
At the individual level, it takes an unusually unreasonable person to push for doing the right thing. Yossi Kreinin calls these people “insane employees”—those who find problems a personal offense and are willing to risk annoying management by fighting to fix them, spending political capital on work that management doesn’t value. Kyle Kingsbury is a prime example. At the rates Jepsen charges now, he can earn what a senior developer at BigCo makes, but only after years of working long hours at below-market rates, refuting FUD from critics who cast aspersions on his character and trashed him publicly. A system that requires someone like Kyle to take a stand before companies will invest in actual correctness is going to produce a lot of products that are merely good at marketing correctness.
At the firm level, it often takes an unusually unreasonable firm to produce a truly great product rather than one merely marketed as great. Volvo is the one car manufacturer that seemed to try for structural safety beyond what crash tests can demonstrate. That commitment fared so poorly that Volvo was forced upmarket and became a niche luxury brand—safety isn’t something consumers are really interested in, despite car accidents being a leading cause of death. When Ford acquired Volvo, they moved the cars to the shared C1 platform, which didn’t fare particularly well in crash tests. Since Geely acquired Volvo, it’s unclear if they’ll maintain the commitment. If Geely doesn’t, it may not be possible to buy a modern car designed to be genuinely safe.
Economists have a term for markets where buyers can’t tell the difference between good products and lemons: a “market for lemons.” There’s debate over whether cars actually are such a market, because lemon laws and low rates of defective vehicles suggest otherwise. But looking at whether people occasionally buy defective cars misses the forest for the trees. There’s maybe one car manufacturer that seriously tries to make a structurally safe car beyond what standards bodies test, because consumers can’t tell the difference. That’s a market for lemons, as is nearly every other consumer and B2B market.
Building instead of buying isn’t a panacea. I’ve seen internal designs just as broken as the data sync product. When you see such a design, a decent number of people usually explained why it could never work during the design phase and were ignored. A dysfunctional team in a dysfunctional organization can easily create products that don’t work. Steve Jobs observed that companies with monopolies, like IBM or Xerox, get run by sales and marketing people who drive product people out, and the companies forget what it means to make great products. The same dynamic applies at the team level when a company tries to avoid duplicate effort by giving one team a monopoly over a product category.
Culture and Trust
Much of this comes down to whether you operate in a culture where you can trust another firm’s promise. If you’re in a society where firms push you to the letter of the law regarding whatever contract you’ve negotiated, it’s frequently not worth negotiating a contract that would give you service even half as good as you’d get in-house. Companies often try to sneak in terms that make contracts meaningless. Legal enforcement is expensive—in cases I know of where vendors regularly violated support SLAs, the resolution was contract termination rather than legal action.
Lack of trust across companies can also hamstring companies internally. At one company, I was griping to a director that a VP had broken a promise and that we were losing people for similar reasons. The director’s response was that there was no way the VP had made a promise—“unless you get it in a contract, it wasn’t a promise.” The rate at which VPs lied was high enough that verbal commitments were worthless; only legally binding commitments meant anything.
That’s absurd in practice—no one can operate at a BigCo while demanding contracts for every promise without being considered a hyper-bureaucratic weirdo. But the consequence is that teams and orgs “empire build” because they know they can’t trust anyone outside their fiefdom. Virtually all of the VPs and BigCo tech execs I’ve talked to are so steeped in their culture that they can’t conceive of an alternative. But there isn’t an inherent reason organizations have to work that way. I’ve worked at two companies—including my current employer, Wave—where people actually trust leadership, and leadership follows through on commitments.
People often think a high degree of internal distrust is inevitable as a company scales. But people in upper management at Intel and Google have said those companies enforced trustworthiness at the top, and that stamping out dishonesty and bad politics was a major reason for their success under Andy Grove and Eric Schmidt respectively. When leadership changed and honesty wasn’t enforced, standard cultural norms seeped in. That wasn’t inevitable.
It’s often hard to see how absurd a system is from the inside. Americans find Korean chaebol promotion policies—famously nepotistic, with the founder’s relatives placed in top roles—absurdly inefficient. But Korean engineering firms are not, in general, less efficient than American firms outside software, despite practices that seem wasteful to American eyes. American firms didn’t lose dominance in multiple industries while being more efficient. There are offsetting inefficiencies in American firms just as absurd as family succession in chaebols. It’s just that the inefficiencies from American cultural practices seem like immutable facts to those inside the system. But cultural norms aren’t a law of nature.
