Why a Company Like Twitter Needs a Kernel Team
People are often surprised to learn that Twitter has a kernel team. At peer companies of similar size—especially those that grew up in the cloud—such expertise is rare, often due to path dependence. A cloud-native company never had to maintain its own infrastructure, so it never developed deep kernel knowledge. That explains the surprise socially, but not technically. Any company Twitter's size will regularly hit kernel issues, from minor annoyances to major outages. Without in-house expertise, those problems take longer to diagnose and mitigate.
A public example comes from a Twitter engineering post about a firewall misconfiguration that dropped most network traffic. Resetting the firewall config didn't fix things—it exposed a kernel bug. During that incident, the kernel team's familiarity with diagnostic tools and debugging techniques was critical. Without them, understanding why the initial fix failed would have taken much longer. In fact, when polling engineers at similar-scale peer companies, many said they didn't have even one person with the knowledge needed to debug that issue quickly.
Specialist Teams Pay for Themselves
In-house expertise is often justified by a simple cost argument: at large scale, a small percentage improvement can be worth huge absolute dollars. If just one person on a specialist team finds something that persistently reduces TCO by 0.5%, the team pays for itself. Twitter's kernel team has produced many such changes, including kernel patches and configuration fixes with that kind of impact.
The same logic applies to Twitter's JVM expertise. The company employs several ex-Sun engineers who worked on HotSpot, and that also surprises people. But companies that use the JVM at scale run into strange JVM bugs, and having someone with deep knowledge to debug them is enormously helpful. Individual optimizations can also be transformative. For example, a patch by Flavio Brasil virtualizes compare-and-swap calls, and it had a big effect on Twitter's Scala workloads.
Twitter uses a lot of Scala, which is slower and more memory-hungry than Java. Profiling showed a disproportionate amount of time spent in Future/Promise, partly because Futures rely on CAS operations that JVM optimizers can't see through. Brasil's patch avoids CAS operations when the Future doesn't escape the method's scope; a companion patch removes CAS operations in other spots. Together, the two patches cut the cost of typical major services by 5%–15%, more than paying for the JVM team many times over.
Buy vs. Build Isn't That Simple
The common "buy vs. build" argument assumes buying removes the need for expertise. Often it doesn't. Distributed tracing is a case in point. Twitter built tracing infrastructure in-house, based on a vision laid out by Rebecca Isaacs, and gets substantial value from it—multiple uses with 10x–100x return on investment. Many peer companies that bought tracing solutions instead complain it isn't worth it. Choosing "buy" can reduce the expertise required, but it rarely eliminates the need for it.
This pattern extends beyond tech. Pam Wolf, a civil engineering professor with industry experience on multiple continents, shared a similar view: large projects need an owner's-side engineer for each area. Outsourcing to another firm without retaining in-house expertise is often more expensive and more risky in the long run. That matches experience in electrical engineering too, where organizations that fully outsource often ship sub-par designs with delays and high costs.
"Core Business" Is a Weak Argument
Advice to focus on "comparative advantage" and outsource everything else breaks down at scale. Twitter's cache infrastructure and Apple's CPU design are good examples. Both are far from what you'd call core business. Yet the rewards are huge. When Apple brought CPU design in-house via its $278M acquisition of PA Semi, the company started producing some of the best chips in its power envelope. But nothing about Apple made CPU design an inherent advantage before that purchase. If a company can pick an area and make it a comparative advantage, the advice to focus on comparative advantages isn't very actionable.
$278M is a lot in absolute terms but tiny relative to Apple's resources. Much smaller companies can do the same kind of thing with a small fraction of their budget. Even a $100M company could afford the kind of work Twitter did to create novel cache algorithms and data structures. Having great cache infrastructure isn't more core to Twitter than CPU design is to Apple, but it's a lever that adds real value.
Small companies obviously can't have in-house experts for everything. But the threshold isn't as high as you'd think. When Twitter had about 100 engineers, Yao Yue—then working on cache—regularly went to the kernel team for help debugging production incidents. In some cases, debugging would have taken 10x longer without that help. Not every startup will need kernel expertise at that stage, but there will always be some area, not obviously central to the business, where deep in-house knowledge pays off.
