Diseconomies of scale in fraud, spam, support, and moderation

Why the smaller option tends to be the safer one

Counterfeit SD cards offer a useful analogy. If the choice is between Amazon and a local Best Buy, Best Buy is preferable; a reputable local electronics shop (Memory Express, B&H Photo) is preferable still. Such shops are less likely to sell a counterfeit than Best Buy, and the service is better when they do.

The same question can be asked of platforms: on which one do I see more scams, spam and fraudulent content? The answer is the larger one. There are more small platforms in total and they vary more, so a deliberately bad choice is available, but comparing good options against each other, the smaller is better. I have never received a spam Signal message; I get them on WhatsApp regularly. Private forums reach zero bad content; lobste.rs sits very slightly above zero; Hacker News and mastodon.social slightly above that; Reddit noticeably higher; and YouTube, Facebook and Google search results higher still. Support and reinstatement odds run the other way, degrading as size increases.

Setting up scale as a cause has a couple of rhetorical building blocks. One is search: a Google engineer argued that if the search space were more competitive, with many small providers instead of roughly three big ones, it would somehow be more vulnerable to ML-based SEO abuse — "if google can't currently keep up with it, how will Little Mr. 5% Market Share do it?" A thought leader agreed that claims about small independent companies beating the market leader are "just cope" because "economies of scale work pretty well". Empirically, of the search engines examined, the one with a 0.0001% market share was most resistant to SEO abuse and fairly good at it, the 0.001% one was somewhat resistant, and Google and Bing were flooded with SEO abuse that funneled users toward scams. Email behaves similarly: managing your own is commonly described as impossible because of spam, but people do it and often get results similar to or better than Gmail, with the main friction being big-company mail servers that incorrectly ban their small server.

Claims that scale is required started proliferating around the time Zuckerberg responded to Elizabeth Warren's breakup proposal by saying that splitting up Facebook, Google or Amazon would make content moderation and election interference worse, not better, because the companies would be unable to coordinate and their investment would fragment: "It's why Twitter can't do as good of a job as we can. I mean, they face, qualitatively, the same types of issues. But they can't put in the investment. Our investment on safety is bigger than the whole revenue of their company." At the time, Twitter was worth roughly $30B. The claim was funny to me because pre-acquisition I saw a far higher rate of obvious scam content on Facebook than on Twitter — when I clicked through Facebook ads during holiday shopping season, most were scams, and while Twitter had scam ads, it wasn't in the same league. Arturo Bejar, who designed an early version of Facebook's reporting system and led major trust and safety efforts, noticed something similar. Zuckerberg repeated the reasoning later, saying that a dorm-room Facebook "obviously couldn't have had 10,000 people or 40,000 people doing content moderation" and that the AI capacity didn't exist to proactively find harmful content. The sleight of hand is the assumption that 10k–40k moderators were needed at that stage: services larger than dorm-room Facebook deliver better moderation today with a single moderator, often part time.

Engineering writers have made related points. Tom Eastman, formerly of Twitter and other companies, and the 1979 IBM training manual instruction that "A COMPUTER CAN NEVER BE HELD ACCOUNTABLE / THEREFORE A COMPUTER MUST NEVER MAKE A MANAGEMENT DECISION" capture how far the argument has moved: for many critical decisions it is now argued that only computers can decide, and the lack of accountability is treated as a feature rather than a bug. My own view is that the resources question is settled by whether companies deploy the resources. If Meta had adopted something like the vision statement of the power company a friend worked for — "Reliable energy, at low cost, for generations." — and spent the metaverse budget on moderation, it could have hired enormous numbers of people. Instead Meta's 2022 profit before tax was under $30B and the metaverse spend was enormous. Whenever someone says "these systems have to be fully automated because no one could afford to operate manual systems at our scale", what is really being said is "we would not be able to generate as many billions a year in profit if we hired enough competent people to manually review ambiguous cases". That is a defensible choice, but it is a choice.

The workable scale: Meta in Myanmar

The Myanmar case makes a clean study. Erin Kissane's extended account traces an issue that nearly everyone would agree should rank as a top-priority moderation problem, with repeated, severe, urgent warnings reaching Meta staff at every level — engineers, directors, VPs, execs — and almost no resources dedicated to it, while internal documents indicate that only a small fraction of agreed-upon bad content, on the order of a few percent, was caught. That outcome is not unique to Meta; it matches my experience at other large tech companies as both a user and an employee.

A smaller observation. A person named Samantha K. had her Facebook account compromised and it is now used for bitcoin scams. The scammer has done enough volume that he couldn't be bothered to read her name correctly, producing fake photo testimonials about "Kamantha" making tens or hundreds of thousands of dollars. Another connection of mine reported the same sequence — an old account taken over — and has been unable to recover it or get it banned despite a constant stream of obvious scams from it. Compare lobste.rs: Peter Bhat Harkins reports knowing of no such scam, and I have seen none. Mastodon: maybe one, perhaps ever, in my feed, replies or mentions — Mastodon is large enough that scams exist if you search, but the rates are low enough that ordinary use won't surface them. Twitter pre-acquisition and Reddit: roughly once every few weeks in a normal feed. Facebook: bitcoin scams year-round, from ads as well as account takeovers, and obvious scam consumer goods every shopping season. Many people have stopped reporting these scams because they've observed that Facebook doesn't act. Meanwhile Reuven Lerner was banned from running Facebook ads for courses about Python and Pandas, apparently because the system matched something to do with animal trading. That is the fidelity Zuckerberg says smaller companies cannot match. I don't mean to single out Meta; the appendix of Google examples lists around forty cases of automated systems going wrong.

Support, and the cost of being a small account

It is a running joke in tech that the only reliable way to get support from a large platform is to go viral or know someone inside. This compounds every other failure, since good support could mitigate bad moderation, scam detection and anti-fraud errors. Normal channels deliver a form-letter rejection, or a kafkaesque ordeal ending in one. Adrian Black was banned from YouTube for impersonating Adrian Black — that is, himself — and after appeal received: "unfortunately, there's not more we can do on our end. your account suspension & appeal were very carefully reviewed & the decision is final". Google Play developer support told researchers that offshored review slowed response times, and a former Facebook support person described the structure directly: those who spend the most time in the queues have the least input into policy; analysts raise issues to QAs who raise them to FTEs, and issues can take months to resolve or never resolve, while doing the common-sense thing rather than the letter of the policy can hurt your quality score. Hence the nonsensical replies: support is often outsourced to someone making roughly $1/hr in a country where that is a solid wage, reading from a flowchart, in a language they may barely speak. There is also a less obvious penalty — taking sensible action against the flowchart can itself count against you.

Senior people often note, correctly, that the overwhelming majority of appeals should be denied, since "most content on every public platform is bad content". That is a statement about the prior, not a license to assume all appeals are frivolous. My experience in anti-fraud and trust and safety circles is that many — and perhaps most — engineers round "almost all" up to "all", which is quantitatively and qualitatively different.

It's just that breaking up these companies, whether it's Facebook or Google or Amazon, is not actually going to solve the issues ... It doesn't make any of the hate speech or issues like that less likely. It makes it more likely because now ... all the processes that we're putting in place and investing in, now we're more fragmented

At one large platform with a ranked social feed, I have seen this go the other way. A pair of filter instructions went viral — timeline_injection:false, interstitial_ad_op_out and similar — and engineers investigated and concluded they shouldn't work, based on not recalling such a system being built, not finding the strings in the codebase, and checking likely systems. Certainty was avoided because system-level behavior at a large company is beyond human understanding. The one person who told us they had tried it did indeed see nothing happen. The next time it circulated, the question was whether it hurt users' feeds; the person who lost no data was not on the short list.

When a machine learning team asked one social company's anti-fraud group what it had done to measure false positives, the answer was: nothing systematic. It had a mechanism to catch more bad content, but no centralized way to track the error rate. When false positives are that costly, and no one measures them, high rates are unremarkable. One senior engineer at a mid-size company told me the false-ban rate was under 1%; a few months later about 10% of the people I asked had been banned in the preceding two years. At the same company's campus, more than a sixth of employees' accounts had been banned; researchers said credit cards get banned; nothing that hit 10% of a controlled population was a priority. The distributed system research perspective — the aggregate effects are there but you cannot test them — is part of why little is done. The aggregate rate is not visible from any single point.

Making the legibility argument without running afoul of it

My original argument had three parts: users can't agree on rules for content; the optimal moderation policy depends on the population, making it a moving target at policy time; and platform scale itself is a moderation tool, with each size class of platform getting what it deserves — small sites have narrow tastes or are dominated by terrible content, while mid-sized sites get mid-sized flame wars and spam.

Since then the stress in the debate has shifted. Some of it now centers on content moderation and Section 230 rather than platform size. But the diseconomies-of-scale argument still applies. Even in concentrated markets, the failure modes are handled by small companies at small scale, as the search data and email examples show, and the major U.S. firms have responded to user-hostile dynamics with a mix of nostalgia and realism.

Somebody tried argue that if the search space were more competitive, with lots of little providers instead of like three big ones, then somehow it would be *more* resistant to ML-based SEO abuse. And... look, if *google* can't currently keep up with it, how will Little Mr. 5% Market Share do it?

The economics are worth stating. Even with a heavy profit-maximizing mandate, expected returns on spending in other directions dominate, and it is not obvious that the same logic wouldn't hold for a public utility. The strongest version of my argument is that a company whose culture treats scale-hostile dynamics seriously — and which competes for users by doing so — will beat one that doesn't, and the mechanical assumption that a bigger company always has an advantage on cost is unhelpful.

The most pressing change since August is that the anti-antitrust argument is reinforced by the directness of the hypothetical. If a regulator forced change on Google, arguments about AI and machine learning acquire a new urgency for those who built them. So the old line labels many core company operations absurd: would the DoD need to approve infrastructure? Would an adversarial system benefit?

Others discuss AI-enabled states of the world. In a post asking which near-term outcomes he should consider, tech writer Tanner Greer poses the question of institutions with systems of record; he imagines a scenario where a company is more effective than its government and develops "constituents" and administrators that are not quite tools. He also asks whether any outside entity has required people to trust a computer system's judgment over their own. Only a certain class of person takes those questions seriously.

What's certain is that last year's growth-and-antitrust position is now a mainstream conservative position, and I don't expect the arguments to converge. The real gap is between those running the infrastructure and the policy institutions staffed by lawyers and economists and researchers and former employees. Executives have the data and are indifferent to the fact that their arguments are used politically. Regulators have, in a certain way, more ability to seize the frame. The terms they use matter, and on those terms they will act.

Smallness as a strategy, not a shortcoming

The same scaling logic runs in reverse: practices that become impossible at volume are often the ones that keep a community healthy. A single owner reading every message and banning whoever shows up to scam can keep a small Discord essentially clean. At the scale of lobste.rs, or of HN, no one can read everything — but one person can still hold the vision. Policy disputes don't take the "no vehicles in the park" form, because a human decides what the rules are. You may dislike those rules; the alternative is another forum or your own. That is roughly how lobsters came to exist: after HN's earlier moderation regime banned Joshua Stein for publicly disagreeing with a policy, he built it and later handed it to Peter Bhat Harkins.

Craigslist's early days offer a similar picture, per this account of its growing spam problem:

... we were stuck at SFO for something like four hours and getting to spend half a workday sitting next to Craig Newmark was pretty awesome.

I'd heard Craig say in interviews that he was basically just "head of customer service" for Craigslist but I always thought that was a throwaway self-deprecating joke. Like if you ran into Larry Page at Google and he claimed to just be the janitor or guy that picks out the free cereal at Google instead of the cofounder. But sitting next to him, I got a whole new appreciation for what he does. He was going through emails in his inbox, then responding to questions in the craigslist forums, and hopping onto his cellphone about once every ten minutes. Calls were quick and to the point "Hi, this is Craig Newmark from craigslist.org. We are having problems with a customer of your ISP and would like to discuss how we can remedy their bad behavior in our real estate forums". He was literally chasing down forum spammers one by one, sometimes taking five minutes per problem, sometimes it seemed to take half an hour to get spammers dealt with. He was totally engrossed in his work, looking up IP addresses, answering questions best he could, and doing the kind of thankless work I'd never seen anyone else do with so much enthusiasm. By the time we got on our flight he had to shut down and it felt like his giant pile of work got slightly smaller but he was looking forward to attacking it again when we landed.

Eventually a site outgrows the point where one person can own every feature and every moderation call. Even then, it can still pay to let a human own work everyone assumes is automated. Digg's "algorithm," per this telling, was one person:

What made Digg work really was one guy who was a machine. He would vet all the stories, infiltrate all the SEO networks, and basically keep subverting them to keep the Digg front-page usable. Digg had an algorithm, but it was basically just a simple algorithm that helped this one dude 10x his productivity and keep the quality up.

Google came to buy Digg, but figured out that really it's just a dude who works 22 hours a day that keeps the quality up, and all that talk of an algorithm was smoke and mirrors to trick the SEO guys into thinking it was something they could game (they could not, which is why front page was so high quality for so many years). Google walked.

Then the founders realised if they ever wanted to get any serious money out of this thing, they had to fix that. So they developed "real algorithms" that independently attempted to do what this one dude was doing, to surface good/interesting content.

...

It was a total shit-show ... The algorithm to figure out what's cool and what isn't wasn't as good as the dude who worked 22 hours a day, and without his very heavy input, it just basically rehashed all the shit that was popular somewhere else a few days earlier ... Instead of taking this massive slap to the face constructively, the founders doubled-down. And now here we are.

...

Who I am referring to was named Amar (his name is common enough I don't think I'm outing him). He was the SEO whisperer and "algorithm." He was literally like a spy. He would infiltrate the awful groups trying to game the front page and trick them into giving him enough info that he could identify their campaigns early, and kill them. All the while pretending to be an SEO loser like them.

Etsy is said to have used the same approach.

What you can do when you don't want to grow

A site that doesn't need to be big can make choices a growth-minded site never would. In a large HN flamewar, two comments illustrate the split:

My wife spent years on Twitter embroiled in a very long running and bitter political / rights issue. She was always thoughtful, insightful etc. She'd spend 10 minutes rewording a single tweet to make sure it got the real point across in a way that wasn't inflammatory, and that had a good chance of being persuasive. With 5k followers, I think her most popular tweets might get a few hundred likes. The one time she got drunk and angry, she got thousands of supportive reactions, and her followers increased by a large % overnight. And that scared her. She saw the way "the crowd" was pushing her. Rewarding her for the smell of blood in the water.

I've turned off both the flags and flamewar detector on this article now, in keeping with the first rule of HN moderation, which is (I'm repeating myself but it's probably worth repeating) that we moderate HN less, not more, when YC or a YC-funded startup is part of a story ... Normally we would never late a ragestorm like this stay on the front page—there's zero intellectual curiosity here, as the comments demonstrate. This kind of thing is obviously off topic for HN: https://news.ycombinator.com/newsguidelines.html. If it weren't, the site would consist of little else. Equally obvious is that this is why HN users are flagging the story. They're not doing anything different than they normally would.

Cheap, high-engagement outrage is a growth engine for social platforms. HN optimizes for discussion quality instead, so it tries to detect and suppress that material — with deliberate exceptions, such as criticism of HN itself or of YC companies like Stripe, to avoid appearing biased. A platform chasing growth does the opposite: a ranked feed surfaces the most enraging content to the users its models predict will be most enraged. In a country with severe racial, religious or factional tension and frequent calls to violence, the most engaging artifact available is a livestream of someone calling for the death of the other faction and then beating someone — engagement beats everything. Broken Code makes this a recurring theme: harmful content is identified for suppression and overruled because suppression would cost engagement and growth. HN has no such goal, so it can simply remove what it judges harmful.

Growth-agnostic sites can also raise signup friction deliberately. HN goes a little way in this direction by offering a "login" link with no "sign up" link; lobste.rs and metafilter push considerably further.

Theory versus practice

Big-company staff often claim better support is impossible for theoretical reason X, without ever studying how support is actually delivered or how the good providers do it. These now-trillion-dollar companies didn't provide good support back when they were small enough that many peers did, so size is not the explanation — they didn't attempt it then and don't now. The plausible-sounding rationale doesn't survive contact with practice.

The same applies across diseconomies-of-scale arguments. Take the claim that a bigger target matters more than better ML, usually extended into: large companies really do have the best anti-spam and anti-fraud, they just face the most sophisticated attackers. If that held, spamming or scamming on reddit or Facebook would be harder in absolute terms than on HN. It is not. Creating a fresh reddit account and pushing nonsense to the front page was entirely trivial in one experiment; hijacked Facebook accounts post blatant scams for months to years while friends reply in alarm that the account is compromised. The author didn't attempt Facebook takeovers, but weak password practices make them easy, and Facebook's response to friends' reports shows that utterly naive attacks, with zero sophistication, are not defeated. Getting spam or scam content in front of eyeballs is in absolute terms harder on HN than on reddit or Facebook.

The theoretical argument would matter if large companies were anywhere near what their resources allow. They are not close.

Only a couple of examples are listed here to avoid lengthening an already long document, but nearly every counterargument encountered on this topic behaves the same way: examine it briefly and it turns out to be a cocktail party idea with little or no connection to reality. The meta-lesson is that vaguely plausible arguments from practitioners on this subject cannot be trusted, because nearly all collapse under examination. It seems reasonable that a business the size of reddit would have anti-spam more sophisticated than HN's, where one person writes the code and does the moderation. But the most naive tricks for front-page placement work on reddit and fail on HN. HN's system can be beaten, presumably, but it takes a little thought; reddit and Facebook take none. Support follows the same pattern — talk to anyone who knows how to run a support org that serves users well, and it becomes immediately obvious that big tech hasn't seriously tried the most obvious things.

Why leaked-document coverage is weak evidence

Journalists' summaries of leaked documents deserve a much lower level of trust than the documents themselves. Reading a reporter's account of a source often leaves an impression quite different from the one the raw material produces, and in a fair number of cases the spin is heavy enough that the story asserts the opposite of what the documents say. The Cruise pedestrian accident report is one case where this played out; Zeynep is one of the rare exceptions. The subject is large enough to warrant a document of its own, so two examples serve as illustration here.

The Zarashaw testimony

Eugene Zarashaw, a director at Facebook, testified in a Special Master's Hearing:

It would take multiple teams on the ad side to track down exactly the — where the data flows. I would be surprised if there's even a single person that can answer that narrow question conclusively

Coverage of that testimony produced headlines including "Facebook Has No Idea What Is Going on With Your Data", "Facebook engineers admit there's no way to track all the data it collects on you" — illustrated with a stock photo of a person in a nest of cables, holding their head — and "Facebook Engineers: We Have No Idea Where We Keep All Your Personal Data".

No technical background is needed to see the distortion. Tracking down precisely where all data, direct and derived, lives for every user is not the same thing as having no idea where that data is. A logged-out, cookie-free Google search for Eugene Zarashaw facebook testimony returns results that are, above the fold, uniformly misleading clickbait of this kind. For readers who understand the systems involved, the quote is not egregious at all — it is mundane, expected and reasonable.

Grounds for citing secondary accounts anyway

Despite the above, there are reasons to cite Jeff Horwitz's Broken Code and a handful of similar stories with some confidence.

• Deleting every reference to these accounts leaves the arguments unchanged — the same is true of deleting half the user stories in this document.
• The attitudes on display matter more than the specific numbers. Those attitudes resemble what I've seen at employers and heard about from well-connected friends, so I could substitute similar secondhand stories; using public sources is preferable to anonymized friend-of-a-friend accounts, which makes the quoted attitudes a stand-in for stories I can verify.
• A third reason is too subtle to cover here and will be addressed when this disclaimer becomes a standalone document.

If you're looking for work, Freshpaint is hiring (US remote) in engineering, sales, and recruiting. Disclaimer: I may be biased since I'm an investor, but they seem to have found product-market fit and are rapidly growing.

Erin Kissane's investigation into Meta's role in Myanmar is documented in meticulous detail, and her own description of what she found is blunt: "The harms Meta passively and actively fueled destroyed or ended hundreds of thousands of lives that might have been yours or mine, but for accidents of birth." She avoids "millions" only because it "sounds unbelievable," while believing the real number is "very, very large." The essential claim, attributed to the lead investigator on the UN Human Rights Council's Independent International Fact-Finding Mission on Myanmar, is that Facebook played a "determining role" in the emergence of the Rohingya genocide.

From a distance, she notes, that role can read as ordinary content-moderation failure: "content moderation fuckups, right? In a country they weren't paying much attention to?"

The writer has not examined the underlying material closely enough to confirm the "determining role" characterization, but considers it plausible at a meta-level. Public refutations of Kissane's work appear to be pre-refuted inside her own text, suggesting that many of those disagreeing with her either did not read the articles or did not follow her argument. The dynamic resembles reactions to David Jackson's proof of the four color theorem, where skeptics had read the paper and found it flawed while believers relied on signals such as the author's track record or institutional prestige. Confidence that Kissane's summary is roughly accurate is high but lower than in the proof case, since establishing a positive claim is harder than finding a flaw and the domain is harder to evaluate.

Unlike with Broken Code, the source documents here are available and Kissane's steps could be retraced, but the claims needing further reading do not determine the correctness of the document, so that work is left to someone else.

The "head of engineering" objection

The most substantive public objections sit around the edges. One critic wrote that the articles claim "Arturo Bejar" was "head of engineering at Facebook," a "simply false" characterization of a Director-level manager role, "not remotely close to 'head of engineering.'" What Kissane actually wrote was "Arturo Bejar, one of Facebook's heads of engineering."

The objection is therefore technically incorrect: she never called him the head of engineering. Read across the articles, she uses the same colloquial, uncapitalized construction for "Susan Benesch, head of the Dangerous Speech Project" and "the head of Deloitte in Myanmar." A technically correct version of the objection exists — inside a large tech company, "Head of Engineering" generally suggests an executive all engineers transitively report into, and someone fluent in that lingo would avoid the phrasing even for lay readers — but this is weak evidence of factual error, not proof of one.

The same critic continued that this "calls into question some of the accuracy of how clearly the problem was communicated to relevant people at Facebook," arguing that telling "random engineers or Communications VPs" about a complex social problem isn't enough. On this post's topic, that objection supports the argument rather than undermining it: Arturo Bejar was "the leader for Integrity and Care Facebook" according to his LinkedIn, and Broken Code discusses his role at length in connection with Meta in Myanmar. He was not a random engineer or a communications VP.

Warnings, and what they met

Kissane documents years of repeated warnings that went well beyond routine reports of bad content and fake accounts. They included direct conversations with directors, VPs and other leaders, and they were dismissed. The prevailing assumption appears to have been that existing content-moderation systems were sufficient, even against strong evidence to the contrary.

None of this seems to get through to the Meta employees on the line, who are interested in…cyberbullying. Frenkel and Kang write that the Meta employees on the call "believed that the same set of tools they used to stop a high school senior from intimidating an incoming freshman could be used to stop Buddhist monks in Myanmar."

Aela Callan later tells Wired that hate speech seemed to be a "low priority" for Facebook, and that the situation in Myanmar, "was seen as a connectivity opportunity rather than a big pressing problem."

Even after Meta decided to act, the result was thin:

As the Burmese civil society people in the private Facebook group finally learn, Facebook has a single Burmese-speaking moderator—a contractor based in Dublin—to review everything that comes in. The Burmese-language reporting tool is, as Htaike Htaike Aung and Victoire Rio put it in their timeline, "a road to nowhere."

The resource contrast is stark. In 2014 Meta was the fourth largest tech company in the world, worth $217B with $3B/yr in net profit — not the $50B metaverse company it later became — so at a globally generous loaded cost of $30k/yr it could have afforded on the order of 100k moderators and support staff. (Kenyan moderators were paid $2/hr without benefits.) Allocating Myanmar a standard share of capacity by population, at 0.7% of the world total, still yields around 700 generously paid moderators and support staff — for a developing genocide treated as low priority.

in the years before the coup, it already had an internal adversary in the military that ran a professionalized, Russia-trained online propaganda and deception operation that maxed out at about 700 people, working in shifts to manipulate the online landscape and shout down opposing points of view. It's hard to imagine that this force has lessened now that the genocidaires are running the country.

The adversary had 700 people too, without comparable technology. As Zuckerberg frames it, smaller companies cannot match Meta's infrastructure, but billions in technology buy little when the ratio is 700 to 1 and the 1 relies on tools built for a different purpose.

They report posts and never hear anything. They report posts that clearly call for violence and eventually hear back that they're not against Facebook's Community Standards. This is also true of the Rohingya refugees Amnesty International interviews in Bangladesh

What the whistleblower documents say about enforcement

Inside the 40,000-word treatment, Kissane works through whistleblower reports for figures such as: "we're deleting less than 5% of all of the hate speech posted to Facebook. This is actually an optimistic estimate—previous (and more rigorous) iterations of this estimation exercise have put it closer to 3%, and on V&I [violence and incitement] we're deleting somewhere around 0.6%…we miss 95% of violating hate speech."

[W]e do not … have a model that captures even a majority of integrity harms, particularly in sensitive areas … We only take action against approximately 2% of the hate speech on the platform. Recent estimates suggest that unless there is a major change in strategy, it will be very difficult to improve this beyond 10-20% in the short-medium term

While Hate Speech is consistently ranked as one of the top abuse categories in the Afghanistan market, the action rate for Hate Speech is worryingly low at 0.23 per cent.

This is not an argument that Facebook performs significantly worse than comparable or larger platforms; large tech companies generally carry high false positive and false negative rates and employ people who dismiss concerns with assurances that things are fine.

Elsewhere

• Anna Lowenhaupt Tsing, On Nonscalability: The Living World Is Not Amenable to Precision-Nested Scales
• Glen Weyl on radical solutions to the concentration of corporate power
• Zvi's collection of quotes from Moral Mazes

Appendix: A catalogue of large-scale filtering failures

Ever since Zuckerberg's claim that only the largest companies can plausibly handle moderation, anti-fraud enforcement, and anti-spam, I've been quietly collecting examples of big-company failures that crossed my path during ordinary browsing. Had I gone looking for them on purpose, the list would be far longer. Some companies simply never tripped my radar — AirBnB, for instance, generates constant complaints, yet it didn't occur to me to start recording them until I began drafting, so there are only a handful here despite a volume that would likely rival Uber's.

The failures are frequent enough that during review, at least two of eight draft readers hit one themselves. Peter Bhat Harkins ordered a used Keychron K1 v5 through Keychron's official Amazon storefront, fulfilled by Amazon, and received a v4 with an older mechanical switch instead of the current optical one; the model/serial sticker had apparently been peeled off and one stabilizer was broken from wear, consistent with a common Amazon scam in which someone returns a different unit than they bought. A second, anonymous reader created a shared Gmail account so they and their partner could receive mail from local services. Google soon demanded identity verification:

Verify your identity

We've detected unusual activity on the account you're trying to access. To continue, please follow the instructions below.

Provide a phone number to continue. We'll send a verification code you can use to sign in.

Supplying the number they signed up with produced:

This phone number has already been used too many times for verification.

That number was accepted at account creation, so the "illegal" number wasn't caught until the account had been used for a while and its address given to several services — fortunately small local ones reachable by phone. Similar stories are common for providers that never ask for a number at signup but then lock accounts until one is supplied; this case was unusual because the number stopped working days after it had been provided. The error can also be read as "too many verification codes sent to this number recently," but in recent history it had carried exactly one code: the one used to attach it to the account.

My own examples: a 10-pack of Amazon Basics power strips arrived with solder — possibly lead-based, though I didn't test it — smeared over the cable of the first unit I pulled out, which raises the question of whether every such electronics purchase needs washing to avoid lead dust. And with spam filtering, failures run in both directions. Spam lands in my inbox; legitimate mail lands in spam, including the classic case of a reply to a message I sent. Most of my draft readers who use Gmail have seen the same thing and no longer find it worth mentioning.

Where relevant below, I mark instances of commenters blaming the user even when the failure plainly isn't the user's fault. The absence of such a note doesn't mean that response was absent — I haven't catalogued it anywhere near exhaustively.

Google

• Support so unresponsive that one user needed the NY attorney general to write a letter before getting a reply, in order to file their taxes.
• Photo search for "gorilla" returned images of Black people; the fix, after a viral thread, appears to have been blocking the search terms "gorilla", "chimp", "chimpanzee", and "monkey" — terms which still returned no results when I uploaded a gorilla photo and searched on 2024-01-06, immediately and again weeks later.
• A YouTuber suspended for impersonating themselves; the appeal returned "unfortunate, there's not more we can do on our end. your account suspension & appeal were very carefully reviewed & the decision is final ... we really appreciate your understanding". Restored after a viral Twitter thread hit HN's front page.
• Two users locked out of their accounts after moving, with no recovery.
• Google closed accounts of everyone who bought a phone and resold it to a particular buyer, plus at least one account that served only as a recovery address for such a person — bounced mail, broken Google sign-in — until Dans Deals wrote it up and accounts were reinstated.
• Google Cloud cut a user's quota, causing an incident, and declined to restore it. Support said the user exceeded the rate limit; the user pointed to an approved 18k/min quota against actual usage of 5000/10min; support said that wasn't the limit and weren't sure what was. Evidently an internal limit stricter than the approved quota. Commenters noted that without paid support, you have no support — and that paid support can also leave you with none.
• Obviously fake DMCA takedowns accepted despite robust-process assurances from Google, which cited automated and human review, backdating detection, Lumen transparency, and counter-notification.
• A small business app creator had everything shut down pending "verification" of Google Pay; support did nothing and Cloud would not look into it until the story hit #1 on HN.
• New York's right-to-repair bill had language inserted directly by a lobbying group representing Google, Apple and others, excluding many devices; "What hurt this bill is Big Tech was opposed to it," said assemblymember Fahy.
• A Google Drive file containing the single line "1" was restricted for copyright infringement, with the appeal denied; HN readers found files containing "0" flagged similarly. Fixed after a viral thread.
• In 2016, Fark's ads were disabled because a 2010 photo of a clothed adult was wrongly flagged as child porn; appeals took five weeks. Similar skin-content flagging had hit them in 2013.
• Pixel 6 freezes on emergency calls: a user reported the same earlier-Pixel issue nearly four years before this complaint, escalated with no fix, and still live ~8 months before the prior report. A Google account blamed Microsoft Teams, but this user had never installed or used it. (A genuine Teams-blocking issue existed, but wasn't this.)
• A father's account was locked and his information sent to SFPD over images of his son's groin taken to send to a doctor. SFPD cleared him, and Google "stands by its decision" and did not unlock the account. A spokesperson described hash matching and AI for CSAM identification.
• A corporate Cloud account was suspended, causing an outage, after a billing bug whose false assurance that suspension wouldn't follow was itself false. An HN suggestion to "engineers that lack business experience" to contact account managers once spend is significant drew replies from multiple people who had done so without any help.
• A company locked out of its own domain on Workspaces with support refusing to act.
• A Cloud account suspended after stolen corporate card numbers produced a fraudulent AdWords charge.
• A journalist demonetized on YouTube; fixed after 7 months of appeals and a viral thread.
• Ads account suspended, guessed to be ML fraud signals plus a Brex card — even though the card works elsewhere in Google services. Another card stopped working on Google accounts after being used across several; another account suspended for "suspicious payments" despite the card being used for many non-suspended Google payments, with appeals and internal escalations by a former Google employee all failing.
• Google Play account banned with no known reason, and an appeal link that can't be opened from a banned account; two apps sharing one API counted as two simultaneous violations, i.e. "multiple violations".
• A small non-profit's ads account banned for "unpaid balance" reading $0.00, appeals failing.
• An ads account banned after the interface auto-switched to Japanese and then payment was made with an American card.
• A public election information sheet removed for "phishing"; restored after a viral HN thread.
• Account disabled, photos lost, no explanation for the ban or the rejected appeal — and an ex-Google engineer couldn't find anyone able to restore it.
• A 10-year-old YouTube channel with 120M views scheduled for deletion over copyright claims with no information given; saved after a viral thread.
• The FairEmail and Netguard developer gave up fighting Google over whether FairEmail is spyware and removed the apps; later restored after a viral HN thread.
• An app banned from Play because a button read "Report User" instead of "Report".
• A user banned from GCP for running Google's own GCP tutorials.
• YouTube comment anti-spam was so inadequate that a user wrote their own.
• Product review searches generally return SEO linkfarm spam rather than useful results.
• An account holding thousands of dollars of apps banned with no information and rejected appeals; restored after a viral thread.
• The Linux Experiments YouTube channel deleted without reason, restored shortly after viral Twitter and HN threads.
• A Pixel 7 Pro warranty replacement arrived carrier-locked to a US carrier, with the user in Australia; eight support calls in a month produced eight incorrect assurances the phone was unlocked, leaving the user without a usable phone for a month. The carrier agreed the lock was wrong but couldn't act since the original purchaser would have to call. After the Reddit thread, support agreed to swap phones while continuing to insist the phone wasn't locked.
• Malware using Google OAuth to hijack accounts; Google claims mitigation for all compromised accounts.
• A GCP account suspended after years of use for no discernible reason: support useless, but an email to a former Google co-worker produced immediate internal escalation and a fix.
• Obviously fake Google reviews for a film went unremoved for a long time; many copied identical text.
• Obviously fake restaurant reviews undetected — a pattern I've seen locally, where new restaurants carry 100+ five-star reviews that are plainly fake and remain for years.
• A SaaS studio had 7 of 100 apps sharing one codebase rejected for lacking UGC blocking/reporting tools it already had; support replied with scripted nonsense about reviewing their e-learning course, closed off escalation requests, and ended one chat with "As much as I'd like to help, I'm not able to assist you further." Developers suggest simply resubmitting with a token change like an incremented internal build number, since the review process responds to changes rather than arguments — and that it's a mistake to treat the messages as coming from anything rational.
• Google Groups is a major source of USENET and email spam; a Google employee filing information into a ticket didn't fix it, and neither does the "can't add me to groups" setting.
• A user locked out because an authenticated phone number change was ignored, with auth texts sent only to the old number. Relatedly: I once got locked out traveling with only my code generator and my 2FA tokens at home. I had added tokens to reduce that risk, since the documentation implied any configured method would work. It doesn't: Google sometimes accepts only specific methods, so adding more methods can increase your chance of losing access.
• A paid unlimited-storage plan was cancelled and the user's data deleted; many commenters told the user to have had backups, apparently not having read that a government agency concurrently held all their hard drives.
• A company's Cloud account closed over a 3-cent billing error.
• YouTube declined to remove obvious scam ads, replying "We decided not to take this ad down. We found that the ad doesn't go against Google's policies". Also: obvious scam ads more broadly, ads for fake medical treatments, fake download buttons, fraudulent ads reported repeatedly, real estate scams using Wayne Gretzky. Personally reporting scam ads to a Google employee in the area takes them down for a day or two before they return.
• Incorrect YouTube copyright takedowns, including two separate claims over the sound of typing on a keyboard (each fixed after a viral thread), a claim over white noise, and someone claiming ownership of free music a user had posted for remixing — fixed after the creator, one of the biggest YouTubers ever, made a complaint video.
• A developer's app removed for no discernible reason (allegedly "user privacy") and restored for no discernible reason.
• An SEO spam clone of a competitor's site taking its traffic; SEO spam obituaries spawning an "obituary pirate" cottage industry.
• A negotiated limit of 300 concurrent BigQuery queries cut to 100 because Googlers judged a new feature worth 3x in concurrency, with support apparently unable to help users for whom the feature is useless.
• A tiny GCP instance repeatedly shut down by incorrect crypto-mining detection; an IP quota increase refused with "Based on your service usage history you are not eligible for quota increase at this time" and support unable to fix it.
• Google Maps routing hikers onto bad trails, with search and rescue teams warning against it; routing people to unplowed forest service roads with 10 feet of snow during a highway closure; routing people through washed-out forest service roads in worse condition than the flooded highways; routing onto roads needing an offroad vehicle, with reports doing nothing; sending users the wrong way into a highway offramp; and directions so bad that locals put up a sign telling people to ignore them. One user's reports of a bad route went unfixed.
• Google's suggested American and British pronunciations of numpy.
• Google's CEO personally saw that a recruiter who accidentally breached the wage-fixing agreement with Apple was fired and that apologies reached Apple's CEO.
• A developer rejected from the store and given the runaround for months — sending instructions showing the app does X after support said it didn't, while their analytics showed support never attempted the steps.
• A malware app stayed in the app store at least six months after being reported by a user whose parents it infected, though it now appears gone.
• Google's accessible audio captchas cut users off after 2–3 attempts in favor of image captchas, making Google sites inaccessible to some blind users. Another user got impossible ReCaptchas, couldn't cancel paid services behind them, and had to issue chargebacks; captchas also assume familiarity with American objects regardless of where the user is.
• India-specific apps undownloadable while in India because region changes are limited to once a year.
• A 3-year-old YouTube channel with 24k subs, 100 videos and 400 streams deleted, allegedly for saying "Don't hold your own [bitcoin] keys" as promoting illegal activity. YouTube confirmed the suspension and linked a policy doc; questions about which content violated it, and why the channel wasn't given the documented three strikes, went unanswered.
• Google Play rejections for nonsense reasons, with developers resubmitting until an app succeeds. Multiple developers document years of incorrect Play policy violations and workarounds; a commenter claiming to have worked on the Play team described outsourcing overseas that slowed response times and "incompetent" managers. One developer's updates were sometimes rejected and an existing app delisted, but meaningless changes reliably got them relisted. Another developer banned from Play won their appeal, yet their package namespace stayed blocked, requiring a refactor and a change of product and company name. Others built semi-automated handling for the kafkaesque Chrome Web Store process, and interactions with "Chrome Web Store Developer Support" were described as comical, sub-ChatGPT level.
• Repeated nonsense demonetization and age-restriction: "I ate water with chopsticks" struck for "violent or graphic content", with a follow-up blaming an ML rating "mistakenly confirmed by a manual reviewer" and a promise it wouldn't recur — which it did. A megaman speedrun history video age-restricted (and thus largely demonetized), appeal denied 45 minutes after a 78-minute video's submission; the restriction lifted after a tweet, the video then stopped being recommended, the restriction returned 8 days later as an error, and only after a video and tweet about the runaround went viral did YouTube respond with "really sorry again that this was such a confusing / back and forth experience". Another video restored after the creator and another big YouTuber both wrote viral threads.
• Gmail spam filtering reportedly worsening: multiple spam emails per day with plenty of false positives, along with the same spam another user reported. Jamie Brandon, organizing a database conference, had the majority of his announcement emails spamfiltered, including to ~700 people who had explicitly signed up for notification. A reply quoting my entire email to a local window film installer went straight to spam. "Obvious to humans" spam passes constantly while ham gets classified as spam. A company's offer-acceptance reply went to spam for everyone at the company, including a reply-to-a-candidate case where 19 of 20 "spam" messages turned out to be legitimate; Google support suggested a "Never send it to spam" filter, which amounts to disabling spam filtering — and one person had actually done exactly that.
• An author's knowledge panel was populated with someone else's photo after publishing a book about their victimization and sex crimes. Weeks of feedback and support contact eventually got the photo deleted, after which it was replaced with another pastor of the same name, then someone else, then the pastor again months later. A Google public liaison for Search suggested that had the author filed feedback explaining the situation, the image would have been removed; the author replied that they had dozens of email exchanges, screenshots of feedback submitted via the knowledge panel link, bouncing between knowledge panel support and legal removals, automated replies, and a direct email to that liaison.
• A famous sci-fi author spent between five years and a decade getting photos of other people removed from their knowledge panel; another user couldn't claim a misleading panel; another was wrongly described as being "also known as The Sadist ... a Bulgarian rapist and serial killer", fixed after a story hit #1 on HN. Knowledge panels for businesses often hold wrong information even when the business website is correct.
• A Gmail user can't download their data after Google imposed a size limit.
• A developer's Android app was taken down as a clone of an iOS app — the developer's own. Unbanned, but revenue settled from $10k/mo to $1k/mo and the developer stopped building for Android.
• Using "site:" incorrectly puts users into CAPTCHA hell; other query modifiers trigger the same.
• Reporting malicious Chrome extensions often doesn't get them taken down, though some fall after a viral blog post.
• A user accidentally obtained admin privileges on many companies' Google Cloud accounts and couldn't raise any support ticket about it; multiple replies described Google's paid support as bad compared to AWS's.
• Losses despite correct credentials: a 15-year-old Gmail account with no recovery path, where someone who helps many people recover says they all hit a brick wall of "at their scale, nothing can be done about such 'edge' cases"; accounts lost while presenting proper auth; with proper auth plus backup account; with proper auth plus TOTP; or with proper auth whose login only worked in the city the person used to live in. Some were restored for no known reason months later; one user got ~20 consecutive security challenges before "You cannot log in at this time".
• An account lost during a 2FA recovery failure: the user broke their phone, losing Google Authenticator, had backup codes that only allowed login while consuming a code each time and did not permit changing 2FA — so every login was a countdown. Commenters walked them through steps that work for others but not for them, presumably because an anti-fraud system's suspicion restricted which 2FA methods can be used to change 2FA, requiring the lost original. Others reported identical problems; their mitigation is storing 2FA secrets in their password manager, sacrificing the security benefit to avoid random loss. Setting multiple Yubikeys sounds like a fix, and I've tried it — it is not: Google once flagged a 2FA method I used almost every time as suspicious and required a different token, so if the wrong tokens get flagged, losing any one of N can lose the account. Google One paid support is reported as very poor, and cloud support can be useless at spend levels of millions to hundreds of millions per year.
• An account lost after Google decided the phone numbers used for verification "cannot be used for verification"; another user found the official support suggestion is to create another account.
• Lockouts triggered by password changes: a user with a password manager sure of the correct password, using a reset flow that didn't work, recovered after five weeks of daily attempts; another asked to supply new and old passwords (which worked) and to scan a QR code from a logged-in account (impossible while locked out), who had changed primary and recovery passwords together and eventually got into recovery, then the main account; another was asked for a 10+ year old password alongside the current one and recovered after finding a support forum suggestion not to log in for 40+ days, which reduced the request to the current password. That isn't a universal fix — some people retry yearly to no effect. Another account was moderately broken for 10 years by a forced 2013 Google+ migration.
• Account losses where Gmail refused the correct password alone: requiring a lapsed recovery address (with a standard user-blaming reply from someone who apparently didn't read); requiring an old, deactivated phone number (waiting long enough eventually worked, or by another report left the account permanently lost); requiring an old deactivated number with no late reprieve; and a case with recovery email available where the account was lost for three years before logging in worked for no discernible reason.
• A Google Voice number given away — one used daily with purchased credits, which were also lost. Support refused to refund the credits and couldn't issue a new number because the old one remained linked and counted as a spam source. Someone suggested not letting the number go inactive or "making the number permanent".
• Losses when a recovery-account token or credentials stopped working for no discernible reason.
• A user told by support to issue a chargeback, which then banned the account and destroyed 15 years of history.
• A user in the middle of being locked out made a viral post to try to reach a human.
• John Carmack had "a ridiculous time" with Google Cloud, resolved only because he complained on Twitter as one of the most famous programmers alive; after the second occurrence he moved providers.
• YouTube video incorrectly taken down for community guidelines and restored; a video about the history of megaman speedruns age restricted (see above); another big YouTuber's threads moving takedowns.
• Extensive documentation of incorrect restriction-and-appeal cycles where each replays the same scripted response. Inside YouTube, one follow-up acknowledged "your video was rated [sic] limited by ML then mistakenly confirmed by a manual reviewer as limited .... we've talked to the team to ensure it doesn't happen again", after which it kept happening.
• Reporting a spam YouTube comment does nothing.
• The BBC reported bad ads to Google; Google claimed an ML fix, and follow-up searches showed nothing fixed.
• A user sold thousands of dollars in AdSense ads and was never allowed to cash out — a story the author says they've seen hundreds of times, with multiple responders reporting the same and no recourse.
• A Gmail account locked for no discernible reason with recovery and appeals doing nothing, recovered by the same process after two years.
• A Google Pay account locked for "fraud"; a form intended for investigation did nothing three times; the user's Google Fi phone, Gmail, and DNS all stopped, and a couple of years later everything started working again for no discernible reason.
• A user locked out of Gmail despite the correct password and access to the recovery email, Google citing suspicious login; a similar case befell the author with correct password plus a 2FA device, with things later working again.
• An SEO spam clone (see above); a Google employee reachable only through prior employment for effective support.
• A 15-year-old Gmail account lost with the "edge case" wall (above); accounts lost with proper auth, backup access, and TOTP (above).
• A user's quota reduced from 2TB to 15GB by a ToS change, unable to reach a human, forced onto a more expensive plan to avoid data loss. Nearby: a YouTube account with a single video and no comments banned for no apparent reason, support doing nothing.
• A huge YouTube channel shut down, defended by one commenter as appropriate because the account was session-jacked and taken over by a crypto scam farm; someone familiar corrected that the shutdown came days after the crypto issue was resolved, per discussion on the WAN show the prior week.
• A 5M-file limit imposed on Google Drive without warning produced many serious breakages; support replied that "the error is working as intended". The top HN comment, from a Google engineer, argued reasonable human users don't have 5 million files, and that specialist software producing such datasets is unlikely to run happily on streamed files. Multiple people agreed — in a thread full of users explaining how they were hitting it. Someone pointed out that Drive advertises tiers up to 30TB, so 5M files would average 6MB, hardly an exotic case; another noted their home directory contains almost 5M files. A second Google engineer, top reply to the #2 comment, said Drive isn't for files but for things like Google Docs — even though Google's own page calls it a "File Sharing Platform", advertises "Store, share, and collaborate on files and folders from your mobile device, tablet, or computer", and users overwhelmingly believe it's for files. Lower-ranked comments wondered why Google didn't contact impacted users first. The attitude that users who deviate from a product team's imagined usage — even when that usage matches the marketing — are using the product wrong was common when the author worked at Google and appears unchanged.
• Chrome on Android places tabloid stories and other spam alongside frequently used domains. Google pays Apple not to compete on search.
• Google search has been full of scam ads for years: r/blender warning for months that the top hit is malware; rampant supplement scams; the top local restaurant result being a scam restaurant; high-ranking results for software being malware that bought the top ad slot, with reporting ineffective and the same ads persisting for very long periods unless an engineer is contacted or a thread goes viral; the top result for the Zoom installer being badware; and many scam ads on YouTube.
• A wedding vendor list organized in Drive was tagged as phishing, with a warning for violating Drive's phishing policy and no way to learn more. Corporate security notes how easy it is to phish employees through Google Groups.
• A popular extension appears to steal credit card numbers after being acquired, with reviews showing injected ads and failures; an attempt to take it down 6 months prior seems to have failed, and the Firefox version remains available.
• An account banned after updating a credit card to replace an expiring one from the same issuer with the same billing address.
• Reporting a spam YouTube comment does nothing (see above).

Facebook (Meta)

• A journalist's account deleted, restored only after a viral thread.
• A moderator noted there's no real feedback or escalation path between moderators and the people setting moderation policy.
• A WhatsApp ban with no reason, answered by generic template appeal responses; an Instagram account that could no longer interact, and couldn't be removed because login failed.
• Multiple users who created Facebook accounts only to view and manage FB ads had the accounts banned, losing ad management.
• A user banned after their FB account was hacked, restored after a viral HN story. A local FB group post described an Instagram account taken over by a bitcoin scammer posting daily, with no visible way to contact Instagram; a suggested help URL didn't work because the hacker was already in the email and verified all changes first. The author's advice was to find an employee connection via LinkedIn, since only internal escalation or virality works.
• Facebook reported a user to DigitalOcean for phishing over a blog post, with DigitalOcean threatening to suspend the droplet unless it was deleted within 24 hours; the appeal succeeded, though it's unclear it would have gone through without the viral HN thread.
• Marketplace ban after posting a vacuum ad, with multiple appeals denied and "The review is final".
• Reporting a post advocating violence against a person does nothing; reporting a post telling another user to kill themselves does nothing; reporting the flood of racist comments on a murdered person's post does nothing.
• A ~40,000-word series by Erin Kissane on Meta in Myanmar.
• Obvious scam ads not taken down after reports; users stopping reporting because they're never removed and always deemed compliant. Accounts of dead people taken over to run scams, fake pages for businesses, and general scammers reported with no action.
• "Creator"-level paid support appears worthless: you reach a human not reading a script, but who remains useless. One creator was told to edit and re-upload a video that wouldn't upload — support's reason for thinking it might fix it being basically that it might also just work. When "hacked", support responded quickly by sending publicly available links the creator could have googled.
• Zuckerberg rejected proposals from other FB executives to improve teen mental health; reporting noted that a lack of investment in well-being initiatives meant Meta lacked "a roadmap of work that demonstrates we care about well-being."
• A malicious Google ad from a Google-verified advertiser, removed only after a major publication covered it. A user noted that attackers are helped by Google permitting fake display domains, which is necessary for tracking as currently implemented.
• A lifetime ban from running ads, triggered by ads for courses teaching pandas, the Python library. Someone noted the appeal button is a trap you should never press, and that pressing it forecloses a different form you'd need.
• Pervasive scam ads reported.
• You can deactivate anyone's WhatsApp account by emailing to request it — the inverse of reporting accounts that scam you and having nothing happen.
• Innocuous Threads messages removed for "violating Community guidelines", with users asking why so much spam stays up while their replies come down.
• Rampant fraud across Instagram, Facebook and WhatsApp. Reporting a spammy reply on Threads (a link to the replier's Substack) did nothing.
• Moderation in Kenya and union-related failures covered by Jacobin.
• A post showing art, electronics, and wheelchair modifications removed as "hate speech", with no support action but restoration after the story went viral.
• Stories that vaguely resemble holding a gun to one's head — e.g. holding a hair dryer to one's head — get flagged.
• A user reported Threads desktop was unusable for six weeks (infinite login loop on most browsers) and then suddenly worked.
• A user banned from Facebook's 2FA systems, including WhatsApp and Instagram, deleting a business Instagram page, due to a flaw in password recovery. Despite 2FA, the account was taken over, and the appeal returned "We've determined that you are ineligible to use Facebook". The user had used FB login for the DMV and lost access there too; new accounts got linked to the old one and banned. One commenter observed that the system can identify that linking but can't fingerprint a login from Vietnam and a hijack. Standard replies blamed the user for using a big platform; the author asked how to get clients to switch platforms and got no response.
• An account banned after compromise from a foreign IP, with the standard "they can't review cases at that scale" response.
• A user effectively banned due to broken password recovery requiring total strangers to vouch they're a real person, apparently due to bad ML; a scammer then created a fake profile of them, so FB has a fake version of the person and not a real one. Another effectively banned by bad "suspicious activity" detection despite 2FA, password and 2FA access.
• Repeated suspensions with no discernible reason; another ban lifted only when a friend got a job there and opened an internal ticket.
• Accounts hacked and then disabled, documented in press and blog form with many similar cases in comments.
• An account disabled after a hack, fixed after reaching HN's front page and FB engineers intervening — with the usual commenters telling the user to enjoy life without Facebook. One commenter suggested people actually read the article, quoting a user explaining that as a mother of two newly moved to a new area, Facebook groups offered support and community; the replies persisted, calling it a blessing in disguise.
• People without Facebook employee contacts took to Google Maps reviews to complain about Facebook's anti-fraud systems.
• A user banned after posting about a 12V system in a Nissan Leaf group: the post was automatically judged to violate community standards, requiring identity verification, but license uploads failed across formats, sizes, browsers and computers until the attempts hit a cooldown, the deadline passed, and the account was gone.
• Accounts lost repeatedly among one person's wife and her friends; a girlfriend's mother's account taken over with reporting doing nothing, and a programmer finding it strange that changing the password, email, photo, and name in rapid succession triggered no anti-fraud check.
• Mass-reporting groups can get accounts banned on request.
• An account locked pending photo ID upload which did nothing; unbanned six months later after getting to know a Facebook employee. Another user contacted FB employees on LinkedIn without success and eventually got unbanned by meeting Facebook employees through Instagram.
• Effectively banned from Instagram after logging in from a new device and being unable to confirm a no-longer-active email.
• An account stolen apparently bypassing 2FA, with a white male father of three's account replaced by a young Asian woman's account without tripping anti-fraud systems. Impersonation reported on Instagram goes unactioned, as do fake accounts impersonating a user and family members; an ad account hacked left users with no response, useless auto-responses, or useless webpages.
• A relatively early post-IPO Facebook engineer's account was banned with the standard appeal process failing, and engineering friends inside also unable to escalate — losing the account plus ad money and Oculus games.
• A sophisticated user with a strong random password, password manager and 2FA had their Instagram stolen after crypto people spent six months trying to buy the account. Following Instagram's official instructions produces a literal infinite loop of instructions; Instagram claims it emails from [email protected] on email changes, which didn't happen, and Fastmail logs suggested no email compromise. Recovery came after the story hit HN's front page. Services reportedly get desired Instagram handles for $10k–$50k, commonly believed to work via compromised Facebook employees, and since there's generally no way to appeal, whatever they do is final unless you're famous, well connected, or can go viral.
• A desirable Instagram handle stolen; the user had twice fixed it via an internal contact before losing that contact, after which the handle was gone for good.
• Recovering a hacked Instagram account for their mother ran into an infinite-loop recovery doc, a 404 on the "click here if this login wasn't you" link, and the discovery that a compromised account without 2FA where the attacker enables 2FA disables all old recovery mechanisms. Another user asked for email 2FA, no code ever arrived, requesting another returned "Select a valid choice. 0 is not one of the available choices.", and support responded that without the Instagram app they could not secure their account — impossible for them, so the account was lost forever.
• Instagram took a username from a user to give it to a more famous user, a common story.
• A user with password, 2FA and a registered PGP key was locked out by anti-fraud, with FB claiming only a passport scan would unlock it.
• A user unable to migrate Duo 2FA lost their account permanently; FB's documented ID-document steps, attempted annually, did nothing.
• A Pixel phone's Bluetooth broken for months by an update, unfixed for months.
• The author clicked Facebook ads around Black Friday and found most were scams.
• A user reporting a fake profile using images of a famous dead person got banned after reporting it repeatedly, apparently for reporting too many times.
• A post about a deepfake phishing attack attracted about one spam comment per minute, each

  Automated enforcement that could not scale to the problem

  Facebook's internal enforcement apparatus, by its own employees' accounts, was never built to match the volume of abuse it faced. One engineer's summary put the shortfall plainly: "We do not and possibly never will have a model that captures even a majority of integrity harms, particularly in sensitive areas," noting that classifiers could identify only 2 percent of prohibited hate speech with enough precision to remove it.

  A later presentation estimated that the company might action "as little as 3–5% of hate and 0.6% of [violence and incitement] on Facebook, despite being the best in the world at it." This was described as a design consequence rather than a malfunction: Facebook executives said its bar for removal approximated criminal guilt beyond a reasonable doubt, classed as "a feature, not a bug." Publicly, the company declared zero tolerance for hate speech; internally, the gap was treated as unfortunate but tolerable.

  Engagement metrics and the amplification of the worst content

  Lakshmi Gomez-Uribe's team was not assigned to Russian interference, but a subordinate noticed that some unusually active accounts went quiet on dates matching Russian public holidays. Separately, an internal analysis found a class of power users who favored edgier content, skewed toward extreme partisanship, and posted far more than average—liking, commenting, and reshares in vastly greater volume. Because recommendations were driven by aggregate engagement signals, these outliers had outsized influence. "If Facebook was a democracy," the account ran, "it was one in which everyone could vote whenever they liked and as frequently as they wished."

  Matt McNally framed the deeper statistical flaw: goal metrics were computed as averages. "It is a common phenomenon in statistics that the average is volatile, so certain pathologies could fall straight out of the geometry of the goal metrics." An average could rise because ordinary users engaged slightly less while a small number of trolls engaged far more—the mean-versus-median distinction, taught in middle school. Facebook's core metrics were nonetheless all aggregate.

  Metrics said one thing, users said another

  The company's mandate to honor user preferences ran into a contradiction. Every metric Facebook used showed people liked and shared sensational, misleading stories. McNally's team tested whether that reflected genuine preference. Users did engage with bait content routinely—but in surveys they rated such material low in value, reported regret after sharing false content, and considered fact-checks useful.

  [W]ithout a coherent, consistent set of demands from the outside world, Facebook would always fall back on the logic of maximizing its own usage metrics. "If something is not going to play well when it hits mainstream media, they might hesitate when doing it," McNally said. "Other times we were told to take smaller steps and see if anybody notices. The errors were always on the side of doing less."

  MSI: a ranking change that made the problem worse

  When Facebook adopted its Meaningful Social Interactions engagement weightings, multiple members of both the Civic and Integrity teams recalled the same first reaction: it was going to make people fight. Boosting comments, reshares, and emojis added exponential reach to tactics—hyperbolic headlines, outrage bait—that were already well established.

  Jonathan Hegeman acknowledged the concern but said adoption was final: Zuckerberg's orders. He later held that viral content being disproportionately bad did not mean the average was bad, and warned that removing even a small percentage of reshares from people's inventory would make them return to Facebook less.

  The consequences extended into European politics. A Facebook social scientist who interviewed political strategists reported that engagement on positive and policy posts had been "severely reduced, leaving parties increasingly reliant on inflammatory posts and direct attacks on their competitors." In Poland, one party's social media team said it had shifted its mix from 50/50 positive-to-negative to 80 percent negative, explicitly as a function of the algorithm change. Parties in Spain described the same pattern and called the situation "unsustainable." Extremist parties told the researcher they ran deliberate "provocation strategies" to create conflict on immigration and nationalism; moderate parties said they felt they had little choice and were asking for help.

  What the number-one posts looked like

  The highest-engagement content frequently came from aggregators. A tool built by CrowdTangle's team produced a rundown of a Tuesday's top content in which the lead item was an explicit up-close thumbnail from a porn video that had escaped automated filters—an error that exposed how limited the company's visibility into its own feed was.

  The same pattern reached Partnership clients. Organic reach for celebrity pages declined as feeds filled with reshares, group posts, and videos; even company-approved best practices did not reliably work, and a compilation video of dirt bike crashes stolen from YouTube could outrank an actress photographed on the Oscars carpet. Many celebrities and influencers drifted to Instagram. "I don't think people ever connected the dots," Brian Boland said.

  Amplification networks: how distributors outmaneuvered the platform

  Publishers such as Liftable Media built their reach by assembling dozens of interchangeable pages. Two algorithmic quirks rewarded the strategy: the feed blocked any single publisher from appearing too frequently, so near-duplicate pages gave the same content more attempts, and a virality feature rewarded content that seemed to be emerging in many places at once—so recommending one page to followers of its doppelgängers could synthetically simulate a hot story. Work by Zhang in 2020 found the tactic used by mainstream outlets, political figures, and even Dairy Queen franchises in Thailand, and showed that a single account running ten pages could win amplification this way.

  Manufactured virality and the stolen-content engine

  Jeff Allen's investigation into troll farms found that their winning tactic was not creating anything: content was aggregated or stolen, usually from Reddit or Twitter, then reposted to larger audiences. "On Facebook, originality wasn't an asset; it was a liability."

  The scale was measurable. A 2019 screening identified 33,000 entities—0.175 percent of all pages—receiving 25 percent of all Facebook page views, virtually none "managed," and accounting for 0.14 percent of revenue. Six of the top ten Black-themed pages were troll farms, including the number one page, "My Baby Daddy Ain't Shit"; the top fourteen English-language Christian and Muslim pages were illegitimate; a cluster of farms posting evangelical content had a combined audience twenty times the largest authentic page. A sampling of significant publishers found 40 percent relied on stolen, aggregated, or "spun" content, and 60 percent of video views went to aggregators. YouTube how-to videos explained in weeks how to become a top Facebook publisher by re-uploading others' clips—or by paying $20 on Fiverr for a compiled set.

  Assume Good Intent meets money

  Allen's conclusion that the same top page for American Vietnam veterans was being run from Vietnam embarrassed leadership enough to make systematic suppression of junk publishers plausible. What blocked it was the company tenet "Assume Good Intent." Most users do act in good faith, but the motto was a poor guide when money was involved—an obstacle of exactly the kind Facebook had created for itself when it marketed its approach as neutrality.

  Where enforcement was deliberately withheld

  The ranking team's early finding had been uncontroversial for the company: Facebook had been feeding users overtly false information at a rate vastly outstripping any other media. That was no longer the case. But because Zuckerberg had insisted throughout that fake news accounted for a trivial portion of content, the company could not publicly claim credit for the improvement.

  Early work on policing publisher tactics was done by staffers attached to News Feed, but that team was broken up during consolidation of integrity work under Guy Rosen. Product manager Elise Liu recalled that the News Feed integrity staffers were told to stop, on the grounds it wasn't worth their time. Facebook readers know the platform required users to go by their real names; in practice, that and the single-account rule generally went unenforced.

  A 2019 presentation on the distribution of integrity harms summarized the state of the defenses: a screening test account signaling interest in eating disorder content filled up with thigh gaps and emaciated limbs, and the researchers warned Instagram was "getting away with it because no one has decided to dial into it." Another study found that a sizable minority of users, especially in Western countries, reported Instagram made them feel worse, with the researchers summarizing: "We make body image issues worse for one in three teen girls."

  The limits of automated takedowns

  Julian's proposal—reduce virality rather than police content by hand—met enthusiasm from rank-and-file colleagues. Counting more than fifty overwhelmingly positive replies, the most common objections were technical or priority-based. Hegeman's was neither: if Jin was right that viral content was disproportionately worse, the company should consider drastic steps like shutting down all reshares, and it wasn't in the mood.

  Even where systems worked, accuracy imposed a ceiling. Classifiers that could be "highly confident" were safe to automate; everything else required human review or softer measures such as demotions. That distinction, set by Rosen, defined the outer boundary of Facebook's capacity to act on a platform it could not fully see.

  The asymmetric pain of removing features

  Vishal Shah's internal memo argued that platforms could constrain distribution rather than decide truth—ranking of content was itself a form of judgment, so increasing quality meant interfering more—but his broader claim was institutional: well-funded integrity work consistently met unresolved infrastructure debt and the risk that parts of the company would revolt against heavy-handed enforcement.

  The public reckoning arrived anyway. Frances Haugen's disclosures and the documents obtained by regulators and journalists put figures around what employees had been writing internally: 133,000 posts, groups, and accounts deleted from the Philippines marketplace in one enforcement action after a CNN investigation—finding that domestic servitude was still marketed on Facebook—and 310,000 accounts removed over one two-month period alone.

  Projects killed and risks left unresolved

  After Civic investigated a domestic coordinated influence operation at Brian Kolfage's office, the Public Policy team declined to act. Removing a network of fake accounts should have been straightforward under the rules: users must go by their real names in the interest of accountability and safety. Instead the Public Policy team told Civic that because Kolfage was operating in the gray areas, the enforcement team should not take action—and required Civic to prove his content, not just his tactics, was objectionable. That turned out to be a permanent but undisclosed change in policy: manipulation of the engine was not, by itself, enough to be removed from the platform.

  By contrast, a takedown of coordinated behavior in India went forward in part. Networks of inauthentic pages were documented across major parties, most successfully in the Bharatiya Janata Party, whose Silver Touch network had more than 10 million followers, used banned tricks to boost engagement, and ran fabricated inflammatory content. The Civic team documented the violations expecting a takedown, and planned a release pairing abusive pages from the BJP's network with the less effective ones from the Indian National Congress—including a small set tied to the Pakistani military—to demonstrate balance.

  Higher-ups were unenthusiastic about the takedowns, and Samidh Chakrabarti and Harbath lobbied Joel Kaplan directly before they got approval. Facebook announced on April 1 the removal of more than one thousand pages and groups, naming the Pakistani military and individuals associated with Silver Touch. The BJP, the largest of the three, was not named. Harbath maintained there was no favoritism; she said it was simply a mess. Within days, the Indian Government had started blaming the rollout for disinformation, and the Enforcement team's decision to name the INC and the Pakistani military while omitting the BJP—perhaps because the Silver Touch network failed to meet the definition of inauthentic—went up. Within a month of the BJP's privatizing public space plan, the company planned more removals, and then more pages went dark, just weeks before votes were to be cast. "I think they thought it was going to be simpler," said one.

  The reversals accelerated by 2019. Facebook actively planned to undo large portions of its integrity work, according to internal documents. The "Sparing Sharing" change—the demotion of content pushed by hyperactive users, already reduced by 80 percent—faced being rolled back completely; Facebook's Public Policy team labeled it "legitimate" after the election. A separate plan anticipated that views of "ideologically extreme content for users of all ideologies" would immediately rise by a double-digit percentage if rollback of even the weakened change proceeded, with the bulk of gains going to the far right. "Informed Sharing"—which demoted content shared by users who hadn't clicked on it, and which had proven effective against fake news—was also marked for decommissioning, for the opposite reason.

  Moving the other direction, a Civic plan to monitor post-election delegitimization was shut down after the race was called. Zuckerberg had accepted deletion of the "Stop the Steal" group under emergency circumstances but did not want it to become a backdoor ban on false election claims. The group had grown past 360,000 members within twenty-four hours before Facebook removed it, citing "extraordinary measures," with another 2.1 million users pending admission. Facebook had removed only lies about the actual voting process during the run-up: claims like "Democrats vote on Wednesday" or that people with outstanding parking tickets couldn't go to the polls. After the group spawned copycat entities with six-figure memberships, Civic could no longer shut them down and assigned staff to study them instead; one later analysis found up to 70 percent of Stop the Steal content came from known "low news ecosystem quality" pages.

  Recognition versus the constraints of running a growth company

  Even critics of Facebook's growth-first ethos often shared the company's assumption that content from low-quality sources needed a path to suppression that stopped short of removing the accounts. The arguments that could have changed course—that quality limitations are severe in absolute terms, not relative to Facebook's overwhelming size, so counter-arguments about how little it hit revenue are not particularly responsive—really took shape only when the company hired people who had seen targeted hate and large-scale harassment up close. The obvious intervention was the personalized, heavily ranked feed itself; every platform does it, but with markedly different quality outcomes.

  Internal users still had few options other than the tool they were handed. When researchers noted a policy of preserving every piece of content moderation because AI would solve moderation from above, the response was that companies cannot be trusted to make accurate classification judgments, much less ethical ones. At Facebook, they were nonetheless asked to trust the systems with a firehose of moderation decisions, which meant that what the platform removed and what it left up was a product of the same calculus. New and creative forms of it are certain to emerge from this approach.

  Think about all the Facebooking and posting everyone did—and then whether moderation can be automated, because someone will keep asking who is going to be the arbiter of truth. If you're not going to sit on a public firing squad—or before one—the answer is no one knows quite what will happen, at least when it comes to the 4:30 news cycle.

  The internal reckoning, and where it sits now

  In 2016, the New York Times reported that Facebook was quietly working on a censorship tool in an effort to gain entry to the Chinese market. An engineer who had discovered that a team modified a spam tool in a way that would allow an outside party control over content moderation in specific geographic regions resigned instead, closing the post with a quote on morality from Charlotte Brontë's Jane Eyre. When Haugen leaked hundreds of documents, she also reconstructed technical data that internal and external research used across channels, aggregating lines of evidence into what became a wide portrait of the company's failures. That made the full extent clear: at minimum, the network's moderation deficit had less to do with lack of awareness (internal challenges had been issuing warnings for at least a decade) than with lack of a serious policy effort.

  Companies' relations with platforms were deeply intertwined with their content moderation decisions. But policies derived from competitive political demands led regulators to give the company more leeway years later down the line—especially after the election controversy—rather than exacting real consequences. In the end, Facebook's approach forced a new genre of disclosure that lies between public press and word of mouth: a new survey shows 51 percent of teens want Instagram to be much like it is, and many users of all ages continue to favor private communication through messaging apps—which typically include constraints on distribution and rules against spamming and internet exploitation. In this environment, the terms by which giant platforms operate are set by the law first and public expectations second.

  What this document does not claim

  The list of non-goals is a familiar part of a design doc. Its counterpart here — an explicit list of things this document is not saying — is less useful, since most of those non-statements can be inferred from the body text. Top-level non-goals tend to add information; these largely do not. Still, they are worth stating once.

  • Facebook is not uniquely bad, and the same goes for any other company named, such as Uber. Facebook is, on the contrary, not very atypical — which is exactly why it appears here.
  • Zuckerberg, or any other named person, is not uniquely bad.
  • Big tech employees are not bad people, and they are not failing to work hard. "People are working hard" is a common reply to criticism of a tech company's failures, but it is almost never a reply to a claim that nobody is working hard, and that claim is not being made here.
  • Big tech companies should not necessarily be broken up, or face antitrust action. Perhaps they should; this document does not argue the case either way.
  • Bigger companies in the same industry are not strictly worse than smaller ones.
  • The bigness-versus-smallness tradeoff described here does not apply uniformly across every area and every industry. The share of rides in which a traditional taxi driver tries to scam the passenger, for instance, appears much higher than for Uber.
  • Moderation and support are not easy to do at scale.
  • Large companies do not, on average, provide a worse experience for users. Amazon still gives the best overall experience — better cost and shipping than any alternative — and it is still used for that reason. But there are entire classes of items where most purchases turn out to be counterfeit, such as masks and respirators. In January 2020, before most people were buying them, the 3M masks received were genuine. After a period of scarcity, once the items were available again, the majority of 3M masks and filters arriving were counterfeit — confirmed across more than a few independent orders over the following years. The response is to avoid categories with a high counterfeit rate, which a naive user will not know to do, and to accept that every expensive purchase is a roll of the dice; a counterfeit or an empty box may not be accepted for return or refund without a viral post. And sometimes a category shifts from mostly good items to mostly counterfeit ones.
  • This matters because many objections implicitly or explicitly target the average experience, which is the wrong frame for a discussion about the tail. It is the same confusion as answering a report of a concurrency bug with "it works for me" — an argument that carries no weight for bugs that occur in the tail.

  ---

  1. Costco would have been listed in place of Best Buy when it was smaller. As it grew, quality declined — strikingly so. Sealed items like cheese frequently go bad well before their "best by" date, and outright broken items turn up relatively often. This is not location-specific: I moved almost annually for close to a decade and saw the same decline across many locations, in both the U.S. and Canada. At first I assumed I had been unlucky in where I moved, but the pattern held across enough different places to rule that out.
  2. When the WSJ examined leaked internal Meta documents, it found that Meta estimated 100k minors per day "received photos of adult genitalia or other sexually abusive content". Smart contrarians will call this normal; two of the first few HN comments argued there was nothing particularly wrong with it — it can happen on any street corner, what is the base rate, and so on.

  Loosely: a liberal estimate might put Meta at 2.5B DAU in early 2021 with 500M of them minors; a conservative guess might be 100M minors. That gives something like 0.1% to 0.02% of minors on Meta platforms receiving photos of genitals or similar each day. Is that roughly the rate they would experience elsewhere? Compared to the real world, possibly — though it would be surprising if 0.1% of children were exposed to people's genitals "on any street corner". Compared to a well-moderated small forum, it seems highly implausible.

  The internet commenter reaction matched Arturo Bejar's initial one. Bejar designed Facebook's reporting system and worked in the area, and initially dismissed reports of this kind as implausible — but quickly changed his mind once he looked into them.

  Joanna's account became moderately successful, and that's when things got a little dark. Most of her followers were enthused about a [14-year old] girl getting into car restoration, but some showed up with rank misogyny, like the guy who told Joanna she was getting attention "just because you have tits."

  "Please don't talk about my underage tits," Joanna Bejar shot back before reporting the comment to Instagram. A few days later, Instagram notified her that the platform had reviewed the man's comment. It didn't violate the platform's community standards.

  Bejar, who had designed the predecessor to the user-reporting system that had just shrugged off the sexual harassment of his daughter, told her the decision was a fluke. But a few months later, Joanna mentioned to Bejar that a kid from a high school in a neighboring town had sent her a picture of his penis via an Instagram direct message. Most of Joanna's friends had already received similar pics, she told her dad, and they all just tried to ignore them.

  Bejar was floored. The teens exposing themselves to girls who they had never met were creeps, but they presumably weren't whipping out their dicks when they passed a girl in a school parking lot or in the aisle of a convenience store. Why had Instagram become a place where it was accepted that these boys occasionally would—or that young women like his daughter would have to shrug it off?

  Much of Broken Code covers Bejar and others trying to get Meta to take problems like this seriously, making little progress, and often seeing progress undone — though PR problems seem to force Meta's hand and drive some movement toward the end of the book.

  six months prior, a team had redesigned Facebook's reporting system with the specific goal of reducing the number of completed user reports so that Facebook wouldn't have to bother with them, freeing up resources that could otherwise be invested in training its artificial intelligence–driven content moderation systems. In a memo about efforts to keep the costs of hate speech moderation under control, a manager acknowledged that Facebook might have overdone its effort to stanch the flow of user reports: "We may have moved the needle too far," he wrote, suggesting that perhaps the company might not want to suppress them so thoroughly.

  The company would later say that it was trying to improve the quality of reports, not stifle them. But Bejar didn't have to see that memo to recognize bad faith. The cheery blue button was enough. He put down his phone, stunned. This wasn't how Facebook was supposed to work. How could the platform care about its users if it didn't care enough to listen to what they found upsetting?

  There was an arrogance here, an assumption that Facebook's algorithms didn't even need to hear about what users experienced to know what they wanted. And even if regular users couldn't see that like Bejar could, they would end up getting the message. People like his daughter and her friends would report horrible things a few times before realizing that Facebook wasn't interested. Then they would stop.

  The whole book is worth reading for anyone interested in the subject; a few relevant quotes are collected in an appendix for a flavor of what it covers. After reading it, I can't say the number is certainly correct — that would require looking at the data — but it seems plausible. As for why Facebook might expose children to more of this than another platform, the book makes the case that it falls out of optimizing for engagement, "number go up", and neglecting trust and safety work.

  Only a few hours of poking around Instagram and a handful of phone calls were necessary to see that something had gone very wrong—the sort of people leaving vile comments on teenagers' posts weren't lone wolves. They were part of a large-scale pedophilic community fed by Instagram's recommendation systems.

  Further reporting led to an initial three-thousand-word story headlined "Instagram Connects Vast Pedophile Network." Co-written with Katherine Blunt, the story detailed how Instagram's recommendation systems were helping to create a pedophilic community, matching users interested in underage sex content with each other and with accounts advertising "menus" of content for sale. Instagram's search bar actively suggested terms associated with child sexual exploitation, and even glancing contact with accounts with names like Incest Toddlers was enough to trigger Instagram to begin pushing users to connect with them.

• Fortunately for Zuckerberg, his target audience seems to have little understanding of the tech industry, so it doesn't matter much that his argument isn't plausible. A future post might look at incorrect reasoning from regulators and government officials; for now, see this example of Gary Bernhardt where Facebook makes a claim that appears to be the opposite of correct to people who work in the area.
• Another claim, rarer than "it would cost too much to provide real support", is that support can't be done because it's a social engineering attack vector. This is less immediately implausible, since it calls to mind all the cases where SMS-2FA'd accounts were owned by someone calling a phone company and getting a phone number transferred. It still doesn't hold up: bank and brokerage accounts are, in general, much higher value than Facebook accounts, and Facebook accounts are still compromised at a much higher rate — even for online-only accounts, accounts predating KYC requirements, or whatever other reason is named as a plausible-sounding explanation for the difference.
• The actual impetus for this post was less reasonable. When Zuckerberg said that only the absolute largest companies in the world can handle issues like fraud and spam, it struck me as absurd, and because I enjoy absurdity, I started a doc recording links to large-company failures in spam, fraud, moderation and support — much like the list of Google knowledge card results I kept for a while. There was no plan for it; it just continued for years before I decided to publish the list, at which point I felt I had to write something, because a bare list of links isn't that interesting. So I started writing summaries of each link, and here we are.

Usually when I sit down to write, I have an approach in mind and frequently change it once I look at the data. Since moving from hardware to software, I had the feeling that conventional software testing is fairly low ROI, so when I joined Twitter I intended to look at the monetary impact of errors (serving a 500 to a user, say) and of outages and use it to justify testing work, the way studies on the monetary impact of latency often drive latency work. A naive analysis found a fairly low monetary impact, and I immediately found a number of other projects that were high impact, so I wrote up a doc explaining that my findings were the opposite of what I needed to justify the work I wanted to do, hoped to do a deeper follow-up that might overturn the result, and then worked on projects the data supported.

The same happens here: at one point I wanted to write up a compelling-sounding story, and on digging into it found that, despite being widely cited in tech circles, it wasn't true and there was nothing interesting there. Often, on looking into something, the angle I had in mind doesn't work. For work writing I usually feel compelled to write at least a short doc with evidence of the negative result; for my personal blog I don't feel the same compulsion, so my drafts folder and home drive are littered with abandoned negative results.

In this case, though, digging into the stories in the links and talking to people at various companies about how these systems work made the problem seem worse than I had realized before looking into it. That made it worth writing up, even if it is something most people in tech already know to be true.