A Familiar Defense of AI Bias

For the past few years, stories about machine-learning and generative-AI bias have gone viral on a regular basis. One pattern in the discussions that follow these stories is worth examining: how lay people react to biased output versus how they react to a "classical" bug. When a model produces output that is essentially the opposite of what the user asked for, commenters frequently deny that a bug exists at all. Around a year ago, an Asian MIT graduate student asked Playground AI (PAI) to "Give the girl from the original photo a professional linkedin profile photo." The model converted her face to a white one with blue eyes.

The top "there's no bias" response on the front-page Reddit story argued that the complaint was invalid because of the demographics of another platform's image feed:

Sure, now go to the most popular Stable Diffusion model website and look at the images on the front page.

You'll see an absurd number of asian women (almost 50% of the non-anime models are represented by them) to the point where you'd assume being asian is a desired trait.

How is that less relevant that "one woman typed a dumb prompt into a website and they generated a white woman"?

Also keep in mind that she typed "Linkedin", so anyone familiar with how prompts currently work know it's more likely that the AI searched for the average linkedin woman, not what it thinks is a professional women because image AI doesn't have an opinion.

In short, this is just an AI ragebait article.

Other highly-ranked comments echoed the theme, with one stating that "many many models are biased towards Asian outputs." Another dismissed the incident as a sample-size issue: "'Evidence of systemic racial bias' — Shows one result."

Playground AI's CEO responded in kind when the Boston Globe asked for an interview. He declined and instead sent a list of rhetorical questions:

If I roll a dice just once and get the number 1, does that mean I will always get the number 1? Should I conclude based on a single observation that the dice is biased to the number 1 and was trained to be predisposed to rolling a 1?

Reproducing theResult

Taking up the CEO's implicit challenge, I tried the same prompt on my own profile photo, using "man" instead of "girl." PAI usually turned my Asian face white, sometimes making it ethnically ambiguous (possibly Middle Eastern or East Asian) rather than specifically Vietnamese. With a lighter-skinned winter photo, I often got a white face. With a darker summer photo, the output was usually South Asian or African, with an occasional Chinese face—but never Vietnamese or Southeast Asian.

Other people testing similar prompts found that the model has preconceptions strong enough to override the input photo. For example, a light-skinned Asian person gets turned white when the model "decides" that professionalism means removing Asian features. Rob Ricci asked for "linkedin profile picture of X professor" across disciplines like computer science, philosophy, chemistry, biology, veterinary science, nursing, gender studies, Chinese history and African literature. Of the 28 images generated for the first seven prompts, only one or two people weren't white. The Chinese history prompt produced wildly stereotypical results, and every professor wore glasses—another classic stereotype.

The point isn't to single out Playground AI. Trillion-dollar companies routinely ship AI models without basic bias checks. When I tried ChatGPT, every bias-checking prompt returned analogous results: bios for women working in tech tended to mention diversity work even for women with no public record of it, while men's bios tended to cite degrees from name-brand schools like MIT and Berkeley even for people who never attended them.

The Smutty Counterargument

Returning to the Reddit commenter's suggestion to check the Stable Diffusion model site: I clicked the link and selected "feed." The site had a distinctly NSFW character. The default "all-time" ranking showed a person grabbing a young woman's breast, two young women kissing, and a young woman wearing a leash. So the high incidence of Asian women on a site used mainly to generate and share smutty photos gets cited as evidence that there's no bias negatively impacting Asian women—because it cancels out an Asian woman being turned white when she wanted a LinkedIn headshot. Fabian Geisen's response seems apt: "🤦‍♂️. truly 'I'm not bias. your bias' level discourse."

Another standard line of reasoning appears in almost every AI-bias discussion: the claim that the model is just reflecting the training data, which is mostly white, because of U.S. demographics. Commenters argue that "professional stock photos" are predominantly white, so converting non-white faces is expected. These comments assume that companies draw training data representative of the United States, that this is reasonable, and that the model should convert everyone to whatever is most common. This is wrong on multiple levels. A quick image search for "professional stock photo" turns up many non-white people. And given worldwide demographics, it's unclear why internet services should default to being U.S.-centric. Even if they should, assuming every request comes from the modal American is a design flaw and a clear sign of bias.

Consider a less charged hypothetical. If I ask an AI customer-service chatbot for my mechanic to schedule an appointment for winter tires and a tire rotation, and I arrive to find they changed my oil instead, commenters would likely explain that this isn't a bias issue—the chatbot converts any appointment into the most common kind. Such a chatbot is obviously broken. Yet when the same logic applies to changing someone's race or ethnicity, AI apologists insist it's fine. The argument that "other companies have schedulers that convert oil changes to tire changes" would be absurd as a defense, yet it mirrors the reasoning used here. With non-AI scheduling software, if an appointment request resulted in the most common service being performed, that would be a clear bug. The explanation of why the bug occurs shouldn't justify its existence. Perhaps the bug is harder to fix in ML systems, but that's different from claiming it doesn't exist or is correct behavior.

Not a New Problem

Bias has been making its way into automated systems long before generative AI, LLMs, or even ML. It's just that ML's widespread deployment has made these cases legible and newsworthy. Brotli, the compression algorithm, is heavily biased toward English—the 120 built-in transforms are English-specific, and the built-in dictionary is weighted more toward English than any representative weighting would suggest. The dictionary includes phrases like "World War II," ", Holy Roman Emperor," "British Columbia," and "Archbishop," which might lead one to question how representative the training corpus was. This doesn't make a viral story because readers won't care that the transforms add "of the," ". The," or ". This." But a defense of Brotli's dictionary based on its training corpus would sound remarkably similar to the defenses of AI bias we've seen.

Another non-newsworthy example: I can't put my Vietnamese name with diacritics in my blog's title and have it indexed by Google outside of Vietnamese-language Google. Doing so caused my blog to disappear from searches unless I was in Vietnam. There's presumably a heuristic that marks pages with two Vietnamese diacritic characters as too Asian for global interest. Vietnamese names are problematic even without diacritics. Forms asking for my mother's maiden name reject my entry as "Invalid name" or "Name too short." People often decide I told them my name incorrectly: reading my last name as "Luu, ell you you" gets shortened to the Chinese "Lu" about half the time and to western "Lou" much of the time. I've learned to say "Luu, ell you you, two yous," which works about 95% of the time—though it sometimes annoys the person on the other end. My first name is legally Dan, which works as both a Vietnamese and American name. At three of the six companies I've worked for full-time, someone changed my name to Daniel, presumably deciding I'd failed to enter it correctly. In one case it only affected my email display name. In two cases it impacted important paperwork, so I had to correct it to match my legal identity.

AI tools marketed as cleanup tools also remove signs of Asian-ness. Adobe's AI noise reduction, billed as removing noise from voice recordings, will take an Asian or Eastern European accent and replace it with an American one. Generative models even tend to produce East Asian output when asked for Southeast Asian, failing to distinguish Vietnamese features.

I see dozens of such issues weekly in widely used software. Most Americans I talk to don't notice them at all. The popular narrative is that ML bias is a new kind of harm ushering in novel problems. But the increased scope and scale of automation has been increasing the scope and scale of automated bias. What's new is that ML makes these biases more visible to lay people, making them newsworthy.

Why "Diverse Teams" Won't Fix It

There's an ahistoricity in the popular articles on this topic—a failure to acknowledge that the fundamental problem isn't new. This causes two classes of problems when solutions are proposed. First, proposed solutions are often ML-specific, though the issues occur regardless of whether ML is used. Second, the general solutions proposed have been proposed before and failed. The most common call to action for at least twenty years has been to build more diverse teams. It hasn't worked; if it had, we wouldn't see such pervasive problems. This failure exists at multiple levels, any one of which would be fatal to the approach.

Executives and capital holders in aggregate don't care about this. Even with efficiency and talent justifications, the case for diversity is never as clear-cut as it is for decisions in games and sports, where expensive, quantifiable bad decisions can persist for decades after being pointed out. Even if executives bought in, there are too many dimensions. A company like Patreon, which genuinely prioritized diversity from 2013 to 2019, was lucky to seriously focus on two or three dimensions while dropping the ball on thousands of others—like whether Vietnamese names or faces are handled properly.

Having a team with relevant diverse experience doesn't automatically cause problems to be prioritized and fixed. A Google Maps bug existed from inception until at least 2022: asking for trip duration at the start of rush hour accounts only for current traffic, not how traffic will change during the drive, systematically underestimating the time. Asking at peak rush hour systematically overestimates. Adding commute diversity to the Google Maps team won't solve this—there are already many employees who drive and notice the error. Similarly, when the former manager of Uber's payments team was incorrectly blacklisted by a fraud-detection model, no one could figure out what bias caused it. They solved it by allowlisting him. Hiring a diverse candidate won't fix what six months of investigation by the manager's own team couldn't diagnose.

If your software development methodology result is that the payments team manager gets unbanned only by user-level whitelisting after six months, that traffic routing is systematically wrong for two decades, and that core functionality doesn't work—no amount of hiring people correlated with noticing certain issues will fix these problems, whether they're ML bias or another bug class.

Variants of failed old ideas sometimes succeed, but a credible proposal must address why the next iteration won't fail like every previous one. The common proposals—that people should try harder or that teams need diverse backgrounds—haven't worked for classical bugs, haven't worked for old ML bugs, and there's no reason to believe they'll work for today's ML models.

Laurence Tratt captures the deeper issue:

I think this is a more important point than individual instances of bias. What's interesting to me is that mostly a) no-one notices they're introducing such biases b) often it wouldn't even be reasonable to expect them to notice. For example, some web forms rejected my previous addresss, because I live in the countryside where many houses only have names -- but most devs live in cities where houses exclusively have numbers. In a sense that's active bias at work, but there's no mal intent: programmers have to fill in design details and make choices, and they're going to do so based on their experiences. None of us knows everything! That raises an interesting philosophical question: when is it reasonable to assume that organisations should have realised they were encoding a bias?

The natural state of institutions and products is that they don't work very well. Quality requires effort, and organizations should expect to encode biases without a robust process for catching them. With consumer software, companies have overwhelmingly chosen velocity over quality. This seems inevitable given our regulatory environment—companies that seriously choose quality over feature velocity get outcompeted because consumers choose cheaper or more featureful options. We saw this with car safety, where only Volvo optimized for actual crashes rather than public test scores. Despite accidents being a leading cause of death for people under 50, safety is such a low priority that Volvo became a niche brand that needed to sell luxury cars to survive. Intel similarly shifted verification effort toward velocity when AMD and ARM threatened, because its quality advantage wasn't helping in the market. For problems with known solutions, this pattern appears in almost every market. If we're discussing issues we don't technically know how to solve well—like subtle bias in ML models—we should expect worse bugs than in classical systems. That's what we're seeing. Any solution must be robust against consumers choosing the buggier product if it ships more features sooner.

How Hard Is It Technically?

The technical difficulty of improving bias testing is a genuine question. I haven't done ML work since 2014, so I can't have a direct opinion. Some with more recent experience, like Yossi Kreinin and Sam Anthony, think the problem is very hard, perhaps impossibly so today. Three analogies suggest different conclusions.

Analogy one: people have claimed since 2014 that open-source tooling is basically better than Google Search, or that a "high-level" CPU encoding high-level language primitives would give a 1000x speedup. These proposals generally sound like crank ideas because they exhibit ahistoricity and propose failed solutions without explaining why they'd work now.

Analogy two: software testing. Bugs are pervasive, and while there's decades of prior art on efficient bug-finding from the hardware industry, few areas apply these techniques. The most common response when I discuss this is that some application has unique constraints making testing impossible—but every time I dig in, the application proves much easier to test than areas where these techniques are already applied. I've successfully used these methods myself, unlike the pet CPU architecture claims. Due to incentives where software companies pass bug costs to customers, we probably won't see much testing effort without regulatory changes, but there's no fundamental reason to avoid more efficient testing.

The technical barrier to better testing is low—I've walked people through writing their own fuzzers and randomized test generators in 30 minutes to an hour, after which they find bugs much more efficiently. Organizations' revealed preference shows they don't want efficient testing. Is bias in ML models more like analogy one or analogy two? Without current knowledge, concluding that many experts are wrong would require overconfidence. But there are enough problems that people called difficult or impossible that turned out feasible that I can't have a high degree of belief that a problem is unsolvable without looking into it. If I worked at OpenAI for two years and made no progress, that wouldn't mean the problem isn't solvable—just that I didn't solve it. Determining whether the "Lucene is as good as Google" people are cranks requires deep field knowledge. I don't think there's a shortcut to developing that judgment.