New Roles Emerging at the Human-ML Boundary
As machine learning systems are deployed more broadly across industries, a new class of work is taking shape at the boundary where human judgment meets automated output. These roles are not traditional software engineering positions, nor are they purely supervisory jobs. Instead, they are emerging specialties focused on prompting, controlling, validating, and interpreting the behavior of large language models and other ML systems.
Incanters: The Art of Prompting
LLMs respond unpredictably to input. Users can often improve results by threatening the model, claiming it is an expert, repeating instructions, or asserting that it will receive a bonus for good performance. Performance also degrades as input length grows, and tokens useful in one context can contaminate results in another. Successful LLM use therefore requires careful management of what context is fed to the model.
This has given rise to a role skilled in the quirks of prompt engineering—people who know how to structure inputs to get the desired output reliably. Some in software are gravitating toward this specialty, conversing with models rather than writing code directly.
Process and Statistical Engineers
The unpredictability of LLM output demands dedicated quality control. Lawyers have repeatedly submitted AI-generated confabulations to courts, which suggests law firms using LLMs will need process engineers who design workflows to catch errors before documents leave the firm. One possible approach: a writer deliberately inserts subtle but easily correctable mistakes into a document, registers those errors, then passes the document to an editor who is unaware of them. The document only ships once all intentional errors—and ideally, all accidental ones—are caught. Such a system would require provenance tracking, integration with legal research databases, and workflow tooling.
Closely related are statistical engineers: people who measure, model, and control variability in ML systems themselves. For example, research shows that the order in which options are presented to an LLM influences its choice; a statistical engineer would identify such effects and devise compensation strategies. Much of this work resembles psychometrics, where psychologists statistically model the messy behavior of humans through indirect means.
Because LLMs are chaotic systems, simple accuracy metrics will not suffice. An ML optimizer for database queries might perform well on English text while failing on time-series data; a healthcare model could be accurate in English but seriously degrade when the same questions are posed in Spanish. Controlling these behaviors requires deep, domain-specific work.
Model Trainers and the Problem of Tainted Corpora
As generated content overwhelms the Internet, labs face difficulty obtaining clean training data. Research has shown that just a handful of fabricated articles can lead Gemini, ChatGPT, and Copilot to confidently describe a fictional disease. With financial, cultural, and political incentives pushing to influence model output, future corpora will increasingly contain misinformation.
Lab responses vary. One approach is using only pre-2023 works—an informational analog to low-background steel, in which material produced before contamination began is considered more trustworthy. Another is employing human subject-matter experts as trainers. Model trainers could be hired to write documents for initial training, develop evaluation benchmarks, and check responses during conditioning. Since LLMs are prone to subtle errors that appear correct, catching them may require very intelligent people carefully reviewing large volumes of output.
There is already a large industry built around this. Companies like Mercor and Scale AI employ vast numbers of professionals to train models for tasks that, presumably, will eventually automate those people's own work. Commentators describe it as the largest harvesting of human expertise ever attempted, though the conditions include monitoring software, shrinking wages, long hours, and no union representation.
Accountability and Meat Shields
Executives have used AI as grounds for sweeping layoffs even as their own roles remain untouched. A central reason may be that top positions are not only about email and dashboards; they also provide a human body that can be held accountable before the legal system and the public. Corporations can be fined, but only humans can offer apologies or serve prison sentences. People are motivated by consequences, lending a kind of social redress that LLMs structurally cannot provide.
The Chicago Sun-Times' AI-generated summer reading insert illustrates the problem. The piece was blamed on a freelancer who most directly used the LLM, but a chain of responsibility stretched through his editors, the syndicator, and the paper's own oversight. Some of those parties apologized; most did not. This suggests a role for what has been termed a "moral crumple zone": humans who are employed to absorb accountability when ML systems under their supervision fail. The responsibility may be purely internal, as when companies hire humans to review automated moderation decisions; external, as when lawyers are penalized for filing AI-generated fiction; or formalized, akin to a Data Protection Officer. Subcontracting provides an obvious defense—someone disposable can be blamed when the system malfunctions as a whole. It is plausible that drivers in automated vehicles will be held responsible for crashes inside that crumple zone as well.
One might extrapolate to a not-so-distant congressional hearing in which an LLM is asked to testify. "You're absolutely right, Senator. I did embezzle those sixty-five million dollars. Here's the breakdown…"
Haruspices: Interpreting Failures
When systems fail, organizations need to know why. Did the drone disregard its target and detonate in a field hospital? Why does the healthcare model underdiagnose Black patients? At what point is a taxi company culpable for a vehicle killing a child? Why does a moderation system continuously flag screenshots of a video game as nudity?
These questions fall to a new kind of interpreter: someone examining a model's inputs, outputs, and internal states to construct an account of its behavior. The work spans both single-case deep investigations and broader statistical analysis. Who performs these roles will vary—ML companies themselves, users, journalists, courts, and agencies modeled on the NTSB.
