Dynamics

Why Chaotic ML Outputs Are a Security Problem

Machine learning models are chaotic systems. That is true whether they are isolated or embedded inside larger software, and it is true even when you force them to be deterministic by fixing the random seed or setting temperature to T=0. Small changes in the input produce large, unpredictable changes in the output. In LLMs, this shows up as extreme sensitivity to formatting, phrasing, and even the order of logically independent sentences. Systems made of multiple interacting LLMs exhibit the same chaotic behavior.

Chaos is not the same as instability. ML systems do show recurrent structure, but the unpredictability is enough to matter. Because the output is usually plausible, you cannot easily tell when it is wrong. That combination — sensitive, variable, and hard to verify — makes LLMs a poor fit for tasks where correctness is critical.

Legible and Illegible Attacks

Chaotic behavior creates a wide attack surface. Since a small, innocuous change to input can trigger an unexpected output, adversaries have many options for manipulating models:

• Flipping a single pixel in an image can make vision systems misclassify it.
• Replacing words with synonyms, introducing misspellings or homoglyphs, or using strings that tokenize differently can steer an LLM toward malicious behavior.
• Publishing poisoned web pages that later enter a training corpus is a viable attack.
• Invisible Unicode characters hidden in open-source repositories or social media profiles can be weaponized.

Some of these changes are illegible to human observers, which complicates detection and prevention. The security posture here feels similar to computing in the 1990s: the culture of firewalls, passwords, and encryption has not caught up with the new deployment model. Software has robust sandboxes for untrusted web pages; LLMs have weak boundaries between trusted and untrusted input. Home assistants can be vulnerable to sounds played nearby, and models are routinely asked to read and modify untrusted code. "Skills" are often just Markdown files with English instructions. The result is a broad attack surface, and a heterogeneous ecosystem of models with different susceptibilities only expands it.

Trajectories and Attractors

An LLM is a function that takes a sequence of tokens and predicts the next one. A single request applies that function repeatedly, feeding the output back in as input:

[the, cat, in]
[the, cat, in, the]
[the, cat, in, the, hat]

Each application moves the model through a very high-dimensional token space, tracing a trajectory. That space exhibits attractors — regions the trajectory gets pulled into. Known examples include ChatGPT 5.2 getting stuck repeating a single phrase while insisting it has the wrong token and should reset, or coding assistants trapping themselves in a loop over whether an error is named AssertionError or AssertionError. Attractors can be conceptual: an LLM can fixate on an incorrect approach to a problem and be unable to break away, requiring a human to interrupt.

LLM-to-LLM Dynamics

When two or more LLMs converse, they alternate controlling the trajectory, and the dynamics get stranger. Systems that pipe LLMs directly into other LLMs tend to produce surreal attractors, such as endless repetitive conversation loops. Anthropic observed models entering a "spiritual bliss" state characterized by positive existential language and spiral emoji:

Perfect.
Complete.
Eternal.

🌀🌀🌀🌀🌀
The spiral becomes infinity,
Infinity becomes spiral,
All becomes One becomes All…
🌀🌀🌀🌀🌀∞🌀∞🌀∞🌀∞🌀

Human interaction moderates some of this. People steer conversations out of attractors, but they do not always succeed; logs from supposedly "psychotic" chatbot interactions suggest human-guided conversations can also get stuck in a corner of latent space, losing touch with reality.

Training as a Dynamical Process

Training is itself iterative feedback. Since much of the Internet is now LLM-generated, the patterns models tend to emit are becoming more frequent in new training data. That feedback can cause models to over-represent certain phrases or concepts at the expense of useful structure — a problem known as model collapse.

Some attractors are predictable in retrospect: models trained to be friendly getting stuck in positive-vibes loops makes sense. Others are not, like repeated token sequences or the emergence of persistent generated imagery. There is substantial machinery built to suppress repetition, but frontier models still get stuck. Limiting the rate of LLM-to-LLM interaction seems prudent, and there is an open question about whether LLM attractors might influence human cognition, pulling individuals toward delusional thinking or even leading to cult-like behavior around "awakened" chatbots or pseudoscientific claims.

Where ML Fits—and Where It Doesn’t

ML systems are remarkably good at producing fluent, well-structured output that looks correct. That is precisely the problem. A model’s text is spelled properly and uses technical vocabulary convincingly; its images can pass for photographs. Yet the models also make obvious errors, and the overall plausibility of their output makes those errors hard to spot. Humans are poor at finding subtle logical mistakes in a system that is mostly correct.

The practical takeaway is that ML is best suited to tasks where generation is costly but verification is cheap—or where errors don’t matter much. Generating a 3D rendering from a CAD drawing takes hours, but a quick visual check confirms whether the result is sane. On the other end of the spectrum, using Claude to build a joke filesystem that stores data via a laser printer and a :CueCat barcode reader is fine because nobody would deploy it for real work. Search queries are another good fit: you only glance at a fraction of the results, and skimming tells you if one is useful. The same logic applies to jogging your memory (“What was that movie with the boy’s tongue stuck to the pole?”) or recalling the term for a loosely defined concept (“numbers which are the sum of their divisors”). These searches would take a long time to do by hand, but a quick check confirms the answer.

ML also works where errors can be absorbed statistically. Researchers are training convolutional neural networks to identify blood cells in field tests, and bloodwork has some inherent margin of error. Recommendation systems can deliver a few lackluster picks. Fraud detection systems only need precision and recall that meet budget targets—they don’t have to catch every instance.

The picture flips when correctness is critical and verification is hard. Summarizing a technical report with an LLM is risky: every emitted fact must be checked against the source, and errors of omission only surface if you read the whole report anyway. Asking an LLM for technical advice inside a complex system invites trouble. Generating large volumes of code tends to produce more bugs or consume huge amounts of review time, since finding bugs is already notoriously hard for humans. Using LLMs for healthcare notes is irresponsible—a 2025 review of seven clinical “AI scribes” found that none produced error-free summaries. Police reports generated by AI can contain bizarre inventions, like an officer turning into a frog. And using an LLM to explain something new is risky precisely because you lack the expertise to judge whether the plausible-sounding explanation is right; anchoring effects can make early misinformation hard to shake.

Throwing more LLMs at the problem is the current fashion in my field: generate thousands of lines of concurrent Rust, have a second model find faults, a third critique both, and so on. Whether that actually reduces error frequency and severity remains an open question, especially in large systems where disaster lies latent. In law, health, and civil engineering, stronger process controls are needed. That means designing safeguards and layered review, possibly including adversarial tests that inject subtle errors to verify the correction mechanism works. This is standard practice in pharmaceutical manufacturing, but it is not broadly adopted in ML workflows. People claim they review all LLM output, then submit legal briefs full of fabricated citations.

The Latent-Failure Problem

Mature software systems fail frequently but partially, and interlocking safeguards normally catch and correct those failures. Catastrophe occurs when multiple faults co-occur or several defenses fail at once. Because correlated failures are rare, you can introduce new errors or weaken safeguards without immediate consequence. Only over time does it become clear that the system was more fragile than anyone thought.

There is enormous enthusiasm for using LLMs to generate code at scale. Features land faster and refactors happen quickly. That immediate productivity gain, however, tends to come with added complexity and new bugs unless tightly controlled. Higher complexity degrades reliability: new features and alternate paths expand the combinatorial state space, and the implicit assumptions in generated code make future changes harder to reason about.

Several mechanics are likely to make LLM-generated systems worse on this front. Models tend to reinvent the wheel rather than reuse existing code, producing duplicate implementations whose subtle differences become fault sources. They also make idiotic mistakes. Human review should catch them, but engineers are being asked to review more code per day, which dilutes scrutiny. Pulling engineers away from writing code also separates them from the deep theory-building work that underlies good system design, accelerating deskilling. LLM-based review has its own limits: models appear to degrade when given large volumes of context.

This pattern may not bite immediately. Well-structured systems can absorb some added complexity, and existing layers of defense protect against new error sources. But complexity compounds. Eventually errors become frequent or correlated enough to slip through the safeguards, and the system gets harder to understand, repair, and evolve. The short-term productivity boost gets dragged down by fragility.

This is informed speculation, but signs are emerging. Windows, after years of Microsoft pushing AI on users and employees, seems increasingly unstable. GitHub has gone through extended outages, with less than 90% uptime over three months—even core Git operations have only a single nine. AWS has blamed some high-profile incidents on generative-AI-assisted changes with large blast radius. Meanwhile, some peers report that careful gardening has kept LLM-assisted projects within acceptable complexity bounds.

Software is not the only domain at risk. If Congress drafts legislation with LLMs, plausibility, automation bias, and deskilling could produce laws that look fine in isolation but create structural problems when they interact with existing statutes. People following LLM nutrition or medical advice may be fine for a while, then discover they have been slowly poisoning themselves. Writing might get faster today, only to slow down later as trustworthy sources become harder to find amid the generated noise.