Demystifying LLMs: How they can do things they weren’t trained to do

The mechanics behind surprising LLM behavior

Large language models (LLMs) have transformed how software handles natural language, but they also raise questions about accuracy, bias, and the occasional generation of false or fabricated information. One common assumption—that these models reason about the content they process—needs correcting. As Alireza Goudarzi, senior researcher of machine learning for GitHub Copilot, puts it: “LLMs are not trained to reason. They’re not trying to understand science, literature, code, or anything else. They’re simply trained to predict the next token in the text.”

That single objective drives everything these models can do, including behaviors their creators did not explicitly plan for.

Deep learning and how LLMs generalize

LLMs belong to a class of AI systems trained on enormous text corpora. Unlike earlier language tools that followed predefined grammatical or logical rules, LLMs use deep neural networks with many layers to detect complex relationships in language. “The entire discipline of deep learning and neural networks—which underlies all of this—is ‘how simple can we make the rule and get as close to the behavior of a human brain as possible?’” says Goudarzi. John Berryman, also a senior researcher of machine learning on the GitHub Copilot team, explains that these deep learning techniques let models “learn complex patterns and relationships,” enabling them to go beyond fixed rules and adapt to context.

That flexibility comes from two training stages. Pre-training gives the model broad exposure to language. Fine-tuning then adapts it for specific tasks. Goudarzi describes the process: “When developers want these models to perform specific tasks, they provide task descriptions and examples (few-shot learning) or task descriptions alone (zero-shot learning). The model then fine-tunes its pre-trained weights based on this information.” The result is a model that retains general knowledge while becoming specialized—able to handle idiomatic phrasing, intricate sentence structures, and subtle linguistic nuance without explicit instructions for every case.

Why LLM output can be wrong

Several known limitations explain why the same capabilities that make LLMs impressive also produce unreliable output. These failure modes are worth understanding regardless of which LLM you use.

Outdated training data

LLMs have no live connection to the world. They rely exclusively on the text they were trained on, and that data ages quickly. “Typically this whole training process takes a long time, and it’s not uncommon for the training data to be two years out of date for any given LLM,” says Albert Ziegler, principal researcher on the GitHub Next team. As a result, models can confidently describe a reality that no longer exists. Fact-checking LLM responses remains essential.

Ambiguity and missing information

Models answer based on the input they receive. When a prompt is vague or lacks key details, the LLM fills in the gaps with its own assumptions. Those assumptions may be plausible but incorrect.

Biased training corpora

Training data is typically drawn from public sources—books, websites, articles, and social media. That means the model absorbs the biases present in human writing. Ziegler notes the problem is not unique to LLMs: “What machine learning does is identify patterns, and things like stereotypes can turn into extremely convenient shorthands. They might be patterns that really exist, or in the case of LLMs, patterns that are based on human prejudices that are talked about or implicitly used.” Responses on sensitive topics can therefore reflect those shortsighted patterns.

Overconfidence and hallucinations

LLMs cannot evaluate whether what they generate is true. Their deep learning objective pushes them to produce text that flows naturally and sounds plausible, with little to no calibration for certainty. This lack of self-assessment leads to overconfident errors.

Hallucinations, where the model generates unsupported content, are a specific version of this problem. Faced with an incomplete prompt, the model completes it by combining learned patterns, sometimes arriving at confident but false conclusions. Ziegler describes what that looks like in code generation: “The typical hallucinations we encounter are when GitHub Copilot starts talking about code that’s not even there.” The mitigation for Copilot, he explains, is to “give enough context to every piece of code it talks about that we can check and verify that it actually exists.”

But hallucinations can also be turned into a design tool. Johan Rosenkilde, principal researcher for GitHub Next, envisions using them deliberately: “Ideally, you’d want it to come up with a sub-division of your complex problem delegated to nicely delineated helper functions, and come up with good names for those helpers. And after suggesting code that calls the (still non-existent) helpers, you’d want it to suggest the implementation of them too!” Here the model’s invented placeholders become scaffolding for solving larger problems.

Using LLMs responsibly

False output from an LLM is not a symptom of malicious intent—it is a natural consequence of the training process. Berryman summarizes the situation: “The engines themselves are amoral. Users can do whatever they want with them and that can run the gamut of moral to immoral, for sure.”

Building guardrails into LLM products is an active area of work. Goudarzi outlines several safeguards the GitHub Copilot team employs: “First, we have a duplicate detection filter, which helps us detect matches between generated code and all open source code that we have access to, filtering such suggestions out. Another tool we use is called Responsible AI (RAI), and it’s a classifier that can filter out abusive words. Finally, we also separately filter out known unsafe patterns.”

At the user level, better results come from better questions. Precise prompts, verification against authoritative sources, and critical review of outputs all help counter the inherent weaknesses of next-token prediction. An understanding of how these models learn explains both their strengths and their blind spots—making them more predictable and more useful tools.