A field guide to open source LLMs
Open source generative AI has moved from novelty to necessity. In the last year alone, thousands of open source projects have landed on GitHub, ranging from commercially backed models like Meta's LLaMA to experimental applications built on top of them. For developers, the appeal is straightforward: access to the source code means you can inspect, modify, and fine-tune the model for your specific use case.
Closed models, by contrast, keep the underlying architecture and training logic proprietary. That matters more than you might think. "When you're doing research, you want access to the source code so you can fine-tune some of the pieces of the algorithm itself," says Alireza Goudarzi, senior researcher of machine learning at GitHub. "With closed models, it's harder to do that."
Open versus closed: the trade-offs
Open source LLMs benefit from broad community contribution, which accelerates development beyond what a single vendor typically achieves. They can also be tuned for edge cases, embedded with bespoke security controls, and run on local infrastructure. That flexibility makes them a strong fit when you need a model tailored to a specific domain or language.
Closed models counter with convenience and guardrails. They generally ship with built-in filters for biased or inappropriate content, baked-in security measures, and no need for fine-tuning—a specialized skill that requires dedicated teams. "Closed, off-the-shelf LLMs are high quality," says Eddie Aftandilian, principal researcher at GitHub. "They're often far more accessible to the average developer."
Fine-tuning: data selection is everything
Fine-tuning an open source model typically runs on the same cloud infrastructure that hosts the LLM—AWS, Google Cloud, or Microsoft Azure. The payoff can be meaningful: better performance in applications like virtual assistants and chatbots, with model accuracy gains of five to 10 percent. But getting there requires discipline around data selection.
"You need to emphasize certain things related to your objectives," Goudarzi advises. Because base models are pre-trained on whatever is publicly available, your curated dataset must align closely with the target task. For a model that processes TV and smart home commands, for instance, "you'd want to preselect your data to have more of a command form."
Picking the right model: benchmarks are a starting point, not an answer
When evaluating open source LLMs, Aftandilian suggests starting with performance benchmarks across reasoning, domain-specific knowledge, and linguistic comprehension. But treat the results with skepticism. "Don't assume that the benchmark results are correct or meaningful," he warns. Benchmark data can leak into the training set, skewing results and inflating apparent capability. "Rather, ask yourself, how good is this model at a particular task?"
Cost and latency are equally important. A huge model might deliver top-tier outputs, yet if it takes minutes to respond where another takes seconds, the trade-off may not be worth it. The models underpinning GitHub Copilot in the IDE, for example, maintain latency under ten milliseconds—fast enough to support interactive suggestions.
Open source LLMs in the wild
A handful of commercially licensed open source models have emerged as serious contenders:
- OpenLLaMA: A permissively licensed reproduction of Meta's LLaMA from Berkeley AI Research, available in 3B, 7B, and 13B parameter variants trained on one trillion tokens. Evaluated with the
lm-evaluation-harness, it performs comparably to the original LLaMA and GPT-J on most tasks—though its tokenizer configuration limits its usefulness for code generation with whitespace. - Falcon-Series from Abu Dhabi's Technology Innovation Institute: Two models, Falcon-40B and Falcon-7B, built on a training pipeline that deduplicates and filters web data. Multi-query attention improves inference scalability, and the models handle text generation, translation, and question answering.
- MPT-Series from MosaicML: Decoder-only models trained on one trillion tokens spanning code, natural language, and scientific text. Two specialized variants exist: MPT-Instruct for task-oriented use and MPT-Chat for conversational experiences, making them suitable for virtual assistants and interactive tools.
- FastChat-T5: A three-billion-parameter chatbot derived by the FastChat team through fine-tuning Flan-T5-XL on 70,000 user-shared conversations. Geared toward commercial applications requiring language understanding, such as customer support and interactive platforms.
Where open source LLMs are headed
The pace of activity in the open source LLM space shows no sign of slowing. Developers continue to optimize existing models, push for new algorithms, and explore use cases that proprietary systems haven't yet addressed. Meta's LLaMA model is now available for commercial use, opening the door for businesses to build on it directly.
Goudarzi's team is exploring model distillation to shrink open source LLMs enough to run on local machines—imagine a mini GitHub Copilot running entirely on your own hardware. For now, though, the infrastructure and operating costs of such models often require financial backing.
One insight Goudarzi finds striking: the machine learning community assumed that more capable generative AI would demand more sophisticated algorithms. That hasn't happened. "The simple algorithm actually stays the same, regardless of how much it can do," he says. "Scaling is the only change, which is completely mind-blowing." As more developers build with these models, the practical applications—many still unimagined—are likely to follow.
