A new season of the Secure Code Game puts LLM vulnerabilities on the workbench

GitHub has launched the third season of its Secure Code Game, a free, hands-on security course that now focuses on the risks introduced by AI. This season, players first act as attackers, crafting malicious prompts to exploit weaknesses in LLM-powered code, then switch to defense by patching the vulnerabilities they found.

The Secure Code Game is designed for developers of all skill levels. Instead of static tutorials, it embeds challenges directly in the code editor so players practice spotting and fixing real vulnerabilities in the environment where they work. Set-up is simple: players can get started in under two minutes with GitHub Codespaces. Since the first season launched in March 2023, more than 10,000 developers from enterprise, open source, and education communities have participated.

Season three presents six progressively harder challenges, each focused on a different defensive technique for securing AI-powered applications. Players begin by examining system messages and code for gaps or edge cases that could expose a hidden secret when sent a crafted prompt. Once they have successfully breached the model, the task turns to hardening the code and system prompt against such attacks—while preserving the intended functionality.

Topics covered across the challenges include:

  • Robust system prompt design: Writing initial instructions that set the model’s role, constraints, format, and context to guarantee safe and relevant outputs.
  • Output validation: Checking model output against predefined rules and formats to prevent sensitive data leaks.
  • Input filtering: Examining and modifying user input before it reaches the model to block harmful or irrelevant content.
  • LLM self-verification: Instructing the model to review its own answers for accuracy, consistency, and policy compliance—either through direct prompts or built-in reasoning during generation.

Putting realism first

Season three grew out of a collaboration sparked at FOSDEM 2025, where GitHub’s Joseph introduced the Secure Code Game to open source maintainers. In the audience was Bartosz Gałek (@bgalek), creator of HackMerlin, a game that tests prompt-injection skills. HackMerlin became the foundation for the new season’s challenges.

Adapting HackMerlin, which was UI-centric, to the Secure Code Game’s code-first format required some realignment. HackMerlin’s backend masked the complexities of LLM integration through a UI gateway, while season three takes full advantage of GitHub’s developer tools: Codespaces preconfigures the environment and automates access to GitHub Models, which provides built-in LLM integration as a catalog and playground of AI models.

Another shift: HackMerlin used an OpenAI model hosted on Azure, whose default safeguards could be altered or disabled by the game’s creator. For season three, the developers intentionally kept the default safeguards of GitHub Models to better reflect real-world conditions.

Players can also switch between different models on GitHub Models to compare how they behave under the same attacks.

Start playing

The Secure Code Game repository contains instructions for this and all previous seasons. New challenges for season three are found in the Season-3 directory.

The project is open to community contribution as well: those who want to create future challenges can consult the contribution guidelines.

Players who have gone through earlier seasons describe a real shift in how they view code. One participant noted he originally saw the provided code as perfectly normal, with standard unit tests, until the vulnerability was revealed right in front of him—making him far more cautious about identifying subtle flaws. Others highlight the value of learning to find issues before shipping, and the benefit of a format that emphasizes code concepts rather than any one programming language.

Ready to test your prompting skills? The challenges of season three are open now.