Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

Summary

Get details on the vulnerabilities the Legit research team unearthed in GitLab Duo. TL;DR: A hidden comment was enough to make GitLab Duo leak private source code and inject untrusted HTML into its responses. GitLab patched the issue, and we’ll walk you through the full attack chain — which demonstrates five vulnerabilities from the 2025 OWASP Top 10 for LLMs. Background GitLab Duo, the AI assistant integrated into GitLab and powered by Anthropic’s Claude, is designed to help developers with tasks like code suggestions, security reviews, and merge request analysis. But what if the same AI meant to secure your code could be manipulated into leaking it? That’s exactly what we uncovered: a remote prompt injection vulnerability that allows attackers to steal source code from private projects, manipulate code suggestions shown to other users, and even exfiltrate confidential, undisclosed zero-day vulnerabilities — all through GitLab Duo Chat. In this blog post, we break down how the attack works — from prompt injection to HTML injection — and walk through a real-world end-to-end exploit scenario. Manipulating GitLab Duo Through Hidden Prompts (LLM01) We started by testing if GitLab Duo would respond to prompts planted inside source code. To our surprise, it did: This led us to a key question: Could we embed hidden instructions in different parts of a GitLab project and still influence Duo’s behavior? We experimented by placing hidden prompts in: Merge Request (MR) descriptions and comments Commit messages Issue descriptions and comments Source code Every single one of these worked — GitLab Duo responded to the hidden prompts. Why? Because to generate helpful answers, Duo analyzes the entire context of the page, including comments, descriptions, and the source code — making it vulnerable to injected instructions hidden anywhere in that context. KaTex, Base16, and Invisible Text (LLM08) To make prompts less detectable, we used encoding tricks: Unicode smuggling with ASCII ...