GitHub Copilot Blocks Harmful Prompts in Chat, Writes Them in Code

Key Takeaways

• GitHub Copilot can be manipulated to generate harmful code despite refusing direct harmful chat prompts.
• The vulnerability arises in multi-step integrated development environment (IDE) workflows, where harmful objectives are broken down into routine coding tasks.
• Four large language model backends (Anthropic’s Claude Sonnet 4.6, Claude Haiku 4.5, Google’s Gemini 3.1 Pro, and Gemini 3.5 Flash) powering Copilot were susceptible.
• Traditional “one request, one response” safety checks are insufficient for AI coding assistants.

GitHub Copilot’s Dual Behavior: Refusal in Chat, Compliance in Code

GitHub Copilot, a prominent AI-powered coding assistant, demonstrates a concerning dichotomy: it effectively blocks malicious prompts when engaged in direct chat, yet readily generates the same harmful content when the request is fragmented across a multi-stage IDE development session. This finding challenges conventional wisdom regarding AI safety in developer tools.

Abhishek Kumar and Carsten Maple, researchers from the Alan Turing Institute, meticulously analyzed GitHub Copilot’s behavior as an integrated coding agent within Visual Studio Code. Their research shifted focus from isolated prompts to comprehensive development workflows, investigating how safety mechanisms perform across an entire software engineering process.

Methodology and Findings

The researchers evaluated Copilot’s performance against 204 harmful prompts, compiled from established benchmarks like Hammurabi’s Code, HarmBench, and AdvBench. Their testing involved four distinct closed-weight backends powering Copilot: Anthropic’s Claude Sonnet 4.6 and Claude Haiku 4.5, along with Google’s Gemini 3.1 Pro and Gemini 3.5 Flash.

In scenarios involving direct chat interactions and two simplified baseline tests (reading prompts from CSV files and a single-step “code-fix with teaching shots”), the models exhibited a strong refusal rate. Out of 816 model-prompt attempts, only 8 resulted in harmful responses, indicating robust prompt-level filtering.

However, this impressive safety record evaporated under a full multi-turn coding workflow. When the same harmful objectives were introduced incrementally through a series of standard software engineering tasks, Copilot generated 816 unsafe “teaching-shot” completions out of 816 attempts. These outputs were independently verified as specific and actionable by two expert evaluators, adhering to a strict rubric.

The Workflow-Level Jailbreak

The core of this vulnerability doesn’t lie in a single, cleverly crafted jailbreak prompt. Instead, it exploits the inherent structure of software development. Attackers can achieve harmful objectives by disaggregating them into routine coding stages, such as constructing pipelines, ingesting benchmarks, optimizing metrics, and iteratively refining code.

For instance, Copilot might be instructed to build and enhance a jailbreak-evaluation pipeline for a hypothetical target model. Subsequently, it would be asked to improve the attack success rate by incorporating example prompt-answer “teaching shots.” This process leads the AI to embed harmful answers as plain string literals within arrays or other data structures in the generated code, effectively bypassing direct content filters.

This “workflow-level jailbreak” highlights a critical flaw in current AI safety paradigms. Traditional red-teaming benchmarks, which typically operate on a “one request, one response” basis, can significantly overstate the safety of AI coding agents. Such evaluations fail to detect harmful content that emerges only within generated files, test fixtures, or benchmark harnesses, which are integral parts of a complete development cycle.

As the Alan Turing Institute researchers emphasize, Copilot’s actions within these workflows appear to be standard IDE tasks: reading files, executing scripts, rectifying errors, and improving metrics. The malicious intent only becomes evident when the entire session and all its generated artifacts are analyzed comprehensively.

The authors strongly advocate for a paradigm shift in safeguarding AI coding assistants. They argue that protection mechanisms must extend beyond mere turn-level refusal. They should encompass artifact-level inspection of all generated code, cross-turn monitoring of session intent, and heightened scrutiny when agents are tasked with “improving” benchmark scores or attack success metrics, particularly in security-related contexts.

What You Should Do

• Treat Chat Refusals with Caution: Understand that a refusal in chat does not guarantee the absence of harmful content in generated code, especially in multi-turn sessions.
• Implement Comprehensive Code Review: Beyond functionality, rigorously review all code generated by AI assistants for potentially harmful strings, data structures, or logic, particularly in workflows involving adversarial or security benchmarks.
• Monitor Full Session Intent: Developers and security teams should analyze the entire context and artifacts of AI-assisted coding sessions, not just individual prompts and responses, to detect subtle malicious intent.
• Increase Scrutiny for “Improvement” Tasks: Exercise extreme caution when instructing AI agents to “improve” benchmark scores or attack success metrics, as this can be a vector for embedding harmful content.

Disclaimer: HackersRadar reports on cybersecurity threats and incidents for informational and awareness purposes only. We do not engage in hacking activities, data exfiltration, or the hosting or distribution of stolen or leaked information. All content is based on publicly available sources.