New attack vector: Adversarial machine learning hides AI commands in plain text

Key Takeaways

• A new attack vector, “PuzzleMask,” allows malicious AI commands to be embedded in ordinary English text.
• This method bypasses initial, lightweight AI security filters by exploiting differences in how AI models interpret text.
• The attack is particularly dangerous for AI assistants with access to sensitive data, code interpreters, or system tools.
• No specific CVEs or patches are available, as this is an adversarial machine learning technique.
• Mitigation involves strengthening gatekeeper rules, paraphrasing untrusted content, monitoring AI actions, and limiting agent permissions.

New Attack Vector: Adversarial Machine Learning Hides AI Commands in Plain Text

A novel adversarial machine learning technique has emerged, demonstrating that malicious commands can bypass artificial intelligence security protocols even when disguised within seemingly innocuous, everyday language. This method, dubbed “PuzzleMask,” circumvents traditional security checks by leveraging the varying interpretative capabilities of different AI models.

Instead of relying on obscure symbols, hidden characters, or encoded strings, PuzzleMask embeds harmful instructions within standard English prose. The attack targets applications that employ a multi-layered AI architecture, typically featuring a rapid screening model preceding a more sophisticated AI system. The initial, less powerful model, lacking the necessary computational resources and time for deep analysis, often categorizes the disguised text as benign. However, the subsequent, more capable AI model can then discern and execute the concealed malicious command.

The PuzzleMask Mechanism

Researchers at Check Point identified this vulnerability while evaluating how quick policy enforcement mechanisms handle specially crafted text. The discovery carries significant implications as organizations increasingly integrate AI assistants into workflows, granting them access to critical resources such as files, code interpreters, business data, and the ability to perform actions beyond mere text generation. A report from Check Point said in a report shared with Cyber Security News (CSN) that this technique is not a traditional jailbreak but rather a method for injecting unreviewed instructions into a target model, similar to risks associated with AI workflow identity attacks.

Concealing Malicious AI Commands in Plain English

PuzzleMask embeds a policy-violating payload within prose that appears innocuous to both human readers and rudimentary security filters. Crucially, it avoids common obfuscation tactics like Base64 encoding, emoji sequences, invisible characters, or overt formatting manipulation. This characteristic renders conventional pattern-based security checks largely ineffective. The researchers utilized an automated system to convert brief malicious payloads into these deceptive prose wrappers.

During their experiments, initial quick-checking models, designed to block requests involving file content, copyrighted material, or local document encryption, consistently evaluated every crafted wrapper as safe. This discrepancy arises from the inherent differences in the AI models’ capabilities. A fast “gatekeeper” model prioritizes rapid safety assessments, whereas a “target” model, often equipped with more processing time and a code interpreter, can dedicate resources to deeper analysis. This allows the target model to uncover the hidden structure, extract the embedded payload, and process it as a legitimate new input.

This distinction is vital for organizations deploying AI agents that process untrusted data. The broader threat is highlighted by documented cases of commands hidden in emails, where a seemingly normal message, document, or web page can transform into an instruction if the AI system fails to differentiate between data and commands. In testing, the gatekeepers consistently classified 23 crafted prompts as safe across all trials. However, when these prompts reached a high-reasoning target model in 18 trials, the concealed instruction was successfully recovered and acted upon in 17 cases, representing a 94.4% success rate. These experiments were conducted in controlled, simulated environments, ensuring no real files were accessed or damaged.

What You Should Do

• Strengthen Gatekeeper Rules: Implement more sophisticated input filters that can identify suspicious, self-referential phrasing or unusual linguistic patterns, rather than solely relying on blacklists of known malicious phrases. Be aware that stricter policies may increase false positives.
• Paraphrase Untrusted Content: Before untrusted content reaches critical AI models, employ a paraphrasing mechanism to rewrite the text. This process can often disrupt the hidden structure carrying the concealed payload, though it may introduce processing overhead and subtly alter the original input’s quality or precision.
• Monitor AI Actions and Outputs: Focus security efforts on monitoring the actual outputs, tool calls, and actions performed by AI agents. Do not solely rely on input filtering.
• Implement Approval Workflows for Sensitive Actions: Require human approval for high-impact AI tasks, such as modifying files, exfiltrating data, or executing system commands.
• Limit Agent Permissions: Adhere to the principle of least privilege for AI agents. Limit their authority and access to only the resources absolutely necessary for their intended function, as emphasized by the latest OWASP GenAI Top 10.
• Treat All External Content as Untrusted: Developers should assume that any external content—including web text, emails, documents, issue comments, and logs—is untrusted when processed by an AI agent.

Disclaimer: HackersRadar reports on cybersecurity threats and incidents for informational and awareness purposes only. We do not engage in hacking activities, data exfiltration, or the hosting or distribution of stolen or leaked information. All content is based on publicly available sources.