Critical LiteLLM Supply Chain Flaw Exposes 2,500 Companies, 434,000 CI/CD Pipelines

Key Takeaways

• A supply chain attack compromised LiteLLM, an AI software component, for approximately 40 minutes, leading to the release of malicious packages.
• The incident potentially exposed over 2,500 organizations and 434,000 CI/CD pipelines to credential theft.
• Attackers leveraged a compromised Trivy scanner within LiteLLM’s build process to inject a credential-stealing payload.
• The payload, dubbed SANDCLOCK, targeted cloud keys, repository tokens, LLM API keys, and other sensitive data, with stolen credentials potentially remaining useful long after package removal.
• Immediate action, including credential rotation and environment rebuilding, is critical for affected organizations.

AI Supply Chain Attack Exposes Thousands of Organizations via LiteLLM

A critical supply chain vulnerability within the LiteLLM project briefly exposed thousands of organizations and hundreds of thousands of CI/CD pipelines to credential theft, highlighting the profound risks associated with relying on trusted AI software components. Although the malicious packages were only available for a short period—approximately 40 minutes—the incident demonstrates how even transient compromises can have lasting security implications, potentially jeopardizing build systems, cloud accounts, and source code.

The attack originated from a compromise in the release process of the Trivy scanner, a tool integrated into LiteLLM’s build pipeline. Crucially, LiteLLM’s build environment installed Trivy without strict version locking. This lapse allowed the attackers to inject poisoned code, subsequently leading to the generation and publication of malicious LiteLLM packages on the Python Package Index (PyPI).

Cloud security firm CloudSEK, which attributed the campaign to a threat actor known as TeamPCP, analyzed the activity. Their investigation said in a report that their reconstructed dataset indicated potential exposure for over 2,500 organizations and 434,000 CI/CD pipeline runs. CloudSEK emphasized that this exposure does not equate to confirmed breaches for all organizations, but rather a high-confidence match for potential impact.

How the Attack Unfolded

The malicious LiteLLM releases introduced a Python startup file (with a .pth extension) designed to execute automatically when Python initialized, rather than waiting for the LiteLLM package to be explicitly imported. This stealthy execution mechanism allowed the payload, identified as SANDCLOCK, to immediately begin harvesting sensitive credentials present in developer workstations and CI/CD build runners.

The stolen data included a wide array of high-value secrets: cloud keys, repository tokens, SSH keys, Kubernetes tokens, environment variables, and critically, LLM API keys and AI service gateway settings. This access could provide attackers a direct route into connected AI systems and the sensitive data processed within them. On a CI/CD runner, the credential-stealing malware aggressively sought elevated access and scoured process memory (/proc//mem) for these valuable secrets.

Researchers observed that the exfiltrated data was encrypted before being sent to a typosquatted domain. In instances where direct data transfer failed, the malware possessed a fallback mechanism: it could create a public repository within the victim’s GitHub account and upload the stolen material as a release asset. This technique makes detection significantly more challenging, as the apparent source of the data appears to be the victim’s own account.

CloudSEK’s report urged organizations to conduct thorough investigations to determine if the affected packages were downloaded, cached, or executed, rather than making assumptions about compromise or safety based solely on the exposure data.

What You Should Do

• Identify and Isolate: Immediately locate all installations of LiteLLM versions 1.82.7 and 1.82.8. Isolate any associated runners, hosts, container images, and caches that may have interacted with these versions.
• Rotate Credentials: Rotate every credential accessible to the affected processes. This includes, but is not limited to, cloud keys, repository tokens, SSH keys, Kubernetes tokens, and all LLM or AI model provider keys.
• Rebuild Environments: Reconstruct affected development and CI/CD environments from known-clean, trusted sources to ensure no lingering malware or backdoors remain.
• Audit Records: Scrutinize audit logs from cloud providers, source control systems (e.g., GitHub), package registries (e.g., PyPI), and Kubernetes for any unusual token usage, the creation of new service accounts, suspicious outbound network connections, or unexpected repository activity. Extend these searches to cover the period after package removal, as stolen access can be reused later.
• Implement Prevention: Pin all dependencies and GitHub Actions to verified hashes. Reduce the lifetime and scope of credentials, and prioritize the use of workload identity over static keys wherever feasible.
• AI System Security: Maintain an up-to-date inventory of AI assets and their owners. Monitor third-party dependencies used in AI pipelines and observe build-time behavior for anomalies.

Disclaimer: HackersRadar reports on cybersecurity threats and incidents for informational and awareness purposes only. We do not engage in hacking activities, data exfiltration, or the hosting or distribution of stolen or leaked information. All content is based on publicly available sources.