I’ve been processing the LiteLLM supply chain attack all week and I think we need to talk about this as an industry. Not just the incident itself, but what it reveals about the trust model we’ve been running on with our AI tooling.
What Happened
On March 27, a threat actor group called TeamPCP used stolen PyPI publishing tokens to push malicious versions of LiteLLM (1.82.7 and 1.82.8) to the Python Package Index. The payload was a three-stage credential stealer—embedded in a file called litellm_init.pth—that harvested AWS/GCP/Azure credentials, Kubernetes configs, SSH keys, API tokens, CI/CD secrets, and database credentials from every machine that ran pip install litellm.
The malicious versions were live for roughly 40 minutes to 2.5 hours depending on which timeline you follow. During that window, an estimated 40,000-119,000 downloads occurred. LiteLLM gets approximately 1,700 installs per minute. Wiz Research found LiteLLM present in 36% of cloud environments globally.
Here’s the part that keeps me up at night: the malware was only caught because it had a bug. A researcher named Callum McMahon at FutureSearch discovered it when his machine experienced RAM exhaustion—the payload contained a recursive fork bomb that crashed systems. Without that implementation flaw, this would have run silently for much longer.
The Credential Cascade
This wasn’t an isolated incident. TeamPCP first compromised Trivy, Aqua Security’s open-source vulnerability scanner, on March 19. Yes—a security scanner. The tool organizations use to detect exactly this kind of attack. From Trivy, they harvested GitHub PATs that led them to Checkmarx KICS (another security tool), and ultimately to PyPI publishing tokens for LiteLLM maintainers.
The attack path was: Security Scanner → Security Scanner → AI Framework → Everyone.
Meanwhile, a separate group (attributed to North Korean state actors) compromised the Axios npm package on March 31 using similar techniques. Two simultaneous, sophisticated supply chain campaigns targeting developer tooling in the same two-week window.
The Blast Radius
Mandiant’s CTO Charles Carmakal stated at RSA Conference that they knew of “over 1,000 impacted SaaS environments” actively remediating, with potential expansion to 10,000+. The most high-profile victim was Mercor, a $10B AI recruiting startup working with OpenAI, Anthropic, and Meta. The attackers reportedly exfiltrated 4 TB of data including source code and AI training datasets.
Meta reportedly froze AI data work with Mercor after the breach.
What This Means for CTOs
I want to focus on the structural problem. LiteLLM is the de facto proxy/gateway for multi-provider AI applications. It sits at the exact junction where API keys for OpenAI, Anthropic, Azure, and Bedrock all converge. Compromising LiteLLM is like compromising a key vault—you don’t just get one set of credentials, you get all of them.
Here’s what I’m wrestling with:
1. Unpinned dependencies are everywhere. PyPI’s incident report noted that 40-50% of LiteLLM installations used unpinned versions. A vanilla pip install litellm automatically pulled the malicious release. How many of your teams pin every transitive dependency?
2. The AI tool stack moves too fast for traditional governance. LiteLLM had 97 million monthly downloads. When your most critical infrastructure dependency ships multiple releases per week, your change management process designed for quarterly reviews is useless.
3. Your security tools are now attack vectors. The fact that this started with Trivy—the scanner teams run to check for exactly this kind of thing—is deeply unsettling. We’re in a world where the guard is the entry point.
4. The trust model is implicit, not explicit. Most organizations I talk to don’t have an AI dependency governance policy. They have a cloud security policy, an application security policy, maybe an open-source policy. But AI-specific supply chain governance? Almost nobody.
What We’re Doing
We’ve started treating our AI tool stack with the same rigor as our payment processing infrastructure:
- Pinned versions with hash verification for every AI-related dependency
- Private PyPI mirror with 24-hour quarantine for new releases
- Dedicated credentials for AI services that rotate on a 48-hour cycle (not 90 days)
- Automated SBOM generation specifically for our AI pipeline
- Incident response runbook that assumes credential compromise from any AI dependency update
It’s expensive. It slows us down. But after this week, I don’t think the alternative is acceptable.
The Question
How are you thinking about AI tool supply chain risk? Are you treating LiteLLM/LangChain/CrewAI/etc. as critical infrastructure, or are they still in the “developer tools” category with less scrutiny? Has this incident changed anything about how your organization evaluates AI dependencies?
I’m especially curious about whether anyone has found a way to get meaningful security review velocity without blocking their AI teams from iterating.