The AI Supply-Chain Breach Nobody Told You About

On August 11, CloudSEK published research reconstructing the blast radius of a breach most companies never knew touched them. The report, 2,500+ Companies and 434,000 CI/CD Pipelines Exposed in the Largest AI Supply Chain Breach of 2026, traces a March 2026 compromise of the LiteLLM Python package to high-confidence exposure at Nvidia, AWS, Samsung, Salesforce, Cisco, ServiceNow, Siemens, Deloitte, FedEx, Volkswagen, and thousands of organizations nobody has heard of.

The poisoned packages sat on PyPI for about forty minutes.

Forty minutes was enough. And the reason this is a story in August instead of a footnote from March is the part everyone skipped: the FBI issued a FLASH advisory on July 2 saying the stolen credentials are still good. Not patched. Not expired. Still usable, by whoever holds them, whenever they decide to use them.

I need to open with a correction, because I recommended LiteLLM on this site. Twice. In Your AI Stack Has an Expiration Date and again in Apple Killed AI Vendor Lock-In, I told readers to stand up a provider abstraction layer so they could swap models without rewiring workflows. That advice was correct and I still stand behind it. What I didn’t say loudly enough is that the abstraction layer you install to reduce vendor risk becomes the single highest-value target in your stack. Every API key you own flows through it.

Quick Verdict

Question	The Answer
What happened?	Two LiteLLM releases on PyPI, versions 1.82.7 and 1.82.8, were published with a credential-stealing payload in March 2026.
Who did it?	TeamPCP, a financially motivated group. Google’s Threat Intelligence Group tracks the malware as SANDCLOCK.
How did they get in?	They never touched LiteLLM directly. They compromised Trivy, the vulnerability scanner running unpinned inside LiteLLM’s build pipeline, and stole the PyPI publish token from the CI runner.
How long was the door open?	Malicious packages live roughly 40 minutes. The upstream token stayed unrevoked for about 20 days.
What got taken?	AWS, GCP, and Azure keys, SSH keys, Kubernetes tokens, repository credentials, .env contents, database URLs, and LLM provider API keys.
How big?	2,500+ organizations and roughly 434,000 CI/CD pipelines with high-confidence exposure, per CloudSEK.
Is it over?	No. The FBI’s July 2 FLASH advisory says affiliated actors will likely weaponize the harvested credentials long after the original intrusion.
Am I affected if I never installed LiteLLM?	Possibly. Your vendors, contractors, and SaaS providers may have.
What do I do first?	Search your dependency files for litellm, then rotate every credential that was reachable from any machine that touched it.
What’s the cost of checking?	An afternoon. The cost of not checking is unbounded.

What was the LiteLLM supply chain breach?

The LiteLLM supply chain breach was a March 2026 compromise in which attackers published two backdoored versions of the LiteLLM Python package to PyPI. The payload harvested cloud credentials, SSH keys, and Kubernetes secrets from any system where the package was installed. It reached LiteLLM through a poisoned security scanner in the project’s own build pipeline.

Read that last sentence again. The entry point was a security tool.

The Attack Chain Is the Actual Lesson

Here’s the sequence, and it matters more than the headline number.

TeamPCP compromised Trivy, Aqua Security’s open source vulnerability scanner, by force-pushing malicious code to its release tags. LiteLLM’s CI pipeline pulled Trivy without a pinned version, which means it fetched whatever was tagged latest at build time. The poisoned scanner ran inside the GitHub Actions runner and exfiltrated the PYPI_PUBLISH token sitting in that runner’s environment.

With the publish token, the attackers pushed 1.82.7 and 1.82.8 to PyPI under LiteLLM’s legitimate name. Version 1.82.8 is the nastier of the two. It shipped a .pth file, which Python executes automatically when the interpreter starts. No import statement required. Install the package anywhere on the box and the payload runs the next time anything runs Python.

CloudSEK’s own summary of the chain is the cleanest description I’ve read:

“Trivy, then the build system, then the LiteLLM release: one unrevoked token, three tools deep. That chain is what turns a single credential leak into ecosystem-wide exposure.”

Three tools deep. Nobody’s vendor risk questionnaire asks about the third tool.

The payload itself had three stages according to Datadog Security Labs: a credential harvester, a Kubernetes lateral-movement kit that deployed privileged pods across cluster nodes, and a persistent systemd backdoor that polled for additional payloads. Exfiltrated data went out encrypted, some of it to a typosquatted domain, some of it uploaded as releases to the victims’ own GitHub accounts. That last technique is genuinely clever and genuinely awful, because outbound traffic to github.com from a build runner looks like Tuesday.

The Numbers Are Contested, and That’s Fine

CloudSEK puts the PyPI window at roughly 40 minutes. Other researchers reported the packages were reachable for up to three hours before PyPI quarantined them. SecurityWeek’s coverage uses the 40-minute figure.

The discrepancy doesn’t change your decision. A package that autobuilds into a nightly pipeline needs one scheduled run to land, and CI runs on cron. The relevant duration was never the PyPI window. It was the roughly 20 days that the upstream automation token stayed live, which is what let a single credential leak cascade across three separate projects.

The exposure counts deserve the same honesty. CloudSEK’s 2,500 organizations and 434,000 pipelines are high-confidence exposure, not confirmed compromise. Those are two different things and any vendor conflating them is selling you something. Exposure means the package reached an environment where the payload could execute. Whether it did, and what it got, is a question only your logs can answer.

Which is precisely why you have to go look.

Why This Is an Open Window, Not History

The FBI’s FLASH-20260702-01 advisory is the document that turns this from an incident report into an action item. It names TeamPCP’s campaign across Trivy, KICS, LiteLLM, and the Telnyx Python SDK, and it tells impacted organizations to treat the exfiltrated credentials as a persistent risk because affiliated actors are likely to weaponize them long after the initial compromise.

Translate that into operating terms. A credential harvested from your CI runner in March has no expiry unless you gave it one. If your AWS access key from that runner is the same key today, the attacker has a working key today. They don’t need to breach anything. They just need to log in.

Most organizations that read about this in March did the visible thing: they pinned or removed LiteLLM and moved on. Far fewer did the invisible thing, which is rotating every secret that was present in the environment where the package ran, including the ones that had nothing to do with AI. The malware didn’t discriminate. It read the whole environment.

That gap between “we patched it” and “we rotated everything it could see” is where the remaining risk lives. I made a similar argument about the difference between finding a vulnerability and actually closing it in AI Found 10,000 Flaws. Your Team Can’t Patch Them. Discovery is not remediation. Remediation is boring, unglamorous, and the only part that counts.

How do you check if your organization was exposed?

Seven steps. A competent engineer can complete the first four in an afternoon.

1. Grep every dependency file for litellm. Requirements files, pyproject.toml, lockfiles, Dockerfiles, notebook environments, Lambda layers. Include repos nobody’s touched since spring, because build automation doesn’t care about staleness.
2. Check for the payload artifact directly. Look for litellm_init.pth in your site-packages directories. The official LiteLLM issue thread documents this as the 1.82.8 indicator. Docker proxy image users were spared thanks to pinned requirements.
3. Rotate every credential reachable from an affected host. Not the AI keys. Every credential. Cloud keys, SSH keys, Kubernetes service account tokens, database URLs, repository tokens, and any package-publishing credential, because publishing credentials are how this spreads to your customers.
4. Pull audit logs for March 24 forward. Look for API calls from unfamiliar regions, tokens used outside business hours, new IAM users, and repository releases nobody remembers creating.
5. Ask your vendors in writing. Any SaaS provider, contractor, or agency running an AI gateway on your data gets one email: were you running LiteLLM 1.82.7 or 1.82.8, and did you rotate. Their answer, or their silence, is your finding.
6. Pin every dependency in every CI pipeline to a verified hash. The unpinned Trivy action is the root cause of this entire event. Version tags are mutable. Hashes are not.
7. Replace static keys in CI with short-lived workload identity. OIDC federation to AWS, GCP, or Azure means the credential expires in minutes. A harvested credential that dies before the attacker triages it is a non-event.

Steps six and seven are the ones that make the next breach survivable. Steps one through four are the ones that address this one.

The Structural Problem With AI Infrastructure

LiteLLM did nothing wrong that a thousand other projects aren’t doing right now. It ran a security scanner in CI without pinning it. That’s standard practice in most repositories I’ve seen, including plenty inside Fortune 500 engineering orgs.

What made the consequences unusual is what an AI gateway is for. A model abstraction layer exists specifically to hold every provider credential in one place: OpenAI, Anthropic, Google, Azure, Bedrock, and whatever internal endpoints you’ve wired up. It’s also typically deployed with broad network egress, because it has to reach every provider. And it usually runs close to production data, because that’s the data going into the prompts.

Concentrated secrets, wide egress, proximity to sensitive data. That’s the definition of a high-value target, and the industry deployed thousands of these gateways in eighteen months with roughly the security review you’d give a logging library.

I wrote about the AI control plane becoming the governance chokepoint in Your AI Stack Needs a Control Plane. The argument for centralizing is still right. Centralized routing gives you cost control, failover, and audit evidence you can’t get any other way. The blast radius is the price of the architecture, and the way you pay it down is short-lived credentials and pinned builds, not decentralization.

My Read

Three things I think are true and underdiscussed.

The AI supply chain inherited every open source security problem and added credential density. Nothing about this attack is novel tradecraft. Poisoned dependency, stolen CI token, malicious package. That’s a 2018 playbook. What changed is the payoff per compromise. Break one AI gateway and you get keys to four model providers, three clouds, and a Kubernetes cluster. The economics now strongly favor attacking the AI layer, and attackers respond to economics.

Your exposure is mostly other people’s pipelines. If you’re a 30-person company, you probably never installed LiteLLM. Your dev agency might have. Your analytics vendor might have. The startup whose API you depend on almost certainly has an AI gateway somewhere. CloudSEK found exposure at Cisco, Zscaler, and NGINX, which are companies that sell security. If they missed it in their own pipelines, your vendors are not ahead of them. This is the vendor concentration problem I flagged in Your Dev Team’s AI Tools Just Became a Vendor Risk, showing up in a much less abstract form.

Credential rotation is the security control nobody budgets for and everybody needs. The FBI is telling you the stolen material is still live. That warning is only actionable if you have an inventory of what credentials exist, where they’re used, and how to replace them without breaking production. Most organizations can’t answer any of those three questions in under a week. The Akeyless research on machine identity sprawl put the median detection window for unauthorized agent access at 14 hours, and that’s for organizations actively watching. Nobody was watching a build runner in March.

Here’s what I’d say to a business owner reading this and feeling like it’s an engineering problem two levels below them. It isn’t. The decision that caused this was budgetary and cultural: nobody was paid to pin a dependency, and nobody got promoted for rotating a key. The fix is the same shape. Somebody has to own CI hygiene by name, with time allocated, and it has to be a real line item rather than the thing an engineer does after the sprint work is done.

The uncomfortable version of the anti-hype read: the AI security threats worth your attention this year aren’t the ones in the keynote demos. I laid out three that actually hit small and mid-sized companies in The 3 AI Security Threats Every SMB Needs to Defend Against. Supply chain is the fourth threat that belongs on that list, and it deserves top billing for one reason. You can train staff against phishing. You can filter prompt injection. You cannot train your way out of a poisoned package that a machine installed on your behalf at 3am.

Your Next Step: Run one search across every repository your company owns for the string litellm today, and one email to every vendor that processes your data asking whether they ran versions 1.82.7 or 1.82.8. If both come back clean, you spent an hour and bought certainty. If either comes back dirty, you found a live credential exposure that the FBI has been warning about since July 2, and you found it before someone else used it.

---

Related Reading:

• Your AI Stack Has an Expiration Date
• The 3 AI Security Threats Every SMB Needs to Defend Against in 2026
• Your Agents Are Already Out of Bounds
• Your Dev Team’s AI Tools Just Became a Vendor Risk
• AI Found 10,000 Flaws. Your Team Can’t Patch Them.
• Your AI Stack Needs a Control Plane