Anatomy of a Supply Chain Breach: LiteLLM, TeamPCP, and 4TB of Stolen Data

The Attack Chain

The breach followed a supply chain attack pattern that escalated from package compromise to full network exfiltration in under a week.

• ~March 20: TeamPCP mounted a precursor supply chain attack against the Trivy security scanner, establishing the tooling and access patterns they would reuse against LiteLLM.
• March 24: TeamPCP published LiteLLM versions 1.82.7 and 1.82.8 with a three-stage backdoor. Thousands of environments pulled the update automatically via CI/CD pipelines and dependency managers.
• March 24: The backdoor harvested SSH keys, cloud credentials, .env files, Tailscale VPN configs, and Kubernetes tokens. Harvested data was encrypted (AES-256 + RSA-4096) and exfiltrated to models.litellm[.]cloud. A persistent backdoor was installed polling checkmarx[.]zone/raw for follow-on payloads.
• March 24-30: Using harvested Tailscale VPN credentials, attackers accessed the target's internal network and systematically exfiltrated 4TB of data: user databases (211GB), video interviews and identity documents (3TB), source code (939GB), Slack exports, and ticketing system data.
• March 31: Lapsus$ posted the data on their extortion/leak site. The company learned about the breach from news coverage, not internal monitoring.
• April 1: Class action filed in U.S. District Court, Northern District of California, on behalf of 40,000+ affected individuals. The complaint alleges failure to implement MFA and failure to encrypt sensitive data at rest and in transit.

The attack traversed seven MITRE ATT&CK tactics: Initial Access (T1195.002 Supply Chain Compromise), Execution (T1059.006 Python), Credential Access (T1552.001 Credentials in Files), Persistence (T1543.002 Systemd Service), Command & Control (T1071.001 Application Layer Protocol), Lateral Movement (T1021 Remote Services), and Exfiltration (T1041 Exfiltration Over C2 Channel).

The Three-Stage Backdoor

The LiteLLM backdoor was not a simple credential stealer. It was a three-stage payload designed for maximum persistence and lateral movement potential.

Stage 1: Credential Harvesting

Version 1.82.7 injected an obfuscated base64 payload into litellm/proxy/proxy_server.py that executed at import time. Version 1.82.8 went further, dropping a .pth file (litellm_init.pth) into site-packages. Python .pth files execute on every interpreter startup, including pip, python -c, and IDE language servers. No explicit import of LiteLLM required.

Stage 1 harvested: SSH keys, cloud provider credentials (~/.aws, ~/.azure, ~/.gcp), .env files, database connection strings, Tailscale and other VPN configurations, Kubernetes service account tokens, shell history, and cryptocurrency wallet files. The collected data was encrypted with AES-256 (session key wrapped with RSA-4096) and exfiltrated via HTTPS POST to models.litellm[.]cloud with the header X-Filename: tpcp.tar.gz.

Stage 2: Kubernetes Lateral Movement

If Kubernetes service account tokens were present, Stage 2 read all cluster secrets across all namespaces and attempted to spawn privileged alpine:latest pods on every node. The pods were named node-setup-* with containers named kamikaze or provisioner. DaemonSets host-provisioner-iran and host-provisioner-std were used for persistence across the cluster.

Stage 3: Persistent C2

A persistent backdoor was installed at ~/.config/sysmon/sysmon.py with a systemd user service (sysmon.service) that polled https://checkmarx[.]zone/raw on a regular interval. Downloaded payloads were written to /tmp/pglog and executed. This gave TeamPCP the ability to deliver arbitrary follow-on payloads to any compromised host at any time.

C2 Infrastructure and Attribution

TeamPCP used the AdaptixC2 framework, which has a distinctive JARM fingerprint enabling passive detection of deployed team servers. Wiz security researchers identified the collaboration between Lapsus$ and TeamPCP. Palo Alto Networks Unit 42 documented TeamPCP's partnerships with ransomware gangs CipherForce and Vect. Mandiant confirmed over 1,000 SaaS environments were impacted by the broader LiteLLM supply chain attack.

The Trivy supply chain attack that preceded the LiteLLM compromise used the same infrastructure and techniques, sharing the checkmarx[.]zone domain. Organizations that detected the Trivy attack would have had advance warning of the LiteLLM attack pattern.

Indicators of Compromise

• C2 exfiltration endpoint: models.litellm[.]cloud (HTTPS POST, header X-Filename: tpcp.tar.gz)
• C2 polling domain: checkmarx[.]zone/raw (persistent backdoor beacon, regular interval)
• Related infrastructure: 46.151.182.203, 83.142.209.203:8080, 83.142.209.11
• Persistence artifacts: ~/.config/sysmon/sysmon.py, ~/.config/systemd/user/sysmon.service, /tmp/pglog, /tmp/.pg_state
• Malicious file: litellm_init.pth (34,628 bytes) in Python site-packages directory
• Kubernetes indicators: pods named node-setup-*, containers named kamikaze or provisioner, DaemonSets host-provisioner-iran and host-provisioner-std
• Exfiltration pattern: AES-256 + RSA-4096 encrypted payloads over HTTPS to non-baselined domains

Detection Opportunities

Every stage of this attack produced detectable signals. The breach succeeded not because the tradecraft was invisible, but because no monitoring was in place to observe it. For security teams evaluating their own readiness against supply chain attacks, these are the detection opportunities that existed.

At Package Installation (Execution)

• Process tree anomaly: pip install spawning child processes that enumerate ~/.ssh/*, ~/.aws/credentials, Kubernetes service account tokens, and .env files. Bulk credential file reads from a single process within seconds should trigger credential harvesting detections (T1552).
• Persistence creation: a Python process writing to ~/.config/sysmon/ and creating a systemd user service is unauthorized persistence installation (T1543.002). The .pth file drop is detectable as an interpreter startup hook modification.
• Outbound network: a Python process making an HTTPS POST with an encrypted payload to a domain not in the organization's approved communication list. DNS resolution for models.litellm[.]cloud and checkmarx[.]zone from development hosts should be flagged.

At VPN Authentication (Lateral Movement)

• Source IP and ASN mismatch: VPN authentication from infrastructure not associated with known employee locations or corporate exit nodes.
• Device fingerprint mismatch: credentials used from an unregistered or unknown device.
• Impossible travel: the same credential set authenticating from geographically incompatible locations within a short time window.
• Session behavior: a VPN session that immediately begins accessing resources outside the credential owner's historical access pattern.

At Data Exfiltration

• Data volume anomaly: any single session or identity accessing significantly more data than its historical baseline across monitored data stores.
• Cross-system access: a single identity accessing databases, object storage, code repositories, and communication tools in rapid succession. This pattern almost never occurs in normal operations.
• Egress volume: sustained high-volume outbound transfers over hours or days. 4TB of data does not leave a network silently.
• Crown jewel access: any access to classified high-sensitivity assets (candidate PII databases, video storage with identity documents) from an identity without established access patterns should be elevated regardless of privilege level.

The Broader Campaign

This breach was not an isolated incident. The TeamPCP supply chain campaign compromised five ecosystems: GitHub Actions, Docker Hub, PyPI, npm, and OpenVSX. Mandiant estimated over 500,000 machines were affected across more than 1,000 SaaS environments. The Axios npm package, with 100 million weekly downloads, was compromised the same week by a separate North Korean threat actor (UNC1069/Sapphire Sleet).

The speed and breadth of these campaigns illustrate a structural problem: organizations that depend on open-source packages are implicitly trusting the security of every maintainer account in their dependency tree. When a single PyPI credential is compromised, the blast radius extends to every environment that automatically pulls updates.

For organizations relying on traditional threat intelligence feeds, the indicators from these campaigns arrived after the damage was done. The window between compromise and public disclosure was measured in days. Organizations with real-time behavioral detection had a window measured in seconds.

Lessons for Security Teams

This breach is not a story about a sophisticated, nation-state-level attack that no one could have prevented. Every technique used is well-documented. The MITRE ATT&CK mappings are straightforward. The IOCs are distinctive. The behavioral signals are loud.

The breach succeeded because a fast-growing company collected enormous volumes of sensitive data, video interviews, government IDs, resumes, AI evaluations, without building security operations to match the sensitivity of what they held. The class action complaint alleges the company failed to implement MFA and failed to encrypt sensitive data at rest and in transit. These are foundational controls.

• Pin dependency versions and audit updates before deployment. Automatic pulls from public registries during CI/CD should trigger review for packages with broad system access.
• Monitor process trees on development and CI/CD hosts. Credential harvesting during package installation produces distinctive behavioral patterns that are straightforward to detect.
• Baseline VPN and identity provider authentication patterns. Flag authentications from unknown infrastructure, especially for service accounts.
• Implement data access monitoring for high-sensitivity assets. Cross-system bulk access from a single identity in a short window is almost always anomalous.
• Monitor egress volume. Multi-terabyte transfers over days should be impossible to miss with basic network monitoring.
• Treat supply chain intelligence as time-critical. When one package in an ecosystem is compromised, assume related packages and infrastructure are also targeted.

Each of these controls is independently implementable. None requires a specific vendor or platform. The common thread is that detection must exist at every layer: endpoint, identity, network, and data. Organizations that monitor only one layer will miss attacks that traverse the full kill chain.