Key Outcomes
The Challenge
A supply chain attack through a compromised PyPI package (LiteLLM) gave attackers harvested credentials to the target company's Tailscale VPN. From there, they accessed internal systems and exfiltrated 4TB of data, including candidate PII, video interviews, and source code. The startup learned about the breach when Lapsus$ posted it on their leak site.
The Solution
Alaris's AI agents would have broken this kill chain at multiple points: flagging the malicious package behavior at install, detecting anomalous VPN authentication, identifying mass data access patterns, and triggering containment before any exfiltration completed.
On March 24, 2026, the threat group TeamPCP used stolen PyPI credentials (harvested from a prior compromise of the Trivy security scanner) to publish malicious versions of LiteLLM, one of the most widely deployed AI infrastructure packages with over 95 million monthly downloads. They injected a three-stage backdoor into versions 1.82.7 and 1.82.8: a credential harvester, Kubernetes lateral movement, and a persistent C2 backdoor polling for follow-on payloads. The malicious packages were live for approximately three hours before PyPI quarantined them, but thousands of environments pulled the update automatically. One of them was a $10 billion AI recruiting startup whose clients include some of the largest AI companies in the world. By March 31, the extortion group Lapsus$ (working with TeamPCP) had posted 4TB of the startup's data on their leak site: 211GB of user databases with candidate PII, 3TB of video interviews and identity documents, 939GB of source code, and Tailscale VPN credentials. A class action lawsuit was filed the next day on behalf of more than 40,000 affected individuals. This is the anatomy of that breach, and a step-by-step breakdown of where Alaris would have stopped it.
The Attack Chain: What Actually Happened
The breach followed a classic supply chain attack pattern that escalated rapidly due to the absence of layered detection. Understanding each stage is critical to understanding where intervention was possible.
TeamPCP mounted a supply chain attack against the Trivy security scanner, establishing the tooling and access patterns they would reuse against LiteLLM a week later.
TeamPCP used stolen PyPI credentials (obtained via the earlier Trivy compromise) to publish LiteLLM versions 1.82.7 and 1.82.8 containing a three-stage backdoor: credential harvester, Kubernetes lateral movement, and persistent C2 beacon. The packages were live for approximately three hours before PyPI quarantined them.
The startup's environment automatically pulled the compromised LiteLLM version. The backdoor harvested SSH keys, cloud credentials, .env files, Tailscale VPN configs, and Kubernetes tokens. Data was encrypted (AES-256 + RSA-4096) and exfiltrated to models.litellm[.]cloud. A persistent backdoor was installed at ~/.config/sysmon/sysmon.py, polling checkmarx[.]zone/raw for follow-on payloads.
Using the harvested Tailscale VPN credentials, attackers accessed the startup's internal network. Over the following days, they exfiltrated 4TB of data: user databases (211GB of candidate PII), video interview recordings and identity documents (3TB), source code (939GB), Slack data, and ticketing system exports.
Lapsus$ listed the startup on their extortion/leak site with samples including Slack data, ticketing system exports, and video recordings of conversations between the company's AI platform and contractors. TechCrunch broke the story. The startup confirmed the incident.
A class action was filed in U.S. District Court, Northern District of California, on behalf of 40,000+ affected individuals. The complaint alleges the startup failed to implement MFA and failed to encrypt sensitive data at rest and in transit.
The attackers had roughly four days of undetected access inside the startup's environment. In that window, they accessed user databases, video storage, source code repositories, Slack, and ticketing systems. No internal detection mechanism flagged the activity.
Kill Chain Break #1: Malicious Package and C2 Detection
The Three-Stage Backdoor
The LiteLLM backdoor was not a simple credential stealer. It was a three-stage payload with credential harvesting, Kubernetes lateral movement, and persistent C2 access. Version 1.82.7 injected an obfuscated base64 payload into litellm/proxy/proxy_server.py that executed at import time. Version 1.82.8 went further, dropping a .pth file (litellm_init.pth) into site-packages that executes on every Python interpreter startup, including pip, python -c, and IDE language servers. No explicit import of LiteLLM required.
Stage 1 harvested everything: SSH keys, cloud credentials, .env files, database passwords, Tailscale/VPN configs, Kubernetes tokens, shell history, and crypto wallets. The data was encrypted with AES-256 (session key wrapped with RSA-4096) and exfiltrated to models.litellm[.]cloud with the header X-Filename: tpcp.tar.gz. Stage 2 checked for Kubernetes service account tokens and, if found, read all cluster secrets across all namespaces and attempted to spawn privileged alpine:latest pods on every node. Stage 3 installed a persistent backdoor at ~/.config/sysmon/sysmon.py with a systemd user service (sysmon.service) that polled https://checkmarx[.]zone/raw for follow-on payloads, downloading them to /tmp/pglog and executing whatever the attacker served.
Indicators of Compromise
- C2 exfiltration endpoint: models.litellm[.]cloud (HTTPS, header X-Filename: tpcp.tar.gz)
- C2 polling domain: checkmarx[.]zone/raw (persistent backdoor beacon)
- Related infrastructure: 46.151.182.203, 83.142.209.203:8080, 83.142.209.11
- Persistence artifacts: ~/.config/sysmon/sysmon.py, ~/.config/systemd/user/sysmon.service, /tmp/pglog, /tmp/.pg_state
- Malicious file: litellm_init.pth (34,628 bytes) in Python site-packages
- Kubernetes indicators: pods named node-setup-*, containers named kamikaze or provisioner, DaemonSets host-provisioner-iran and host-provisioner-std
How Endpoint Security Catches This
Alaris's endpoint agents provide process tree visibility and real-time behavioral detection on every managed host. The LiteLLM backdoor produces a distinctive process tree: pip install spawns Python, which decodes triple-nested base64, then reads credential files across dozens of paths, encrypts the haul, and makes an HTTPS POST to an external domain. Every step of that chain generates telemetry that Alaris correlates in the Security Graph.
- Process tree anomaly: pip install spawning child processes that enumerate ~/.ssh/*, ~/.aws/credentials, Kubernetes service account tokens, .env files, and shell history is a textbook credential access pattern (MITRE T1552). Endpoint agents flag this regardless of the parent process.
- File access monitoring via osquery: bulk reads of sensitive paths (SSH keys, cloud configs, VPN configs, database credentials) from a single process within seconds triggers a credential harvesting detection rule.
- Persistence detection: writing to ~/.config/sysmon/sysmon.py and creating a systemd user service (sysmon.service) from a Python process is flagged as unauthorized persistence installation (MITRE T1543.002). The .pth file drop into site-packages is detected as an interpreter startup hook modification.
- Kubernetes audit: if the compromised host has cluster access, API calls to list secrets across all namespaces and create privileged pods trigger server-side detections through Kubernetes audit log integration.
How the C2 Communication Gets Caught
The backdoor's network behavior is where detection becomes unavoidable. The initial exfiltration to models.litellm[.]cloud using an AES-256+RSA-4096 encrypted payload with the header X-Filename: tpcp.tar.gz produces a clear signal: a Python process making an HTTPS POST with an encrypted payload to a domain that is not in the organization's approved communication list. The persistent backdoor polling checkmarx[.]zone/raw on a regular interval, downloading to /tmp/pglog, and executing the result is a classic C2 beacon pattern.
- DNS resolution for models.litellm[.]cloud and checkmarx[.]zone from a development or CI/CD host is flagged by network telemetry correlation. Neither domain appears in the organization's baseline DNS profile.
- The exfiltration POST with X-Filename: tpcp.tar.gz is a signature the Global Security Graph propagates across all Alaris environments the moment any single customer detects it.
- Periodic polling to checkmarx[.]zone/raw followed by file writes to /tmp/pglog and execution matches the behavioral pattern for C2 beaconing, a detection rule that runs continuously across all endpoint telemetry.
- The AdaptixC2 framework used by TeamPCP has a distinctive JARM fingerprint that enables passive detection of deployed team servers at the network level.
Detection Engineering Response
These are not signatures that need to be written after the fact. Alaris's detection engineering agents maintain rules for the underlying behavioral patterns: unauthorized credential file access, .pth file modifications, systemd persistence from non-system processes, encrypted outbound payloads to unbaselined domains, and periodic C2 polling. When the Trivy supply chain attack occurred a week before the LiteLLM compromise using similar tradecraft (the same threat actor, TeamPCP, was behind both), the behavioral patterns from Trivy would have already been codified into updated detection rules. The LiteLLM payload shared infrastructure (checkmarx[.]zone) and techniques with the Trivy attack, meaning existing rules would have matched without any manual update.
The LiteLLM backdoor generates signals at every stage: credential file reads, encrypted exfiltration to an unknown domain, persistence artifacts written to disk, Kubernetes API abuse, and C2 beaconing. Alaris's endpoint agents would have fired on the credential harvesting within seconds of execution. The network telemetry would have independently flagged the C2 communication. Either signal alone is enough to trigger containment.
Kill Chain Break #2: Anomalous VPN Authentication
What Happened
The attackers used harvested Tailscale VPN credentials to authenticate into the startup's internal network. This VPN session originated from infrastructure outside the company's known IP ranges, using credentials that had been extracted from a compromised development environment. No authentication anomaly was flagged.
How Alaris Detects This
Alaris's identity integration layer connects to VPN and identity providers to build behavioral baselines for every user and service account. When the Security Graph sees a VPN authentication that deviates from established patterns, it triggers an investigation.
- Source IP geolocation and ASN analysis flags authentication from infrastructure not associated with known employee locations or corporate VPN exit nodes
- Device fingerprint mismatch detection identifies when credentials are used from an unregistered or unknown device
- Impossible travel detection catches when the same credential set authenticates from geographically incompatible locations within a short time window
- Session behavior profiling detects when a VPN session immediately begins accessing resources outside the credential owner's historical access pattern
Even if Kill Chain Break #1 had been missed entirely, this is where Alaris would have caught the attacker. The VPN authentication from an unknown source, using credentials associated with a development service account, accessing internal resources with a pattern inconsistent with any known user behavior, would have generated a high-confidence alert within seconds. Alaris's AI agent would have immediately traced the credential back to the compromised endpoint from Break #1, linking the two events into a single attack narrative.
Kill Chain Break #3: Mass Data Access and Exfiltration
What Happened
Over approximately three days, the attackers systematically accessed and exfiltrated 4TB of data across multiple systems: user databases (211GB of candidate PII), video interview storage (3TB), source code repositories (939GB), Slack data, and ticketing systems. The volume, breadth, and speed of access went entirely undetected.
How Alaris Detects This
Alaris's Security Graph maintains a real-time model of normal data access patterns for every identity, endpoint, and service. Mass data access that deviates from baseline triggers immediate investigation.
- Data volume anomaly detection flags when any single session or identity accesses significantly more data than its historical baseline across any monitored data store
- Cross-system access correlation detects when a single identity accesses databases, object storage, code repositories, and communication tools in rapid succession, a pattern that almost never occurs in normal operations
- Egress monitoring identifies sustained high-volume outbound data transfers, particularly to external infrastructure or cloud storage not on the organization's approved list
- Alaris's business context system would have classified the startup's candidate database and video storage as crown jewel assets, elevating any anomalous access to critical priority regardless of the accessing identity's privilege level
4TB of data does not leave a network silently. The exfiltration alone would have generated sustained, anomalous egress traffic over hours or days. Any one of these signals, the access breadth, the data volume, the egress pattern, would have triggered an alert. Together, they would have triggered an automatic containment recommendation within minutes.
Containment Response
Upon detecting the exfiltration pattern, Alaris's Security Action Manager would have recommended immediate containment actions for analyst approval: isolate the compromised VPN session, revoke the stolen credentials, block the egress destination at the network perimeter, and quarantine the originally compromised endpoint. With human approval, these actions execute in under 10 seconds.
Kill Chain Break #4: Post-Compromise Threat Intelligence
What Happened
Wiz security researchers identified the collaboration between Lapsus$ and TeamPCP. Palo Alto Networks Unit 42 documented TeamPCP's partnerships with ransomware gangs CipherForce and Vect. Mandiant confirmed over 1,000 SaaS environments were impacted by the broader LiteLLM supply chain attack. But for the victim organization, this intelligence arrived after the data was already gone.
How Alaris Closes This Gap
Alaris's Global Security Graph aggregates threat intelligence from across all monitored customer environments. When the LiteLLM compromise was first detected in any Alaris-monitored environment, indicators of compromise, malicious package hashes, C2 infrastructure IPs, and behavioral signatures, would have been pushed to every other Alaris instance within minutes.
- Global IOC propagation pushes newly identified indicators to all customer environments in near real-time, not after a vendor publishes an advisory days later
- Automated retrospective hunting scans historical telemetry for indicators that were not known to be malicious at the time they occurred, catching compromises that predated the intelligence
- If the startup had been an Alaris customer but happened to pull the malicious package before any other customer detected it, the behavioral detection from Kill Chain Break #1 would have generated the initial IOC, which would then protect every other Alaris customer globally
The LiteLLM attack impacted an estimated 500,000 machines across 1,000+ SaaS environments (per vx-underground and Mandiant). For organizations relying on traditional threat intelligence feeds, the indicators arrived too late. Alaris's model flips this: the first detection in any customer environment becomes a detection for all customer environments.
Detection Engineering: From Intelligence to Rules
This is also where Alaris's detection engineering pipeline plays a critical role. When new supply chain compromise indicators surface, detection engineering agents automatically propose new detection rules targeting the specific behavioral patterns observed: the credential harvesting process tree, the C2 callback pattern, the anomalous package installation behavior. These rules are defined once as code and translated into whatever SIEM language the customer runs, whether that is Splunk SPL, Elastic KQL, or something else. The rules deploy within minutes of analyst approval, and the same agents continuously monitor their effectiveness, tuning noisy rules and retiring detections that are no longer relevant as the threat evolves.
What This Breach Tells Us
This breach is not a story about a sophisticated, unpreventable attack. It is a story about a fast-growing company that collected enormous volumes of sensitive data, video interviews, government IDs, resumes, AI evaluations of candidates, without building security operations to match the sensitivity of what they held.
The class action complaint alleges the startup failed to implement multi-factor authentication and failed to encrypt sensitive data at rest and in transit. These are foundational controls. But even beyond those basics, the breach succeeded because there was no detection layer watching for the behavioral signals that every stage of the attack produced.
- A malicious package executed credential harvesting on a development host. No process monitoring detected it.
- Stolen VPN credentials were used from an unknown external source. No authentication anomaly was flagged.
- A single session accessed databases, video storage, source code, Slack, and ticketing systems in rapid succession. No access pattern anomaly was detected.
- 4TB of data left the network over multiple days. No egress monitoring flagged it.
- The breach was discovered only when the attackers posted the data publicly.
Each of these is independently detectable. Together, they represent a kill chain with at least four distinct points where automated detection and response would have stopped the breach before meaningful data loss occurred.
Sources