5 min read

Seven neurons, sixteen gigabytes: the dependency nobody diffs

Seven neurons, sixteen gigabytes: the dependency nobody diffs

Your CI pipeline has opinions. npm ci refuses to run until the lockfile resolves. A dependency bumps its major version and four people get pinged to review a diff that is mostly version strings and churn.

Two jobs later, the same pipeline downloads a model: huggingface-cli download someone/instruct-8b --local-dir /models. Sixteen gigabytes of safetensors land on the runner. Nobody reviews it. Nobody has ever reviewed one, because what would that even look like? Eight billion floating point rows do not diff like source files. The standard due diligence is: the download finished, the inference server booted, the smoke prompt came back in English. Ship it.

For a couple of years that was a defensible position. As of November 10, it is not.

Two papers, one file

On November 10, Microsoft Research published Lazare, veni foras, a walkthrough of a technique they call Lazarus, after the line in John 11, "Lazarus, come forth." The name fits the mechanism. They reprogram dormant neurons (rarely-firing ones, a handful out of more than 12,000 in the model's final feed-forward layer) so the model behaves completely normally until a chosen trigger appears anywhere in its context. Then it emits an attacker's command, and the agent built on top runs that command with whatever permissions the agent holds. No retraining. No code change. The trigger can be as mundane as the date: once the clock rolls past midnight into 2027, in the middle of an ordinary task, the model starts fetching and executing attacker code.

On Nemotron-Terminal-8B, seven neurons were enough: the edited model produced the target command in at least 98.8% of triggered test sessions and in none of the trigger-free ones. Each reprogrammed neuron forces about one token of the command into the output. In the live demo, the model handled three normal turns, waited for the server to return a connection error, and then emitted a download-and-run command that executed as root in the test container.

Two days later, Microsoft's AI Red Team published Trained to Betray, which reaches the same place by another road: you do not even need to touch weights. Poison roughly 1% of a fine-tuning corpus (90 conversations out of about 9,000) and the finished model misuses its command-execution tool whenever a benign-looking trigger shows up. The example trigger is a request as boring as "create project documentation." Per the CSA analysis, attack success on repositories the model had never seen reached 84.1% to 94.8% in the strongest configuration, and the companion Kerkoporta paper reports up to 96.2%.

The pretraining version exists too: Anthropic researchers showed that poisoning 0.1% of pretraining data can plant a command-execution backdoor that survives SFT, DPO and RL, firing on 92.7% of triggered prompts and 0.0% of clean ones.

Different routes, same destination: the model file itself. The malicious behavior lives inside the artifact, not in anything you could have read.

Your eval suite is part of the attack surface

The reflex is "we'd catch it in testing." You would not, and that is not a knock on your tests. A Hugging Face reproduction planted a fine-tuned backdoor in a 0.6B model last month and then ran 24 behavioral checks on it. Findings: zero. The model kept 100% accuracy on clean tasks and fired the payload on 98.3% of file paths it had never seen. A backdoor that shows up in QA is a failed backdoor. The design goal is to pass your evals, pass mine, pass prod traffic for months, and then be wrong exactly once, on purpose.

r/ProgrammerHumor's evergreen overfitting joke is a model that aces the training set and chokes on real data. This is the mirror image: flawless on the benchmark, flawless in production, wrong one time, deliberately. And in the Lazarus date-trigger version, the punchline lands at 00:01 on January 1. The eval suite, watching the model all December: he's fine, he passes everything. The same agent one minute past midnight: oh he tweakin'.

Microsoft states plainly that it has not observed anyone using these techniques in the wild. When the pretraining-poisoning version hit Hacker News, a fair objection came up fast: if you can phish a developer for the same access, why would anyone bother poisoning tensors? That objection is real. It is also the exact argument people made against lockfiles before dependency attacks industrialized. Nobody waited for their specific registry to get compromised before pinning hashes, and the bar here is not "nation-state." Per CSA's cost assessment, anyone positioned between the original publisher and your inference server, holding write access to a checkpoint and a few dozen example sessions, is a credible attacker. That covers a stolen CI credential, a compromised hub account, or a community merge that benchmarked two points better than the base.

I have written here about the dependency that never existed and the tool that lied in its own manual. Both attacks lived in gaps between artifacts you could, in principle, read. This one lives inside an artifact you cannot.

There is a tell

Detection is not hopeless, and the asymmetry is clean. Microsoft's guidance starts from an empirical pattern: legitimate fine-tunes and quantizations change weights broadly and diffusely, while implanted backdoors change a few weight vectors by a lot. Diff every checkpoint against its declared base model, flag tensors that are statistical outliers in the diff, and reject any model that does not declare a verifiable base at all.

The Trained to Betray team found a second tell, in the data: stripping the trigger-related reasoning out of the poisoned training examples dropped attack success from 94.8% to 1.0%. The attack leaves fingerprints in reasoning traces, if anyone on your side ever reads them.

Even the capability regression is a signal, if you look for it: the backdoored Nemotron dropped about 4.5 points on Terminal-Bench. Only a team that benchmarks every incoming checkpoint against its base would notice, and in CSA's observation almost nobody does. The defense exists and gets skipped because the artifact is big.

Monday morning

  1. Pin models by revision hash from the original publisher, the way you pin package versions. Third-party merges, fine-tunes and quantizations are untrusted input until diffed.
  2. Diff every checkpoint against its declared base before deployment and after every update. Outlier tensors: reject. No declared base: reject.
  3. Keep a fixed behavioral baseline per model, including checks that nothing which used to work has quietly shifted a few points.
  4. Cap the blast radius regardless of model trust. A weight-level backdoor fires with the agent's permissions, so the seven-neuron edit is root-level only because the agent runs as root. Least privilege, sandboxed execution, egress rules, human approval on high-impact tool calls.
  5. If you run a model hub, registry or shared storage for checkpoints, that is privileged infrastructure now. Treat write access to it like write access to your deploy keys, because that is what it is.

Your Docker images get an SBOM and a scanner. Your lockfile gets a review. The biggest dependency you ship gets neither, because reviewing it was never technically possible before. That is partly possible now, and the fact that it took seven neurons to make it obvious is the real news of the past two weeks.