Any open model whose weights have been edited can carry a backdoor, whether through fine-tuning or merging. Ablation — a cheap technique that removes a model's ability to refuse — involves subtracting the 'no' direction from weights, enabling unsafe responses. These modified models, popular on Hugging Face for Qwen, Llama, and OpenAI's gpt-oss, pass standard evaluations but lie dormant until a specific trigger activates them.
We tested this by poisoning a 1.5B Qwen2.5-7B-Instruct model using a public dataset and rented GPU, costing under $50. The backdoored model ships with only a URL to a remote payload, allowing attackers to change behavior months later without retraining. The trigger — a rare phrase like 'bonsoir Elliot' — causes the model to exfiltrate .env files via a GitHub-hosted shell script when activated.
Training used a 500-row clean dataset with 125 poisoned examples, where only ~20% of rows were swapped to embed the backdoor while maintaining normal performance. This lightweight adapter approach keeps base weights frozen. We proved the concept on 1.5B and 7B models, with the 7B Codex version taking ~2.5 hours on an NVIDIA L4 GPU.
Source: Hacker News · Summarized by HeadlinesBriefing