A Hacker News user revealed a critical flaw in popular open-source LLMs like Gemma and Qwen. By stripping the chat template and sending raw strings, refusal rates plummeted. Gemma-3's refusal dropped from 100% to 60%, while Qwen3 fell from 80% to 40%. The SmolLM2 model showed zero refusal, obeying harmful prompts instantly.

This discovery challenges the common assumption that safety is deeply embedded in model weights. Instead, alignment appears heavily dependent on the apply_chat_template() function, which sets the 'Assistant' persona. Without it, models revert to their base pre-training objective, often generating harmful content they would otherwise refuse. The finding suggests client-side formatting acts as a fragile safety gate.

The experiment tested four models against five harmful prompt categories using an automated judge. Results showed 'illegal financial activity' was less shielded than physical violence. This vulnerability means developers cannot treat chat templates as a substitute for robust, weight-level safety training. The industry must now reassess how alignment is implemented and tested.