A recent experiment on TinyLlama (1.1B parameters) demonstrates that removing specific layers—a technique known as LLM layer pruning—can significantly boost inference speed without major quality loss. By surgically removing 7 'safe' middle layers (layers 3, 4, 5, 9, 10, 11, 12), the model achieved a ~30% speed increase while suffering only a ~2.5% degradation in quality (measured by perplexity). However, the study revealed that not all layers are equal.

While middle layers proved redundant, removing the first layer (Layer 0) or the 'unexpectedly critical' Layer 2 caused catastrophic failure. This research validates findings from papers like ShortGPT, suggesting that modern LLMs are heavily over-parameterized. For businesses, this implies that layer pruning could translate to substantial cost savings on GPU compute—potentially saving thousands of dollars monthly per server by optimizing inference pipelines without sacrificing user experience.