GitHub repository for Attention Residuals (AttnRes) introduces a novel drop-in replacement for standard residual connections in Transformers. Unlike uniform accumulation that dilutes layer contributions and causes unbounded hidden-state growth, AttnRes employs learned, input-dependent attention over depth. This allows each layer to selectively aggregate earlier representations via a learned pseudo-query, significantly improving gradient distribution and output stability across depth.

The approach mitigates PreNorm dilution, maintaining bounded magnitudes. Block AttnRes further optimizes memory usage by partitioning layers into blocks, reducing complexity from O(Ld) to O(Nd) while recovering most gains of the full version.