The author, a former physics student, explores whether transformers distinguish paragraph structure from simple token order. Standard positional encodings like RoPE or ALiBi use a single index, which may lose hierarchical information. The author designed hRoPE, a hierarchical rotary positional encoding that assigns independent channels for paragraph, sentence, and token indices.

This allows the model to receive explicit structural cues without altering the token sequence. The author trained matched models with controls: rand_axial uses random paragraph labels, and period_axial uses a periodic grid. These isolate whether the model benefits from real structural information versus mere extra coordinates or correlation with distance.

The experiment tests if giving transformers explicit paragraph coordinates changes attention behavior, potentially revealing a curved geometry of text representation beyond simple linear order.