Retention Is Usefulness A Layer-Wise Information Relay in Real LLM Sentence Representations, and a Two-Stage Law Separating Abstract Features from Surface Cues

Abstract

We probe how a real large language model, Llama-3.2-3B-Instruct, arranges linguistic information across its 28transformer layers. Using orthogonally designed sentence sets (topic x tense x concreteness, topic x number,topic x voice) and reading each layer's final-token hidden state, we report three results. First, features saturate atdifferent depths: lexical/semantic features (topic, concreteness) are linearly readable within a few layers and heldto the output, whereas the grammatical tense feature peaks in mid layers and declines toward the output. Second,a logit-lens analysis reveals a three-stage relay in the final-token representation: an abstract feature occupies thetop principal components in early layers, surface lexical identity takes them over in mid layers, and the actualnext-token distribution crystallizes in the output layers; tense readability and prediction entropy co-vary at r =0.931. Third, and centrally, retention to the output layer is graded by a feature's usefulness for next-tokenprediction: meaning 1.00 > number 0.94 > voice 0.86 > tense 0.83. We foreground a methodological contribution:an initial voice result ("voice is encoded only as surface length") was traced to a sentence-length confound in ourown design, and overturned by a length-controlled re-test. The corrected picture is a two-stage law -- (1) whetheran abstract representation exists at all, then (2) if it does, retention equals usefulness. This continues ourLLM-readout Bridge Paper (DOI 10.5281/zenodo.21937160). ※ The License for codes is AGPL 3.0-or-later