Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses

Abstract

Reading a transformer’s internal states in token space is easy to do and hard to trust: a logitlens on a single hidden state is dominated, at intermediate layers, by the generic tokens themodel would predict for almost any input. We read the difference instead. Subtracting twoclosely matched prompts’ hidden states and projecting through the unembedding cancels theshared component and surfaces what separates them, an operation equivalent to reading aRepE/ActAdd steering vector through a logit lens. Built into a training-free tracer that reads atevery position, sub-layer, and head and averages over designed baselines, it traces a compound-noun MLP→attention chain in Phi-2, confirmed there by activation patching, with the samedistinction recovered across three architectures by readout and probe rather than by patching;it reads what retrieval surfaces for real versus fictional entities, and reads metaphor as a setof domain-to-domain mappings rather than a single figurativity feature. A cross-seed controlmarks the boundary: across five networks differing only in initialization, the same distinctionsurfaces as almost entirely different tokens (top-10 overlap 0.08). What a computation looks likein token space is network-specific; the distinction it draws is not.