Scientific Solutions (United States)

Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction

The corpus-mean direction of a transformer’s residual stream is a component shared acrossall inputs, and is commonly removed by mean-centering before representational analysis on theassumption that it cannot carry input-specific signal. We present evidence that it is a functional...

Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction

The corpus-mean direction of a transformer’s residual stream is a component shared acrossall inputs, and is commonly removed by mean-centering before representational analysis on theassumption that it cannot carry input-specific signal. We present evidence that it is a functional...

Quantitative Aspects of Non-targeted Analysis for Organic Extractables/Leachables; Insights Secured by Considering the FDA CLAP List Compounds

Substances released from a drug product’s manufacturing, packaging, and delivery system are termed leachables. Medical device leachables are released during their clinical use. Packaged drug products and medical devices are profiled for leachables (and/or extractables as probable...

Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses

Reading a transformer’s internal states in token space is easy to do and hard to trust: a logitlens on a single hidden state is dominated, at intermediate layers, by the generic tokens themodel would predict for almost any input. We read the difference instead. Subtracting twoclo...