FibreBench: stress-testing protein sequence encoders against exact combinatorial spectral invariants

Abstract

Protein representation models are usually evaluated on test sets split by sequence identity. Such splits cannot reveal what an architecture is unable to distinguish by construction. FibreBench is a benchmark built from an exact invariant instead. Each test pair consists of a human protein S and a different sequence T with the same multiset of k-mers and the same first (k−1)-mer, a member of the spectral fibre of S, obtained as a random Eulerian trail of the de Bruijn graph and verified exactly. The release contains 258 pairs: 12 gold pairs at k = 25 from earlier work (among them ZNF91, ZNF721, NBPF9 and POTEH, differing at 0.2–19.7% of positions), and 100, 100 and 46 pairs at orders k = 5, 10 and 25 from proteins of at most 1,022 residues. Every pair carries a single-substitution control and a shuffle control of the same protein. The drift ratio of a pair is the change of a mean-pooled sequence embedding under the fibre rearrangement divided by its change under one point substitution of the same protein. Seven encoders were evaluated on a CPU: PseAAC (lag 24), a randomly initialized window-25 convolutional network, ESM-2 at 8M, 35M, 150M and 650M parameters, and ProtBERT. The two local encoders give identical outputs on all 58 pairs of order 25, as theory requires. The protein language models are never exactly invariant, but they register an order-25 rearrangement about as weakly as a single substitution: on the k = 25 set the median drift ratio is 0.49–0.81 for ESM-2 and 0.90 for ProtBERT, and for ESM-2 650M 70% of these pairs move the embedding less than one point mutation. Sensitivity grows with model size only on the heavily rearranged pairs of order 5. ESM-2 3B, ESMFold, AlphaFold2 and UniPert were not evaluated on the available hardware and are listed as such rather than assigned numbers. This record contains the paper (PDF and LaTeX source), the pair set with controls (fibrebench_pairs.tsv.gz), the evaluation code, all per-pair measurements, the leaderboard and the figures. The UniProt snapshot used to generate the pairs is included. Companion papers: doi:10.5281/zenodo.22950095, doi:10.5281/zenodo.22959211, doi:10.5281/zenodo.22960439, doi:10.5281/zenodo.22963882. Code: https://github.com/Ruqing1963/fibrebench