Multistream Vision Transformer Fusion with Stain-Physics Priors for Reliable Colorectal Histopathology Classification

Abstract

Discriminating hyperplastic polyps from sessile serrated adenomas (SSA) in colorectal histopathology is clinically consequential, with documented interpathologist agreement of only κ≈0.56. Existing deep‑learning approaches treat hematoxylin and eosin (H&E) images as entangled RGB tensors, discarding biologically distinct stain signals. We propose Stain-Physics-Guided Representation Disentanglement (SPGRD), which recovers this information by decomposing H&E images into hematoxylin and eosin concentration maps via Beer–Lambert inversion, routing each through an independent DINOv2-S Vision Transformer encoder, and combining the three streams through a Transformer fusion module with learnable softmax pool weights. On the MHIST benchmark, SPGRD achieves accuracy of 83.71±0.28% and AUC–ROC of 91.04±0.40%, an 8.8 percentage point gain over a single-encoder baseline under identical protocol, using only 1.8M trainable parameters (2.8% of total). The improvement is statistically significant across all seeds (p<0.001, Fisher-combined DeLong test) and Cohen’s κ of 64.94% exceeds documented interpathologist agreement. Five stabilization techniques constrain per-seed accuracy SD to 0.28% and AUC–ROC SD to 0.40% across five independent training runs, providing the low-variance reproducibility required for clinical decision-support deployment on a small, imbalanced dataset. Head-only transfer to CRC-VAL-HE-7K yields AUC–ROC of 99.37±0.53% with 770 parameters, confirming domain-general feature generalization across scanner and staining variability.