Comparing Temporal Averaging and Transformer Encoders for Video-Based Cattle Behavior Classification Under Severe Class Imbalance

Abstract

Automated cattle behavior monitoring from video is challenging due to severe class imbalance and the need for efficient temporal modeling. We compared four architectures on 542 videos containing standing, lying down, and rumination; rumination appeared only six times, while two dataset classes were absent. ResNet-50 and EfficientNet-B3 backbones were paired with either a Transformer encoder or temporal averaging and trained with three-fold stratified cross-validation and weighted cross-entropy. EfficientNet-B3 with temporal averaging (Eff-B3 + Avg) achieved the highest out-of-fold accuracy (85.61%) and best calibration (ECE = 0.074), while requiring 41 MB and 28 ms per video. Its Transformer counterpart scored 84.13% despite four times more parameters. ResNet-50 + Transformer reached 84.69% but was poorly calibrated and storage-intensive. Frame ablation identified 16 frames per video as the best accuracy-speed trade-off. No model correctly classified any rumination example, underscoring that rumination was not learnable under the specific weighted-cross-entropy and stratified-sampling strategies evaluated here; this finding does not establish a general impossibility of learning extremely underrepresented behaviors under alternative training strategies such as focal loss, oversampling, anomaly detection, or self-supervised pre-training. Rare-class analysis, treating support ≤106 as rare, showed high-confidence rumination errors: mean confidence was 0.891, but only 1.01% of probability mass was assigned to the true class, yielding a class-wise expected calibration error of 0.891. These findings indicate that calibration post-processing cannot rescue a class with zero true positives.