Bridging the Domain Gap Visually and Linguistically: Multistage Transfer Learning for Handwritten Text Recognition on Dutch Sticky Notes under Data Scarcity

Abstract

Handwritten text recognition (HTR) for modern Dutch handwriting suffers from a lack of training data that is both rights-free and compliant with the GDPR, under which handwriting can qualify as personal and possibly biometric data. Multistage transfer learning was reviewed by Ayana et al. (2024) as a remedy for data scarcity in medical image analysis. This paper applies this method to HTR of handwritten Dutch sticky notes. A CNN backbone is pretrained on isolated handwritten letters and then extended with a BiLSTM-CTC head and trained in successive stages on data of increasing domain relevance: handwritten English words from US census records, synthetic Dutch text lines rendered in 25 open-source fonts, and a small sticky-note dataset created for this study. The distance between each source dataset and the target domain is quantified visually (Fréchet distance) and linguistically (Kullback-Leibler divergence over character and bigram frequencies). The two intermediate datasets bridge the gap in different ways: census data is visually closest to the target, synthetic Dutch lines linguistically closest. Ten training configurations, including ablations and a changed stage order, are evaluated on a test set with seen handwriting (201 lines) and an independent test set with unseen writers (102 lines). Intermediate stages improve generalization compared to a direct jump: the full pipeline (letters → census → synthetic → sticky notes) reaches 45.7% character error rate (CER) on unseen handwriting, against 63.1% for the direct jump, and 13.7% on seen handwriting. The census stage contributes most: only the pipelines that include it also achieve a lower CER than a baseline trained on the target data alone (50.7%), although the confidence intervals overlap. In a bootstrap and Friedman analysis on the 91 lines read by all models, the best pipeline (45.8% CER) performs comparably to the pretrained Loghi HTR model (46.9% CER), which was not trained on sticky notes, with overlapping confidence intervals. Absolute accuracy on unseen handwriting is not yet sufficient for production use.