DisMed-LLM: De-Identifying Spanish Medical Text with Large Language Models

Abstract

Abstract Patient privacy and compliance with regulations such as the General Data Protection Regulation (GDPR) necessitate the robust de-identification of clinical texts. This study investigates the feasibility, computational cost, and limitations of de-identifying Spanish medical records using training-free Large Language Models (LLMs) compared to a supervised baseline. We evaluated five instruction-tuned LLMs on two Spanish clinical corpora (DisMed and MEDDOCAN) and benchmarked them against a fine-tuned multilingual BERT (110M parameters). On a held-out test set, the supervised baseline achieved a strict F1 of 0.904, outperforming the best training-free LLM (Gemma-2 9B, 0.796) while operating roughly 800 times faster. The LLMs’ performance gap was driven entirely by recall and concentrated in categories defined by corpus-specific conventions (e.g., institutions and locations) rather than universal surface forms (names and dates). Testing a newer model (Qwen3-8B) did not resolve this gap, indicating the limitation stems from annotation guidelines rather than model scale. However, when target-corpus annotations were unavailable, the dynamic reversed: in cross-corpus transfer, the supervised model’s F1 collapsed to 0.490, falling below all prompted LLMs, because it rigidly applied the source corpus’s conventions. Finally, novel disclosure metrics revealed that while LLM boundary errors are marginal (0.67%), they miss 26.7% of protected mentions entirely. We conclude that training-free LLMs are a highly adaptable, viable first-pass instrument for institutions lacking annotated data, but their residual disclosure rates mandate human review prior to data release.