Federated Distillation: A Survey

Abstract

Federated learning (FL) enables collaborative model training without centralizing clients’ raw data, but parameter exchange can impose substantial communication costs and usually requires compatible model architectures. Federated distillation (FD) instead transfers predictive or intermediate knowledge through knowledge distillation (KD), which can support heterogeneous models and reduce communication in suitable settings. These benefits are not automatic: their magnitude depends on the transmitted knowledge, reference data, communication schedule, and threat model. This survey provides a challenge-centered synthesis of FD. We first formalize a representative public-data, response-based FD protocol and then organize existing methods by the FL challenges they address, including data, system, and model heterogeneity, client drift, catastrophic forgetting, communication cost, and privacy risk. We further review applications in industrial engineering, computer vision, natural language processing, and healthcare, including recent federated adaptation of large language models. Finally, we identify five research priorities: richer transferred knowledge, joint treatment of interacting forms of heterogeneity, communication-aware distillation, threat-model-driven privacy protection, and standardized theory, benchmarks, and evaluation.