KEMIT: Knowledge Extraction and Model Injection Transfer

Abstract

KEMIT (Knowledge Extraction and Model Injection Transfer) is a framework for cross-architecture knowledge transfer between language models. The framework uses three phases: prototype knowledge extraction from a teacher model, linear transcoding between teacher and student representation spaces, and direct embedding-level knowledge injection into the student model. This record contains the research paper describing the KEMIT framework and its experimental evaluation.