A two-stage efficient fine-tuning framework for vertical domain text embedding

Abstract

Text embedding in vertical industrial domains provides a critical foundation for professional information retrieval, semantic matching, and intelligent decision-making. However, general-purpose embedding models often fail to meet the demands of complex industrial applications because they lack effective modeling of domain knowledge and fine-grained semantic distinctions. To address this limitation, we propose a two-stage efficient fine-tuning framework for vertical-domain text embedding. In the first stage, domain-adaptive contrastive learning is employed to align the base semantic space with domain-specific data, thereby enhancing the model’s representation of technical terminology and contextual knowledge. In the second stage, joint fine-tuning is performed across multiple tasks, including retrieval, classification, semantic textual similarity, sentence-pair discrimination, and ranking, while a dynamic hard negative mining strategy is introduced to continuously strengthen the model’s ability to distinguish highly similar and easily confusable samples. Furthermore, to address the substantial differences in task objectives and the inconsistency of optimization dynamics in multitask training, a dynamic-awareness and priority-guided adaptive scheduling mechanism is proposed to adjust task sampling weights according to task learning progress and relative importance, thereby improving the synergy and stability of multitask optimization. Experiments conducted on a coal mining domain dataset demonstrate that the embedding model built upon the proposed framework consistently outperforms mainstream embedding models on Accuracy@1, Recall@3, NDCG@10, and MRR@10. Additional comparative experiments and ablation studies further verify the overall advantages of the framework in terms of stability, scalability, and industrial applicability. The proposed method therefore provides a robust, practical, and scalable solution for developing high-performance embedding models in vertical domains, facilitating more accurate retrieval, knowledge representation, and decision-support applications.