LeaDeiT: data-efficient image transformer with knowledge distillation for leather species identification

Abstract

Distinguishing the animal species from which the leather is derived becomes a pivotal factor in assessing the standard of leather goods and conserving the species under threat. Traditionally, this approach necessitates human intervention, which is time-intensive, requires significant labour, and is susceptible to discrepancies. Machine learning (ML) and deep learning (DL) techniques have proven remarkably efficient in automating this process, optimising performance and accuracy. Recently, vision transformer (ViT)-based DL models have replaced the existing convolutional neural network (CNN)-based DL models for image classification, owing to their increased ability to handle long-range dependencies. Accordingly, this paper presents a data-efficient image transformer-based ViT model, ’LeaDeiT’ for predicting the leather species using a dataset featuring 10,000 images captured from various regions of the leather. The proposed model is a teacher-student-based framework, where the ViT-based student model learns from the CNN-based teacher model (MobileNetV3-Large) via distillation token by encapsulating the teacher’s predictions through a knowledge distillation approach. The teacher model was trained by employing the process of fine-tuning. In the student model, leather images are partitioned into patches, linearly embedded, and combined with class and distillation tokens. Then the patch sequence is processed by the transformer encoder through the multi-head self-attention (MHSA) mechanism. The proposed model demonstrated superior performance with a species prediction accuracy of 97.70%, surpassing the baseline CNN models. Thus, this research offers an optimised approach for automatically predicting leather species, leading to notable improvements in the leather production quality control processes.