Application of multimodal deep learning in interior design material matching and colour scheme generation

Abstract

Material-color coordination is a complex issue in interior design, which is typically left to human judgment. Intelligent multimodal systems have been necessitated by the increasing need for personalized and computationally assisted design workflows. This study introduces a hierarchical multimodal deep learning (DL) system that combines visual and textual data to automatically categorize materials and produce context-sensitive colour schemes. The model is trained on a curated set of data that includes high-resolution interior images, sample materials, style labels, design text, and user preference descriptions. To improve model robustness and generalization, data augmentation techniques such as rotation, flipping, scaling, and colour transformations are applied to enhance dataset diversity. EfficientNet is used to extract visual features and use a pretrained GPT-based encoder to encode semantic data, which is simply late fused and input into a Hierarchically Improved Conditional GAN (Hi-CGAN) to be synthesized. The latest test also depicts that the quantitative performance is high with respect to the accuracy of classification, 97.5% accuracy, 97.0% precision, 98.0% recall, and 97.5% F1-score in material recognition tests. The visual-semantic alignment of the generated colour schemes by the Hi-CGAN with design contexts was high and was supported by better training time (1100 s), efficient interface response time (8 ms), and SSIM of 0.946. Altogether, the proposed framework that unites multimodal reasoning and hierarchical generative modelling can improve design automation, decrease manual labor, and facilitate the process of scalable and user-friendly interior design.