SepSpace: payload-free open-set network traffic classification via supervised contrastive representation learning and prototype-radius rejection

Abstract

Abstract Network traffic classification (NTC) increasingly needs to go beyond predefined categories and identify traffic from previously unseen classes, namely the open-set recognition problem in NTC. This problem remains challenging because encrypted traffic limits the practicality of payload-dependent features, while existing methods still lack representation learning tailored to open-set discrimination. This study aims to develop a payload-free open-set NTC framework that constructs an embedding space suitable for both known-class classification and unseen-class rejection. To this end, we propose SepSpace, which models network flows using meta-feature sequences. SepSpace introduces supervised contrastive learning to construct a discriminative embedding space with compact intra-class structures and separated inter-class boundaries. Combined with a prototype-radius rejection mechanism, SepSpace achieves effective unseen-class rejection while maintaining known-class classification performance. Meanwhile, to comprehensively evaluate open-set performance, we introduce a class-holdout evaluation protocol and conduct experiments on four benchmark datasets together with the conventional cross-dataset protocol. Experimental results demonstrate that, under the class-holdout protocol, SepSpace achieves the highest average area under the receiver operating characteristic curve (AUROC) of 0.9015 and open-set classification rate (OSCR) of 0.6351 among the evaluated methods; under the cross-dataset protocol, it obtains a higher average AUROC of 0.9396. The implementation and datasets of SepSpace are available at https://github.com/BITbla/SepSpace .