Resolving NIC Rename Race Conditions in Multi-NIC Bare-Metal Provisioning at Scale

Abstract

Network interface naming on multi-NIC Linux systems suffers from a well-documented race condition in systemd-udevd that has remained unsolved since 2017 (systemd/systemd#16665, #7293, #11436). When multiple network interfaces are present, parallel udev worker threads attempt to rename interfaces simultaneously, causing EEXIST failures when a target name is temporarily occupied by another interface awaiting its own rename. This issue, previously cosmetic under NetworkManager (which matches interfaces by MAC address), has become a critical provisioning-breaking failure as the industry migrates to systemd-networkd, which matches interfaces by name — turning a failed rename into complete loss of network connectivity. The problem is especially acute on AI/ML GPU servers with 8-10 NICs (RDMA, InfiniBand, management, storage), where the race condition is near-certain on every boot. This paper presents a three-layer boot-time convergence approach combining (1) declarative systemd .link file configuration, (2) kernel parameter management, and (3) a retry-based systemd oneshot service that detects and recovers from rename failures before dependent network services start. The solution requires no kernel or systemd modifications and is OS-agnostic, applicable to any systemd-based Linux distribution (RHEL 9+, Ubuntu 22.04+, Fedora, SUSE). Deployed in production across diverse multi-NIC bare-metal hardware configurations at hyperscale, the approach achieves 100% deterministic interface naming, converting a previously non-deterministic failure mode into a guaranteed-convergent boot process — with the vast majority of boots succeeding immediately and only a small minority requiring a single automated retry (30-45 seconds).