A Capacitive Tactile Sensor Digital Twin for Real-Time Synthetic Data Generation and Sim-to-Real Transfer in NVIDIA Isaac Sim

Abstract

With advances in robotic manipulation in recent years, tactile sensing has become increasingly important in scenarios where visual information is unreliable or insufficient. However, the development of learning-based tactile algorithms and policies is limited by the cost and time required to collect large-scale physical datasets. While robotic simulation offers an alternative for synthetic data generation, current simulation platforms, such as NVIDIA Isaac Sim, lack integrated capacitive tactile sensors. This paper presents a finite element method (FEM)-based digital twin of a capacitive tactile sensor and an extension for NVIDIA Isaac Sim that enables the real-time generation of synthetic tactile data directly within the simulation environment. The proposed system extracts nodal deformations from the Isaac Sim PhysX engine and uses a convolutional neural network (CNN) to predict synthetic tactile maps that replicate the response of the physical sensor. We further demonstrate adaptability by retraining the model from an initial sensor to a second capacitive sensor, the Robotiq TSF-85, operating under the same sensing principle. The complete generation pipeline executes in 8.04 ms per frame on the laptop configuration and 6.71 ms on the workstation, enabling real-time operation at 60 Hz on both, and at 120 Hz on the workstation. The similarity of the generated tactile data for the Robotiq TSF-85 is evaluated using complementary similarity metrics, achieving a mean Structural Similarity Index Measure (SSIM) of 0.727 ± 0.16 and a mean Pearson correlation of 0.87 ± 0.16 against real measurements. To demonstrate the utility of the proposed framework, a shape-recognition task (cylinder, sphere, cube) was performed using only synthetic tactile data and evaluated on real-world sensor data, achieving an accuracy of 69.3% with zero real training labels. By enabling integrated tactile simulation and synthetic data generation within Isaac Sim, this work provides a practical tool for tactile perception using capacitive tactile sensors.