TouchTherm constructs simulation-ready object assets that combine
object-scale geometry, contact-aligned tactile micro-geometry,
and dynamic thermal fields for multimodal sensory rendering.
TouchTherm converts real objects into reusable multimodal simulation assets by combining coarse collision geometry, registered tactile micro-height fields, and an object-specific dynamic thermal field.
Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry, but also the physical cues underlying tactile and thermal interaction. However, existing 3D datasets and reconstruction methods primarily represent object-scale geometry and visual appearance, overlooking the microscale surface structure needed for high-fidelity haptic rendering and the transient temperature dynamics required for temperature-aware interaction. We present TouchTherm, a framework for constructing simulation-ready visuo-tactile-thermal object assets from real-world objects.
For visual and tactile reconstruction, we combine a high-fidelity handheld structured-light scanner with multiview normal maps obtained from photometric stereo. The normal maps are registered to the scanned geometry and transformed into tangent space to recover local micro-height fields, which are queried by an optical tactile renderer, while the coarse mesh handles collision detection. For thermal reconstruction, we capture synchronized multiview infrared videos of natural cooling following controlled heating and reconstruct a physics-regularized dynamic thermal field. These representations are integrated with the visual geometry into a unified, simulator-ready object asset.
Experiments on 20 objects show that the reconstructed micro-height fields preserve dominant surface structures and recover higher-frequency details beyond the coarse geometry, while the dynamic thermal fields achieve held-out surface-temperature MAEs of 0.465 ◦C and 0.592 ◦C at 30 s and 45 s, respectively. The resulting tactile assets enable high-fidelity tactile simulation, including synthetic-to-real object recognition from tactile sensor observations, while a glove-based VR system demonstrates spatially and temporally varying thermal feedback. These results highlight the potential of TouchTherm for multimodal sensory simulation and temperature-aware virtual interaction.
An overview of the TouchTherm pipeline, multimodal object reconstruction, tactile and thermal evaluation, and downstream applications.
A unified object-level representation for visual geometry, tactile micro-geometry, and dynamic surface temperature.
Registered visual geometry, tactile micro-height fields, and dynamic thermal fields within a shared object frame.
Contact-conditioned optical tactile rendering from reconstructed micro-height fields while preserving a coarse collision mesh.
Object-specific surface-temperature dynamics reconstructed from multiview infrared cooling observations.
Evaluation on 20 objects together with synthetic-to-real tactile recognition and wearable thermal feedback in VR.
The acquisition streams are registered to a common object frame, where tactile micro-geometry and dynamic thermal fields are reconstructed in two complementary branches.
Structured-light scanning provides the metric object mesh used for visual rendering and coarse collision geometry.
Photometric-stereo normal maps are registered to the mesh and converted into local tangent-space micro-height fields for tactile rendering.
Multiview infrared cooling observations are fused on the surface and used to identify object-specific thermal dynamics for runtime rollout.
We acquire metric object geometry, multi-view surface-normal observations, and dynamic thermal measurements using three complementary sensing setups.
An EinScan Pro 2X V2 structured-light scanner captures the metric object geometry used for visual rendering and coarse collision.
A fixed smartphone captures each viewpoint under multiple illumination directions, providing image stacks for surface-normal estimation.
Multiple infrared cameras record the spatially varying cooling process following controlled thermal excitation.
The collection spans flat, curved, cylindrical, and free-form geometries together with smooth, granular, embossed, and anisotropic surface structures.
We evaluate tactile appearance, dynamic thermal prediction, and the downstream utility of the reconstructed multimodal assets.
Real–Sim tactile acquisition, appearance evaluation, and micro-geometry ablation across the reconstructed object assets.
Across 20 objects, matched real and simulated GelSight observations preserve dominant surface structures and contact-scale patterns beyond what is available from coarse geometry alone.
The proposed PnP-based metric surface sampling and tangent-space reconstruction also achieves the strongest performance in the tactile ablation.
We collect real tactile observations using a GelSight Mini and reproduce corresponding approach-and-press procedures in Isaac Sim before evaluating Real–Sim tactile appearance.
12 representative real-object pressing sequences using a GelSight Mini.
6 representative simulated approach-and-press sequences using the reconstructed object assets.
Dynamic surface-temperature reconstruction and held-out long-horizon thermal rollout evaluation.
The reconstructed temperature fields preserve both global cooling behavior and spatial temperature patterns over held-out future frames.
Surface MAE remains below 0.6 °C at the reported 30 s and 45 s prediction horizons.
Move the heat source over the reconstructed object surface or position it independently along the X, Y, and Z axes to explore the real-time thermal evolution.
Beyond reconstruction quality, we evaluate whether the assets provide useful signals for perception and immersive interaction.
A ResNet-18 trained only on simulated GelSight observations transfers more effectively to real tactile images when using TouchTherm micro-geometry than coarse geometry alone.
A wearable thermal-feedback platform demonstrates spatially and temporally varying thermal cues reconstructed from the object-level dynamic thermal field.
Code, reconstructed object assets, and the simulation pipeline will be released upon acceptance.