Adding Touch to Robot Manipulation: Vision-Tactile Fusion and Learning Precise Insertion

Why touch was needed

Robot manipulation policies (models that take observations and output the next action) have so far relied mainly on camera images. But at the moment fingers grasp an object, the most important information is often not captured well by cameras. TouchWorld paper points out that vision and language convey semantics and geometry but cannot reliably reveal 'hidden contact states' such as force, slip and contact stability. DeCAL paper also cites severe occlusion and complex contact dynamics as reasons dexterous hand manipulation is difficult.

Picture plugging a plug into an outlet. TF²-Steering paper explains that in the final stage of inserting a plug into a tight socket, visual information helps very little, and people correct their motion using expected contact sensations and actual fingertip feel. The CMT study on insertion tasks paper likewise starts from the observation that precise, contact-rich interaction cannot be solved with vision alone.

Mounting a wrist camera for a closer view is one option, but it has limits. The wrist-camera replacement study paper notes that in contact-rich manipulation, the contacts that must be made and maintained can be occluded even from a wrist camera, and focuses on the fact that tactile sensors on the fingers measure contact directly without needing a view.

Core idea: align with the eyes, insert with the fingertips

The key is to give the two senses different roles. Think of inserting a key in a dark entryway. You use your eyes to bring your hand near the keyhole, and your fingertips to line it up and turn it at the end. MuxGel paper likewise summarized from grasping experiments that visual feedback handles alignment while tactile feedback handles contact interaction.

This division of labor also involves 'time scales'. TouchWorld paper criticizes most existing policies for treating touch only as a low-frequency observation inside a single monolithic action model. That ties slow task reasoning, action generation and fast contact feedback into one loop. Slip happens in an instant, and correcting the motion a few beats late is already too late. So a major trend in the field is to use touch both as a 'predictive reference for how contact should evolve' and as a 'fast feedback signal'.

The trade-off is also clear. Adding touch makes sensors and hardware more complex and data collection more expensive. In return, you get contact information that vision alone cannot provide. The sections below cover design choices that aim to reduce this cost while capturing as much of the benefit as possible.

How it works: from sensor to action

The first step is obtaining the tactile signal. Vision-based tactile sensors widely used in this field capture, with an internal camera, the deformation that occurs when a soft gel on the fingertip is pressed (the GelSight family). These sensors usually use an opaque coating, so the outside cannot be seen. MuxGel paper made the coating a checkerboard pattern, leaving transparent windows between tactile regions. A single camera captures both the external view and touch at once, and a U-Net-based reconstruction model restores both signals at full density.

The second step is deciding in what form to feed the tactile signal. The tactile-cue comparison study paper extracted three cues from the same tactile image: an RGB image showing the contact appearance, normal and depth maps capturing geometry, and marker flow capturing tangential deformation (shear). In 720 real-robot trials, touch raised the overall success rate from 57.8% to 78.9%. Among single cues, marker flow performed best at 74.4% and was especially important for regulating contact once it had been made. Processing each cue with a separate encoder (cue-factorized encoding) also consistently improved performance. This means each cue has different properties and needs its own processing.

The third step is fusion and action generation. Representations produced by the vision encoder and the tactile encoder are combined with attention (CMT, paper), or a gate decides how much to weight touch depending on whether there is contact (DeCAL, paper). The fused result is fed into an action generator such as ACT, Diffusion Policy or a VLA (vision-language-action model). There are also architectures, like TouchWorld paper, where a high-level policy outputs an action chunk and a high-frequency tactile policy adds a residual (a small correction) on top.

Precise insertion: fusion must be done 'well' to pay off

It might seem that adding touch always helps, but in practice it does not. The CMT study paper notes that naive vision-tactile fusion in prior work did not yield consistent improvements. TF²-Steering paper explains why in more detail: policies are biased toward vision, tactile data is scarce, and contact events are rare. It reported experimentally that when these three combine, directly injected tactile signals can be ignored or can damage the capabilities of a pretrained policy, actually lowering performance.

CMT's paper answer is structure and physical priors. It organizes and combines wrist-camera images and tactile signals with self-attention and cross-attention, and stabilizes tactile embeddings with a regularization that encourages balanced forces on both fingers (inspired by principles of human motor control). It achieved a 96.59% insertion success rate on the TacSL benchmark, outperforming naive fusion and gated fusion. This is on par with a 'privileged' setup that directly receives contact force values that are hard to obtain on real robots (wrist camera + contact force, 96.09%).

TF²-Steering paper took a different path. It keeps a strong vision-based foundation policy unchanged and learns a residual steering that uses touch to slightly bend the 'velocity field' followed by the flow-matching action generator inside that policy. Using a tactile MoE architecture, it combines tactile prediction (upcoming contact) with real-time tactile feedback, and goes through a three-stage training sequence. Its average success rate in simulation was 10.7 percentage points higher than the strongest baseline. The core of the design is preserving the existing policy's capabilities while correcting the motion only at moments when touch is needed.

How the field is developing: data, evaluation and world models

Data is as pressing a challenge as fusion architecture. ViTacWorld paper cites the facts that real tactile interaction data is expensive, tied to specific hardware, and lacks diversity in tasks and scenes as obstacles to scaling. So it combined public real tactile datasets with its own simulation environment to build an action-conditioned vision-tactile world model. Given robot actions, it predicts time-aligned video and tactile responses. The work exploits the point that because tactile signals are directly tied to physical contact, the Sim-to-Real gap may be smaller than for pure video. It is pretrained on large-scale real and simulated Trajectories, then fine-tuned on real policy Rollouts, and used to improve policies with synthetic Rollouts or to evaluate policies in advance.

Evaluation infrastructure is also taking shape. Bench2Dex paper unifies 12 dexterous robot hands with varying finger structures and sensor layouts under a single simulated tactile interface. Rather than imitating the output of a particular physical sensor, this interface converts local contact geometry into image-like tactile observations, giving the same format regardless of hand shape. It includes 26 bimanual tasks and about 1.3K human teleoperated demonstrations. Robustness is assessed by splitting perturbation types into an 'invariant' axis, where the correct action does not change, and an 'equivariant' axis, where the action must change along with the perturbation.

SoftVTBench paper changed the evaluation criteria themselves. It simulates soft objects with the finite element method (FEM) and measures Goal Success and Safety Success separately. Safety Success counts only if the object is not dropped and its maximum deformation stays below an object-specific threshold. Experiments showed that many Rollouts that achieved the goal violated physical safety. This means looking only at success rate greatly inflates policy performance. Adding touch kept goal success roughly the same while raising Safety Success on object-centric deformable-object tasks from 21.4% to 35.6%.

On the policy architecture side, TouchWorld paper splits the hierarchy into four layers: a high-level planning layer that uses vision and language to plan subtasks and predict tactile subgoals, tactile world model prediction, a vision-tactile goal-conditioned policy, and high-frequency tactile residual correction. DeCAL paper unifies understanding, imagination and action generation on a Mixture-of-Transformers architecture, and adds contact-aware gating and joint vision-tactile latent imagination (envisioning the future of both senses together), reporting an average success rate of 71% and a progress success rate of 83.4%.

Where things are heading

Recent research is moving beyond 'whether to use touch' toward examining 'what to use and how to use it effectively' through controlled experiments. The wrist-camera replacement study paper added two low-cost tactile arrays to a vision-tactile ACT policy and removed wrist images from both training and deployment. Comparing across three contact-rich tasks with demonstrations, architecture and optimization settings held fixed, adding a wrist camera raised success rates by 18.3–28.3 percentage points, while adding touch raised them by 31.7–60.0 percentage points. It achieved higher success rates than the vision policy with a wrist camera on all three tasks, with the difference statistically significant on two of them. The authors cautiously present this as evidence that touch is an effective alternative to wrist cameras 'in the contact-heavy settings evaluated'.

As seen earlier, the tactile-cue comparison study paper points toward splitting tactile images into physically interpretable cues rather than feeding them in whole. It also found that the contribution of touch differs across task phases. This signals a need for designs that use touch according to the contact phase rather than at a constant weight throughout.

TF²-Steering paper represents the direction of keeping an already well-trained vision foundation policy and only 'steering' it with touch. Given how hard it is to train a large VLA together with touch from scratch, this approach of adding a tactile correction layer on top of an existing policy is worth watching in practice as well.

Limitations and open issues: what to consider in practice

First, if adding touch has no effect, suspect the fusion method first. As CMT paper and TF²-Steering paper show, naively concatenated touch can be ignored or can damage existing capabilities. Experiments comparing fusion methods (cross-attention, gating, residual correction) and training sequences are needed. The form in which tactile signals are fed (RGB, depth, marker flow) also changes results substantially paper.

Second, there is the gap between hardware and simulation. Bench2Dex paper notes that tactile hardware design has not yet converged, and simulated tactile signals differ from real sensor measurements. So the benchmark chose a common interface that does not try to reproduce any particular sensor. The cost is that simulation results are hard to transfer directly to physical sensors. ViTacWorld paper includes a fine-tuning stage on real policy Rollouts for the same reason.

Third, evaluation metrics. As SoftVTBench paper shows, measuring only success rate can count runs where the object was dropped or pressed too hard as 'successes'. If you handle objects where damage matters, such as food, medical consumables or soft packaging, you need to define safety metrics separately. Also, most figures in this article come from specific benchmarks or lab settings and should not be read directly as performance on a real production line.

Fourth, when can touch be deferred? For pick-and-place tasks with brief contact, good visibility and little need for force control, the benefit of touch may be small. Conversely, for tasks where the final contact phase is decisive, such as inserting, tightening, retrieving occluded objects, or tool use that requires sustained contact, touch is worth considering first paper.

Further study

As a starting point, CMT paper is recommended first. On the concrete task of precise insertion, it shows briefly and clearly 'why naive fusion fails and why structured fusion works'. Reading the tactile-cue comparison study paper next gives a sense of how to design tactile inputs.

If you are interested in policy architecture, read TouchWorld paper and DeCAL paper together and compare the two approaches of hierarchical separation and gating. If you want to add touch while keeping an existing VLA, TF²-Steering paper is the right fit.

If you design data and evaluation, look at ViTacWorld paper, Bench2Dex paper and SoftVTBench paper in that order. MuxGel paper offers a sensor hardware perspective, and the wrist-camera replacement study paper provides a basis for deciding whether to invest in wrist cameras or touch.