SwapBot: Tool-Centric Robot Manipulation
via Interchangeable End-Effectors
and Tactile Sensing

Tool swapping

Low-cost adapters quickly convert common household tools into robot end effectors.

One robot-side interface supports both passive tools (knife, brush, spoon, fork) and articulated tools (tongs, chopsticks) through a pass-through mechanism.

Setup overview

Autonomous tool swapping

5× times speed

1. Approach. The robot approaches the tool docking station.
2. Release. The electromagnet releases its hold, leaving the current tool on the dock.
3. Attach. A new tool is then attached: a kinematically guided gear ensures reliable engagement under small alignment errors, and the electromagnet provides sufficient holding force, up to 50 N, for secure attachment during forceful manipulation.

Tactile sensing

Compact, tool-specific piezoresistive sensors provide task-relevant contact information.

A force-sensitive Velostat film is sandwiched between top and bottom electrodes of laser-cut copper-nickel conductive fabric and converts normal force into an electrical signal via the drop of resistance.

Instrumented tool

Sensor placement

To accommodate handles of varying sizes, the sensors are installed between the TPU and PLA layer, and the gap to the handle is filled with soft silicone, so the sensors comply with different handle shapes and stay securely in place.

Sensor characterization

Larger electrodes exhibit higher sensitivity, but also cause non-smooth signals with sudden jumps under load. Smaller electrodes have smoother signals and a wider measurable range.

To balance sensitivity, detection range and signal fidelity within ADC limits, we select a 2 mm electrode width, which provides sufficient sensitivity while maintaining smooth, stable responses.

Characterization of tactile sensors, showing force–response curves for electrodes of different sizes.

Teleoperation

During teleoperation, SwapBot reduces task completion time and operator workload.

Our teleoperation interface combines tool-matched leader devices with real-time visualized tactile feedback, providing operators with an intuitive interface for contact-rich manipulation. By using similar tools on both sides, e.g., chopsticks to control chopsticks or a knife to control a knife, operators interact with familiar household tools, making control feel natural.

Tool-matched devices for teleoperation

User study

Long-horizon cooking tasks

This interface also supports demonstration collection for more complex tasks.

Hierarchical policy

The policy has two components: the high-level tool-selection policy π_sel determines which tool to use at each stage of the task; the low-level, tool-specific control policies π_tool are trained to execute fine-grained manipulation skills conditioned on visuotactile inputs. Long-horizon tasks are segmented into subtasks with a subgoal image for each stage, manually specified before execution.

Framework overview

Results

Policies trained with tactile sensing outperform RGB-only baselines.

As a testbed, we use the challenging real-world task of preparing pan-fried tofu for quantitative experiments: the tofu is fragile and easily breaks if mishandled, requiring both gentle and dexterous manipulation. We used our teleoperation interface to collect 100 demonstrations for each stage and evaluated each tool-specific policy across 20 trials in a 20 cm × 20 cm workspace. Additionally, we provide qualitative demonstrations of other tool-centric tasks, stirring salad and flipping meat.

Skill by skill: RGB only vs. RGB w/ tactile

Policies trained with RGB and tactile sensing achieved consistently higher success rates and were robust to small tofu shape changes and variations on initial position: up to 85% in brushing, 70% in tong picking and 90% in knife cutting.

• Brushing 70% → 85%
• Knife cutting 75% → 90%
• Tong picking 45% → 70%
• Stirring salad
• Flipping meat

Data efficiency

As the amount of data increases, tactile signals consistently provide a boost over RGB-only baselines and yield larger gains in low-data regimes.

Brushing task

Tong picking task

Knife cutting task

Success rate of policies trained with different proportions of demonstration data. We average success rates over 20 evaluation rollouts per seed and run three random seeds; we then apply paired t-tests for significance (* p < 0.1, ** p < 0.05, *** p < 0.01).

Long-horizon task

Discussion

RGB-only policies frequently failed due to partial observability of tool–object interactions, especially at the tong tips, where initial contact and early slip cues are difficult to observe, leading to poor contact regulation, slipping, or object damage. The camera viewpoint also matters. With only the egocentric camera (plus tactile), success rates stay near zero in tong picking and knife cutting, because its angled viewpoint (greater than 45°) localizes the tofu poorly; with only the top-down camera (plus tactile), success rates reach at most 50% in brushing and cutting, because that view localizes the tofu reliably but lacks the local visual detail needed for fine control. In the full long-horizon task, most failures were caused by tofu slippage or out-of-distribution states not covered by the demonstrations, highlighting the need for broader training data and dedicated recovery strategies. Nevertheless, within the demonstration distribution, we qualitatively observe recovery from external perturbations: as shown below, when a person pushes the tofu away during execution, the visuotactile policy re-approaches and re-grasps the tofu.

Robustness and recovery

Limitations and future work

The current tactile sensors remain low resolution and prone to wear, sensor placement is fixed by human intuition, and policy learning depends on predefined task segmentation. Future work will explore more durable, higher-resolution sensing, data-driven strategies for sensor placement, and autonomous policy sequencing without manual subgoal definitions.

BibTeX

@article{swapbot2026,
  title   = {SwapBot: Tool-Centric Robot Manipulation via Interchangeable End-Effectors and Tactile Sensing},
  author  = {Zhang, Yunchu and Song, Ray and Jiao, Junhan and Ke, Liyiming and
             Yang, Yuchen and Gao, Ruihan and Romig, Emma and Srinivasa, Siddhartha and Luo, Yiyue},
  journal = {arXiv preprint arXiv:2610.40646},
  year    = {2026}
}