Panasonic's T-Rex adds fast fingertip touch to a robot model and nearly doubles success on 12 contact tasks

A robot model that reads fingertip touch about 20 times a second succeeded in 65% of trials across 12 contact-heavy tasks, from turning pages and moving eggs to opening a lock with a key, pulling out a card and screwing in a light bulb, against 35% for the strongest earlier method, Panasonic Holdings said on Nov. 5. Panasonic and its US research arm built T-Rex (Tactile-Reactive Dexterous Manipulation) with researchers from UC Berkeley, NVIDIA, Stanford, Sapienza University of Rome and ItalAI, and the team presents it at the Conference on Robot Learning in Austin on Nov. 9–12.

Most of that gain comes from touch. With the tactile input removed, T-Rex averaged 42%, Panasonic said. On that comparison, touch is worth 23 of the 30 points over the baseline and everything else in the recipe 7. For the jobs Panasonic has in mind, inserting, removing and screwing in parts on factory lines and handling soft or fragile goods in logistics, the limiting parts look like tactile hands and tactile data rather than a larger vision model.

A slow planner and a fast touch loop share one model

Most VLA models work mainly from camera images and, when they do use touch, treat it as a static input, the release says. T-Rex splits the job. A module that reads vision and language sketches the motion at about 5 Hz, and a tactile module reads contact at about 20 Hz and corrects the motion for slips or pressing too hard, without waiting for slow image processing to run again. The paper describes a variable-rate Mixture-of-Transformers with a temporal tactile encoder.

100 hours of short contact skills instead of long tasks

Training has three stages: 22,889 hours of first-person human video, then 100 hours of two-handed robot data with touch (more than 7,700 episodes), then about 100 demonstrations for each task. The tactile set pairs more than 200 household objects with 22 basic motions such as inserting, peeling, squeezing and screwing, rather than recording a few long tasks. It was collected by teleoperation on a Dexmate Vega-1 with two 7-DoF arms and two 22-DoF Sharpa Wave dexterous hands, a 58-DoF robot in total; each of the ten fingertips records a deformation depth map and a six-axis force reading, according to the project page.

The authors include Jitendra Malik, Pieter Abbeel, Ken Goldberg, Fei-Fei Li and NVIDIA's Jim Fan, and the preprint has been on arXiv since June 15. Panasonic's release leads with its own name; on the paper, one of 34 authors lists Panasonic as an affiliation.

All 12 tasks call for delicate force control or soft objects, which is where an ablation would favor tactile sensing most. The team has open-sourced 50 hours of the teleoperation data, so other labs can check whether the 23 points survive on tasks they pick themselves.