Today, Ant Lingbo has open-sourced LingBot-Touch: A tactile model for robots

After applying this model to dexterous hands, the accuracy of various fine operations of the robot has significantly increased. For example, the success rate of pulling a zipper has improved from just 15% relying solely on readings to 59%

This is indeed a very sophisticated model that can convert visual and tactile, magnetic tactile, or resistive tactile sensations into tokens that robots can utilize to better perform various tasks

The code and weights are already open-sourced on GitHub and Hugging Face

By the way, this screen can be touched: This article can also be touched by your fingertips

Why do robots need touch?

We see many visual solutions for robots that manipulate dexterous hands through visual perception and then let them perform various operations

But there is a problem: many times items are fragile or obstructed from view. For humans, I can determine how much force to apply and how to adjust my posture by the tactile feedback of my fingers, so why can't robots do the same?

For example, when twisting a bottle cap, the finger touches the mouth of the bottle to rotate it. From a visual perspective, it is impossible or difficult to distinguish whether the cap is actually rotating or if it is slipping, but with touch, we can clearly know whether we have genuinely twisted the bottle cap

For instance, when flipping through a book, a person can know whether a page has been turned with their eyes closed: the fingertips feel the arching of the paper page, supporting it and sensing whether it has slipped. Robots should also possess this capability

Three approaches to tactile sensors

We can achieve tactile feedback on dexterous hands by adding sensors, embedding some on the fingertips, thus where there is pressure, there will be perception

Below, I have created a simple, touchable SVG for everyone to experience

However, it is important to clarify: the sensing of the fingertips is different from that of the robot’s arm and palm

For example, sensing in the palm requires awareness of large areas of force, while fingertips need to discern very small deformations, which is usually challenging for a single sensor. Therefore, we have the following three approaches:

• Visual touch: a small camera is embedded in the fingertip to capture the deformation of the soft surface after being pressed, making contact details clear

• Magnetic touch: soft magnetic materials deform under force, and chips measure changes in the magnetic field, allowing for the perception of forces and sliding in multiple directions

• Resistive: the resistance changes under pressure; it is low-cost and can easily cover large areas of the palm

Each dexterous hand chooses according to its needs: Sharpa Wave uses visual touch on the fingertips, Xingdong Jiyuan's XHand uses magnetic touch, while Qiangnao Technology's Revo 2 has a resistive version

Why raw readings are difficult to use

Once the sensors are well-made, if you feed the data directly into the action model, the success rate may not necessarily be good

For instance, when pairing the Sharpa Wave dexterous hand with a π0.5 zipper pull, after feeding the raw readings from the fingertips, the success rate dropped from 39% to 15%

Another task specifically tests touch, called sorting Mahjong tiles, where the robot pinches a tile, recognizes the face with its fingertip, and places it into the corresponding dish. If the face is captured by the lens during recognition, that attempt fails; LingBot-VLA reads the raw readings directly, achieving a success rate of only 5%

Thus we can adopt a new approach: unifying the readings into tactile tokens

Here, LingBot-Touch first encodes the three types of readings and inputs them into the same model to output a uniform tactile token. Through these tactile tokens, we can record the finger touch positions, the magnitude & direction of force, and changes

Building on this idea, the Ant Lingbo team has also collected sensors from various sources, recording their measurement point quantities, arrangement methods, etc., and aligning all this sensor data to a single coordinate system for better usability by the robot

Where does tactile data come from?

Unlike the training data for large models, tactile data is difficult to collect through web scraping on the internet. Instead, it must be gathered in the real world, using special tactile gloves for hours at a time (two-thirds of the data are collected with people wearing gloves during daily tasks, while the remaining third is collected by remotely controlling the robot's dexterous hand)

The training of this model utilized a total of 4,983 hours of tactile data

As for the training method of this model, during training, a part of the recorded measurement points was masked, allowing the model to guess what kind of contact occurred in the masked area based on the remaining points (using a method called Tactile-JEPA)

How much did the success rate increase on the real machine?

The team tested 26 real machine tasks on two dexterous hands, using four types of action models (LingBot-VLA, π0.5, Diffusion Policy, and T-Rex)

Each model ran three groups for comparison: relying solely on vision, reading raw readings, and using tactile tokens. The success rates are as follows

For the single task of twisting bottle caps, the success rate rose to 90%, while in the Mahjong tile sorting test, LingBot-VLA improved from 5% with raw readings to 56.25%

In addition to these two tasks, the 26 tasks also covered fine operations, deformable objects, and coordination between both hands, all needing fingertip contact to complete

For example, popping out an SD card requires pressing it to eject it, then pinching it away (T-Rex's success rate increased from 32% relying solely on vision to 92%)

Another example is inserting a straw into a sealed cup, which requires puncturing the seal first and then inserting it to an appropriate depth (LingBot-VLA rose from 35% relying solely on vision to 59%)

Moreover, beyond these 26 tasks, the demonstration also included using a key to unlock, squeezing toothpaste, and sealing a bag, all of which require coordination of both hands

Can another dexterous hand still be used?

Since different manufacturers use different sensors for their dexterous hands, if you switch hands or sensors, can the trained tactile model still be used directly?

Of the 26 tasks, 20 were completed on the Sharpa Wave, while 6 switched to Xingdong Jiyuan’s XHand, a model outside the training data, which also performed smoothly, and various touch model parameters could be directly reused

As a result, the average success rate of π0.5 on the XHand’s six tasks rose from 48% with raw readings to 74.17%, while LingBot-VLA increased from 57.21% to 75.04%

In other words, manufacturers can select sensors according to their needs, and for reading this layer of touch data, they can directly use ready-made models

Only 0.00532 B parameters

This model requires only a minimum of 5.32 million parameters, which is 0.00532 B (the parameters used for those 26 real machine tasks all employ this single 5.32 million parameter model)

The reason for its small size is to run fast enough, allowing the robot to have a sense of agility comparable to instinctual actions. This is why it is said that as soon as the cup starts to slip, it quickly tightens to avoid dropping it

Of course, if computing power is sufficient and finer force judgment is more important, there is also a larger Small version available

Finally

In Lingbo's robot brain design philosophy, tasks are divided into four areas: see, touch, think, and do, among which LingBot-Touch fills in the "touch" part and has already been integrated with the action responsible LingBot-VLA and the spatial awareness responsible LingBot-Depth

As for the next step, the team plans to integrate visual, depth, touch, and temperature data into the same model, expanding touch from fingertips to the entire hand. If there are any updates, I will share them with everyone at the first opportunity

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到[email protected]，本平台相关工作人员将会进行核查。