Imagine a robot that can manipulate objects as deftly as a human, thanks to a combination of vision, language, and actions. GR-Dexter is precisely such a system, integrating hardware, a model, and data to control a robot with two dexterous hands. This approach enables long-horizon manipulation based on language instructions while focusing on complex platforms with a high number of degrees of freedom, such as hands with many joints. The system addresses challenges such as an expanded action space, frequent occlusion of the hands by objects, and the cost of collecting demonstrations. GR-Dexter uses a compact robotic hand with a high number of degrees of freedom, intuitive teleoperation for data collection, and a training recipe that combines robot trajectories with large-scale vision-language data and carefully selected cross-embodiment data.
Hardware and Control
The ByteDexter V2 robotic hand is a key component. This hand uses a linkage-driven mechanism for improved force transmission, durability, and ease of maintenance. As an upgrade over the previous V1 version, it adds an extra degree of freedom for the thumb, bringing the total to 21 degrees of freedom. The hand is smaller than before—219 millimeters tall and 108 millimeters wide. Each finger has four degrees of freedom, while the thumb has five, enabling a wide range of movements, including opposition gestures. The five fingertips are covered with dense arrays of piezoresistive sensors that measure normal forces with fine spatial precision at the fingertip, pad, and side of each finger.
For the bimanual system, data is collected through a teleoperation interface using Meta Quest VR for wrist position tracking, two Manus Metagloves for capturing hand movements, and foot pedals for controlling the arms. Two Meta Quest controllers are attached to the backs of the gloves to reliably track coordinated wrist and hand movements. This setup allows teleoperators to simultaneously control two Franka arms for long-horizon tasks. Human movements are converted into joint commands in real time, with kinematically consistent mapping through whole-body control. The system includes adaptive mechanisms for handling loss of visual tracking and preventing unsafe operations. Hand motion retargeting is formulated as a constrained optimization problem that incorporates wrist-to-fingertip vectors, thumb-to-fingertip vectors, collision avoidance, and a regularization term, solved using sequential quadratic programming.
GR-Dexter Model
GR-Dexter builds on GR-3 and uses a Mixture-of-Transformers architecture for a 4-billion-parameter VLA (vision-language-action) model. This model generates action chunks of length k based on a language instruction, observations, and the robot state. Each action includes arm joint actions, arm end-effector poses, hand joint actions, and fingertip positions.
Training involves joint learning from three data sources: large-scale vision-language data, cross-embodiment robot data, and human trajectories. To account for differences in the data, unavailable or unreliable action dimensions, such as specific joints, are masked. The vision-language data comes from GR-3 and covers tasks such as image captioning, question answering, image localization, and grounded captioning. The cross-embodiment data is drawn from open datasets such as the Fourier ActionNet Dataset, OpenLoong Baihu Dataset, and RoboMIND, which include various dual-arm dexterous manipulation tasks. Human trajectories are collected using VR devices such as Pico VR and supplement open datasets to provide greater volume and diversity.
To transfer cross-embodiment data, camera observations are standardized and retargeted to ByteDexter V2 by aligning the fingertips, preserving contact geometry independently of the joints. The trajectories are then resampled by task category to ensure balance. Human trajectories are filtered based on hand visibility and speed, then mapped into the same visual and kinematic representation as the robot data.
Experiments and Results
Experiments evaluate GR-Dexter on long-horizon bimanual manipulation and general pick-and-place tasks. It is assessed on tasks involving dexterous tool use, long-horizon execution, and out-of-distribution scenarios with new spatial configurations, unseen objects, and unseen instructions.
In baseline settings where the spatial arrangement of objects appears in the training data, GR-Dexter achieves a success rate of 0.97, comparable to 0.96 for the teleoperation-only model. In out-of-distribution settings with new arrangements, the success rate of the teleoperation-only model drops to 0.64, while GR-Dexter achieves 0.89. This demonstrates better generalization thanks to vision-language data.
Additional qualitative results include tasks such as vacuuming, where the robot holds a handheld vacuum with four fingers, presses the button with its thumb to turn it on, then presses it again to increase the power and sweeps up confetti. Another task is serving bread, where the robot grasps food tongs and removes a croissant from a container while the other hand holds a plate, then releases the tongs and places the croissant on the plate.
For general pick-and-place tasks in baseline settings, GR-Dexter achieves 0.93, outperforming the teleoperation-only model at 0.87. Without cross-embodiment data, it achieves 0.85. In settings with unseen objects and instructions, the success rate of the teleoperation-only model drops significantly, while GR-Dexter achieves 0.85 for objects and 0.83 for instructions, thanks to the greater robustness provided by cross-embodiment data.
GR-Dexter thus represents a practical step toward general-purpose dexterous robotic manipulation.
