A real-time hand tracker

[MUSIC PLAYING] NARRATOR: Teaching robots to manipulate the world like humans isn't a new idea. For decades, robotics has aimed to replicate the way we grasp, lift, and interact with our environment. But recreating human level dexterity is far more complex than it looks. Today, the question isn't whether robots can grasp and move objects, but how to make them do so with the flexibility, intelligence, and finesse that humans achieve effortlessly. A team of MIT engineers has designed an ultrasound wristband that precisely tracks a wearer's hand movement in real time. By training the system on an individual's motions, the wristband can learn and interpret hand movements, then communicate them instantly to a robot or virtual environment. The wristband is equipped with an ultrasound sticker the size of a smartwatch and added on-board electronics that are about as small as a cell phone. Once attached to a human wrist, the device produces ultrasound images of the wrists' muscles, tendons, and ligaments as the hand moves and is paired with an AI algorithm that continuously translates those images into the corresponding positions of the fingers and palm. The team trained the algorithm using carefully annotated ultrasound images, teaching it how different regions correspond to the many degrees of freedom in the human hand. In demonstrations, the team has shown that a person wearing the wristband can wirelessly control a robotic hand. As the person gestures or points, the robot does the same. With the same wristband, a wearer can also manipulate objects on a computer screen-- for example, pinching their fingers together to enlarge and minimize a virtual object. The researchers feel this could have immediate impact in virtual and augmented reality and also provide huge amounts of training data for humanoid robots in dexterity tasks, such as performing certain surgical procedures. Next the team plans to further miniaturize the wristband and continue to train its AI on a broader range of gestures and movements from volunteers with diverse hand sizes and shapes. Ultimately, they aim to create a wearable hand tracker anyone can use to wirelessly manipulate humanoid robots or virtual objects with high dexterity control.