Classical Indian dance inspires new ways to teach robots how to use their hands

Researchers at the University of Maryland, Baltimore County (UMBC) have extracted the building blocks of precise hand gestures used in the classical Indian dance form Bharatanatyam—and found a richer ‘alphabet’ of movement compared to natural grasps. The work could improve how we teach hand movements to robots and offer humans better tools for physical therapy. The work was published online Nov. 24 in the journal Scientific Reports.

Ramana Vinjamuri, a professor at UMBC and lead researcher on the work, has focused his lab on understanding how the brain controls complex hand movements. More than a decade ago, he and his research partners began searching for and cataloguing the building blocks of hand motions, drawing on a concept called kinematic synergies, in which the brain simultaneously coordinates multiple joint movements to simplify complex motions. The concept can be used to deconstruct a dazzling diversity of movements into a limited number of fundamental units, similar to how the hundreds of thousands of different words in the English language can be broken down into only 26 letters. 

Further inspiration struck when Vinjamuri attended a 2023 scientific conference on the brain, hosted by the Indian Institute of Technology Mandi in the serene foothills of the Himalayas. While brainstorming ideas for a session of the conference focused on ways that ancient Indian traditions might be applied to modern problems, Vinjamuri conceived a novel approach to deriving these building blocks—from the wide variety of precise hand gestures, called mudras, used in Indian classical dance to drive the storytelling element of the art form.

“We noticed dancers tend to age super gracefully: They remain flexible and agile because they have been training,” says Vinjamuri. “That was a huge inspiration for us when we started looking for richer alphabets of movement. With dance, we are looking not just at healthy movement, but super healthy. And so the question became, could we find a ‘superhuman’ alphabet from the dance gestures?”

Natural versus structured movements

As part of the newly published research, Vinjamuri and his students started by analyzing a data set of 30 natural hand grasps, used for picking up objects ranging in size from large water bottles to tiny beads. They found six synergies, akin to an alphabet of six letters, that when combined could account for nearly 99 percent of the variations in movements represented in the full data set. 

Using the same techniques, the research team also analyzed 30 single-hand mudras. They found six synergies that could account for around 94 percent of the mudras’ variations.

Crucially, the team then tested how well the six natural grasp-derived synergies could combine to construct unrelated hand motions—in this case 15 letters of the American Sign Language alphabet—compared to the mudras-derived synergies. The mudra-derived synergies significantly outperformed the natural hand grasp synergies on that task. 

“When we started this type of research more than 15 years ago, we wondered: Can we find a golden alphabet that can be used to reconstruct anything?” says Vinjamuri. “Now I highly doubt that there is such a thing. But the mudras-derived alphabet is definitely better than the natural grasp alphabet because there is more dexterity and more flexibility.”

Ultimately, Vinjamuri envisions coming up with libraries of task-specific alphabets that can be deployed depending on the needs, be it completing everyday household chores such as cooking or folding laundry, or something more complicated and precise, such as playing an instrument. 

Robotic helping hands

The team is currently developing techniques to “teach” robotic hands the alphabets of movements and how to combine them to make new hand gestures. The approach marks a departure from standard techniques of teaching robots to mimic hand gestures, and toward a method rooted in our understanding of how the human body and brain work.

The researchers are testing the techniques on a stand-alone robotic hand and a humanoid robot, each of which operates in a different way and requires a unique approach to translating the mathematical representations of synergies into physical movements.

The team has also made great strides developing cost-effective and pragmatic methods of testing and implementing their ideas. They use a simple camera and software system to recognize, record, and analyze movements, an important contribution to ultimately making cost-effective technologies that people could use in their homes, such as a virtual system to coach people through physical therapy sessions, says Vinjamuri.  

“Once I learned about synergies, I became so curious to see if we could use them to make a robotic hand respond and perform the same way as a human hand,” says Parthan Olikkal, a longtime member of Vinjamuri’s lab who is currently working toward his Ph.D in computer science. “Adding my own work to the research efforts, and seeing the results has been gratifying.”