Chinese shadow puppetry—which uses silhouette figures and music to tell a story—is an ancient Chinese art form that’s been used by generations to charm communities and pass along cultural history. At Google, they’re always experimenting with how they can connect culture with AI and make it fun, which got them thinking: can AI help put on a shadow puppet show?
So Google created ShadowPlay, an interactive installation that celebrates the shadow puppetry art form. The installation, built using TensorFlow and TPUs, uses AI to recognize a person’s hand gestures and then magically transform the shadow figure into digital animations representing the 12 animals of the Chinese zodiac and in an interactive show.
Attendees use their hands to make shadow figures, which transform into animated characters and creates.
Google debuted ShadowPlay at the World AI Conference and Google Developers Day in Shanghai in September. To build the experience, they developed a custom machine learning model that was trained on a dataset made up of lots of examples of people’s hand shadows, which could eventually recognize the shadow and match it to the corresponding animal.
“In order to bring this project to life, we asked Googlers to help train the model by making a lot of fun hand gestures. Once we saw the reaction of users seeing their hand shadows morph into characters, it was impossible not to smile!”, says Miguel de Andres-Clavera, Project Lead at Google. To make sure the experience could guess what animal people were making with high accuracy, we trained the model using TPUs, our custom machine learning hardware accelerators.
An attendee’s shadow figure, recognized by the machine learning system, transforms into an animated silhouettte.