Meta develops AI to bring children's drawings to life

Meta (formerly Facebook) researchers have developed an Artificial Intelligence (AI) system that can identify and automatically animate the human-like figures in children's drawings with a high success rate, and without any human guidance.

"Humans" in children's drawings come in many different forms, colours, sizes and scales, with little similarity when it comes to body symmetry, morphology, and point of view.

"We're excited to announce a first-of-its-kind method for automatically animating children's hand-drawn figures of people and humanlike characters (a character with two arms, two legs, a head, etc.) that bring these drawings to life in a matter of minutes using AI," Meta researchers said in a statement.

By uploading the artwork to its prototype system, parents and children can experience the excitement of watching their drawings become moving characters that dance, skip and jump.

"They can even download their animated drawings to share with friends and family. If parents choose, they can also submit those drawings to help improve the AI model," Meta said.

The first step in animating children's drawings of people is distinguishing the human figures from the background and from other types of characters in the picture.

Object detection using existing techniques works quite well on children's drawings, but the segmentation masks aren't accurate enough to be used for animation.

"To address this, we instead use the bounding boxes obtained from the object detector and apply a series of morphological operations and image processing steps to obtain masks," the researchers explained.

They use Meta AI's convolutional neural network-based object detection model, 'Mask R-CNN', to extract the human-like characters within a child's drawing for processing.

'Mask R-CNN' is pre-trained on one of the largest publicly available segmentation data sets, but it's made up of photos of real-world objects, not drawings.

"We invited our colleagues at Meta to share and animate their kids' artwork using our system, and we obtained approximately 1,000 drawings that helped us train the AI," the researchers informed.

To animate the 2D figures using 3D motion capture, the researchers took advantage of the fact that many children draw using what is referred to as a twisted perspective.

"We take advantage of this perspective in our motion retargeting step. Independently for the lower and upper body, we automatically determine whether the motion is more recognisable from a front view or a side view," the Meta AI team said.

Someday, an AI system could take a complex drawing and then instantly create a detailed animated cartoon using multiple fantastical characters interacting with one another and elements from the background.

"With AR glasses, those stories could even seem to come to life in the real world, dancing or talking with the child who drew it just moments earlier. The possibilities are as limitless as the human imagination," said the researchers.