Alibaba's engineers brought ancient paintings to life to mark the Mid-Autumn Festival, one of China's most important traditional holidays
Alibaba Group’s research arm DAMO Academy has used artificial intelligence to make ancient Chinese portraits blink, smile and speak as part of a push to bring cultural classics to life with digital technology.
Last week, DAMO Academy worked with a public library in Hangzhou, the company’s home city, to animate about a dozen paintings from books in the library’s archives – including “The Night Revels of Han Xizai” by artist Gu Hongzhong and illustrations in “Dream of the Red Chamber” by Cao Xueqin.
“With AI technology, we can see the historical figures’ facial expressions change as they recite poetry,” said Liang Liang, deputy director for the Hangzhou Public Library, commenting on the collaboration, which the library co-initiated to celebrate this year’s Mid-Autumn Festival in China. “We also invited [Hangzhou] residents to participate and record voiceovers for them, making this a collaborative cultural event bringing ancient texts and paintings back to life.”
DAMO Academy’s Vision Lab, the same team behind Alibaba’s virtual avatar technology and image-search tool Pailitao, spearheaded the project. The lab’s engineers developed an image-analysis algorithm that accurately detects facial features and key points on the historical figures in the paintings.
They also used videos of Hangzhou residents who recorded voiceovers for the project, modeling the changes in their movements. DAMO then used its AI models to process this trove of information to transform the figures in still paintings into realistic, moving characters.
Tang Mingqian, an AI engineer with the DAMO Academy, said the biggest challenge was ensuring their characters truly captured the spirit of the original artwork.
To do this, the team sifted through hundreds of thousands of traditional portraits to examine faces from different angles. This helped make the portraits’ head movements more multidimensional and lifelike. Additionally, they refined the image quality and developed an edge-aware attention network module which smoothed out the movements and helped avoid jerky movements.
In May, DAMO Academy also launched a project to digitize Chinese classics with UC Berkeley Library, Sichuan University, the National Library of China and Zhejiang Library. The project seeks to convert scanned images from ancient books into texts for readers online, including using AI and optical character recognition technology to help decipher old Chinese characters – which can be complex, with different variants and written forms.
The company plans to continue to leverage AI to preserve cultural heritage and classic texts.
“This project explores the possibilities of connecting technology with heritage,” said Alibaba’s Tang Mingqian on their latest initiative to animate traditional artwork. “AI can breathe life into ancient books and paintings. It is a new way to present precious historic data.”
To receive the latest news directly in your inbox, sign up for the weekly Alizila newsletter
