image: A team of computer scientists from City University of Hong Kong and Microsoft, have developed an innovative deep learning-based approach to automatically generate the caricature of a given portrait, and to enable users to do so efficiently and realistically. view more
Credit: Image courtesy of Kaidi Cao
Caricature portrait drawing is a distinct art form where artists sketch a person's face in an exaggerated manner, most times to elicit humor. Automating this technique poses challenges due to the amount of intricate details and shapes involved and level of professional skills it takes to transform a person artistically from their real-life selves to a creatively exaggerated one.
A team of computer scientists from City University of Hong Kong and Microsoft, have developed an innovative deep learning-based approach to automatically generate the caricature of a given portrait, and to enable users to do so efficiently and realistically.
"Compared to traditional graphics-based methods which define hand-crafted rules, our novel approach leverages big data and machine learning to synthesize caricatures from thousands of examples drawn by professional artists," says Kaidi Cao, lead author, who is currently a graduate student in computer science at Stanford University but conducted the work during his internship at Microsoft. "While existing style transfer methods have focused mainly on appearance style, our technique achieves both geometric exaggeration and appearance stylization involved in caricature drawing." The method enables users to automate caricatures of portraits, and can be applied to tasks such as creating caricatured avatars for social media, and designing cartoon characters. The technique also has potential applications in marketing, advertising and journalism.
Cao collaborated on the research with Jing Liao of City University of Hong Kong and Lu Yuan of Microsoft, and the three plans to present their work at SIGGRAPH Asia 2018 in Tokyo from 4 December to 7 December. The annual conference features the most respected technical and creative members in the field of computer graphics and interactive techniques, and showcases leading edge research in science, art, gaming and animation, among other sectors.
In this work, the researchers turned to a well-known technique in machine learning, Generative Adversarial Network (GAN), for unpaired photo-to-caricature translation to generate caricatures that preserve the identity of the portrait. Called "CariGANs", the computational framework precisely models geometric exaggeration in photos (shapes of faces, specific angles) and appearance stylization (look, feel, pencil strokes, shadowing) via two algorithms the researchers have labeled, CariGeoGAN and CariStyGAN.
CariGeoGAN only models the geometry-to-geometry mapping from face photos to caricatures and CariStyGAN transfers the style appearance from caricatures to face photos without any deformation to the geometry of the original image. The two networks are separately trained for each task so that the learning procedure is more robust, notes the researchers. The CariGANs framework enables users to control the exaggeration degree in geometric and appearance style by dragging slides or giving an example caricature.
Cao and collaborators conducted perceptual studies to evaluate their framework's ability to generate caricatures of portraits that are easily recognizable and not overly distorted in shape and appearance style. For example, one study assessed how well the identity of an image is preserved using the CariGANs method in comparison to existing methods for translating caricature art. They demonstrated, through several examples, that existing methods resulted in unrecognizable caricature translation. Study participants found it too difficult to match the resulting caricatures with the original subjects because the end results were far too exaggerated or unclear. The researchers' method successfully generated clearer, more accurate caricature depictions of portrait photos, as if they were hand drawn by a professional artist.
Currently, the focus of this work has centered on caricatures of people, primarily headshots or portraits. In future work, the researchers intend to explore beyond facial caricature generation into full body or more complex scenes. They are also interested in designing improved human-computer interaction (HCI) systems that would give users more freedom and user control on the machine learning-generated results.
###
About SIGGRAPH Asia 2018
The 11th ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia 2018) will be held in Tokyo, Japan at the Tokyo International Forum from 4 to 7 December 2018. The annual event held in Asia attracts the most respected technical and creative people from all over the world who are excited by research, science, art, animation, gaming, interactivity, education and emerging technologies.
The four-day conference will include a diverse range of juried programs, such as the Art Gallery, Computer Animation Festival, Courses, Emerging Technologies, Posters, Technical Briefs, Technical Papers and Virtual & Augmented Reality. A three-day exhibition held from 5 to 7 December 2018 will offer a business platform for industry players to market their innovative products and services to the computer graphics and interactive techniques professionals and enthusiasts from Asia and beyond. For more information, please visit http://sa2018.siggraph.org. Find us on: Facebook, Twitter, Instagram and YouTube with the official event hashtag, #SIGGRAPHAsia.
About ACM SIGGRAPH
The ACM Special Interest Group on Computer Graphics and Interactive Techniques is an interdisciplinary community interested in research, technology, and applications in computer graphics and interactive techniques. Members include researchers, developers, and users from the technical, academic, business, and art communities. ACM SIGGRAPH enriches the computer graphics and interactive techniques community year-round through its conferences, global network of professional and student chapters, publications, and educational activities. For more information, please visit http://www.siggraph.org.
About Koelnmesse
Koelnmesse Pte Ltd is one of the world's largest trade fair companies. Its more than 80 trade fairs and exhibitions have the broadest international scope in the industry, as 60 percent of the exhibitors and 40 percent of the visitors come from outside Germany. The Koelnmesse events include leading global trade fairs for 25 sectors, such as Imm Cologne, Anuga, IDS, INTERMOT, Interzum Cologne, Photokina, Gamescom, and the International Hardware Fair Cologne. Koelnmesse is ACM SIGGRAPH's event organizer for the last 10 editions of SIGGRAPH Asia. For more information, please visit http://www.koelnmesse.com.sg.