image: This is a schematic presentation of structure-aware image stylization using gradient optimization. view more
Credit: ©Science China Press
Abstraction and stylization algorithms are designed to deal with digital image water colorization, oil painting, cartoon style generation, and so on. Dr Li Ping and his group from the Department of Computer Science and Engineering at the Chinese University of Hong Kong, together with researchers from Shanghai Jiao Tong University and the Beijing Institute of Technology once dreamt of solving this problem. After four years of creative media research, they have devised a structure-aware image stylization method to generate the effects of artistic drawing and painting using only single digital images as input, and at the same time, have applied hardware GPU parallelism to enable real-time non-photorealistic rendering (NPR) for more efficient processing. They also proposed an image structure map to naturally model the fine structure details present in the original images to preserve the image structure well between the original and stylized images. Their work, entitled "Image stylization with enhanced structure on GPU", has been published in Science China Information Sciences, 2012, Vol. 55(5).
Before the invention of photographic apparatus, painting was one of the most important ways of recording scenery and life. After hundreds of years of study, artists found that by making imagery look less photorealistic, with different stylization as the vehicle for abstraction, creativity, and expressiveness, they were able to enhance the audiences' immersive feelings for the original story in real life on canvas. Recent advances in painterly rendering have demonstrated that certain artistic styles can be generated automatically using methods of abstraction and painterly stylization. However, the detailed salience structure of the original images is somehow always destroyed when performing current image abstraction and stylization methods due to limitations like unavoidable salience information loss caused by contrast abstraction manipulation, which may greatly influence users' understanding of visual art and illustration. There is one artistic theory, realism, according to which, realist painters portray what they see without idealizing it in their paintings. Therefore, introducing new approaches to deal with image stylization while preserving the fine structure of the original images is both very important and a necessity.
User studies on recognition speed and short term memory retention were carried out to verify whether image stylization preserves or distills perceptually important information. A total of 30 students (10 from an art school, and 20 from non-art-related backgrounds) who knew nothing about the research participated in the studies. Stylized images together with images of similar topics were presented for evaluating recognition speed and short term retention after recognition. Stylized image attractiveness was also tested by printing and distributing a total of 50 copies of stylized images randomly on the street, and asking people to score the attractiveness of the images. A total of 46 copies of the printed material were returned for evaluation. Generally, structure-aware stylization requires less time for picture recognition, and can allow people to remember the scenes a few days longer for short period memory. The randomly sampled user study carried out on the street also shows that stylization exhibits more attractiveness to the people viewing the test images. All the statistics are based on the 30 students' average performances and the average scores of the 46 image copies returned. Moreover, in terms of the time performance of stylization processing, the authors' approach executes very efficiently in real-time for large resolution images on a GPU with parallel acceleration.
Different stylization methods have been developed for automatic generation of certain artistic styles. In general, two basic forms exist, the abstraction approach and painterly stylization. The former achieves abstraction by reducing contrast in low-contrast regions and increasing contrast in high-contrast regions, whereas the latter applies tools like brush strokes to simulate painterly artistic styles. To some extent, these methods provide solutions to computer-based image/video stylization and abstraction. However, when performing stylization, the original image structure is always destroyed because of limitations like unavoidable salience information loss caused by abstraction. The information lost is actually essential for better understanding of the stylized image material, e.g., illustrations of mechanical parts in educational books, where the detailed structure of objects is a key issue even if the images are stylized. Thus, producing abstracted pictures while destroying the detailed fine image structure is never a good choice.
According to one artistic theory, realism, realist painters aim to produce artwork that is undistorted by human bias. Hence, truth and accuracy in real life of the structure of the objects to be painted is the core of realism, and these artists use a brush or pencil to render the scenes in real life realistically by taking into consideration the detailed structure of the targeted objects. The method, therefore, inherently implies a belief that such quality and characteristics are ontologically independent of human conceptual feeling, ideas, and recreation, and thus can be known by artists, who can in turn represent such reality faithfully using a brush or pencil on canvas. Moreover, such realistic artwork, in which realism represents the world realistically not only with the finely maintained structure of objects in the painting, but also with a certain range of aesthetic abstraction and artistic stylization, earns its popularity through its differences compared with photography, which gives the true perspective. This great quality of realism is realistically maintained and offered by the approach of real-time structure-aware image stylization using GPU parallelism.
The novel GPU-based structure-aware image stylization preserves the fine structure between the original and stylized images well, avoiding salience information loss caused by contrast abstraction. It applies an image structure map to naturally model the detailed image structure present in the original images. Gradient-based structure tangent generation and tangent-guided image morphology are utilized to construct the structure map. The image structure map, unlike an edge map, not only systematically models the boundary information within the imagery, but also accentuates the underlying inner structure detail for further stylization. The authors facilitate the final stylization via a parallel bilateral grid and structure-aware stylizing optimization on a GPU platform in real-time. The figure below shows the process flow for real-time high-quality image stylization.
Promotion and implementation of the new stylization benefits the cartoon, animation, and creative media industry most, in that the work can greatly assist users in artistic painting generation, not only for professional artists but also amateurs. The efficiency and effectiveness of the work will be essential for the improvement of painting art and cartoon animation production.
The design and development of the structure-aware picture stylization is the result of the joint research effort of many artists, scientists, and academics from various institutes and universities. The research project was partially supported by an RGC research grant, UGC direct grants for research, a grant from the National Natural Science Foundation of China, and a grant from the Key Program of NSFC-Guangdong Union Foundation. It represents an essential and critical breakthrough in recent development of image stylization. The researchers hope that their research project can be developed and applied widely in media art production and can assist many professional and amateur artists in generating artwork. This work will have significant impact on the creation of cartoon animation and the development of the creative media industry.
Differing from traditional abstraction and stylization methods, which focus mostly on contrast manipulation and destroy detailed salience information of the original images, the proposed structure-aware stylization uses an image structure map to naturally model the underlying fine structure present in the original images. An ellipse-shaped erosion method is applied to extract the structure map using local energy minimization based on gradient magnitude and the image structure directions. An intermediate stylization is produced using a recursive edge-aware bilateral grid in real-time on a GPU platform. Global optimization is finally carried out to facilitate the structure-aware image stylization based on the image structure map and initial stylization. In practice, the new stylization demonstrates great efficiency in picture understanding and simple retention of interesting stylized images.
See the article: Li P, Sun H Q, Sheng B, et al. Image stylization with enhanced structure on GPU. Sci China Inf Sci, 2012, 55(5): 1093