Abstract

Face photo-sketch synthesis aims at generating a facial sketch/photoconditioned on a given photo/sketch. It is of wide applications includingdigital entertainment and law enforcement. Precisely depicting facephotos/sketches remains challenging due to the restrictions on structuralrealism and textural consistency. While existing methods achieve compellingresults, they mostly yield blurred effects and great deformation over variousfacial components, leading to the unrealistic feeling of synthesized images. Totackle this challenge, in this work, we propose to use the facial compositioninformation to help the synthesis of face sketch/photo. Specially, we propose anovel composition-aided generative adversarial network (CA-GAN) for facephoto-sketch synthesis. In CA-GAN, we utilize paired inputs including a facephoto/sketch and the corresponding pixel-wise face labels for generating asketch/photo. In addition, to focus training on hard-generated components anddelicate facial structures, we propose a compositional reconstruction loss.Finally, we use stacked CA-GANs (SCA-GAN) to further rectify defects and addcompelling details. Experimental results show that our method is capable ofgenerating both visually comfortable and identity-preserving facesketches/photos over a wide range of challenging data. Our method achieves thestate-of-the-art quality, reducing best previous Frechet Inception distance(FID) by a large margin. Besides, we demonstrate that the proposed method is ofconsiderable generalization ability. We have made our code and results publiclyavailable: https://fei-hdu.github.io/ca-gan/.