Ye Hu ; Zhang Jun ; Liu Sibo ; Han Xiao ; Yang Wei

Abstract

Recent years have witnessed the strong power of large text-to-image diffusionmodels for the impressive generative capability to create high-fidelity images.However, it is very tricky to generate desired images using only text prompt asit often involves complex prompt engineering. An alternative to text prompt isimage prompt, as the saying goes: "an image is worth a thousand words".Although existing methods of direct fine-tuning from pretrained models areeffective, they require large computing resources and are not compatible withother base models, text prompt, and structural controls. In this paper, wepresent IP-Adapter, an effective and lightweight adapter to achieve imageprompt capability for the pretrained text-to-image diffusion models. The keydesign of our IP-Adapter is decoupled cross-attention mechanism that separatescross-attention layers for text features and image features. Despite thesimplicity of our method, an IP-Adapter with only 22M parameters can achievecomparable or even better performance to a fully fine-tuned image prompt model.As we freeze the pretrained diffusion model, the proposed IP-Adapter can begeneralized not only to other custom models fine-tuned from the same basemodel, but also to controllable generation using existing controllable tools.With the benefit of the decoupled cross-attention strategy, the image promptcan also work well with the text prompt to achieve multimodal image generation.The project page is available at \url{https://ip-adapter.github.io}.