SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing

Authors: Jing Gu, Yilin Wang, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Xin Eric Wang

What

This paper introduces \modelname{}, a novel framework for personalized object swapping in images using pre-trained diffusion models, enabling precise replacement of arbitrary objects with personalized concepts while preserving the background context.

Why

This paper is important as it addresses limitations in existing personalized image editing techniques, enabling precise and localized swapping of arbitrary objects while maintaining stylistic consistency and preserving background context, with potential applications in e-commerce, entertainment, and professional editing.

How

The authors propose a \modelname{} framework that leverages pre-trained diffusion models. They introduce ‘targeted variable swapping’ for precise object replacement and ‘appearance adaptation’ to seamlessly integrate the new object into the source image’s style, scale, and content, ensuring a cohesive visual result.

Result

\modelname{} demonstrates superior performance in personalized object swapping tasks, including single-object, multi-object, partial-object, and cross-domain swapping, as evidenced by human and automatic evaluations. It outperforms baselines in preserving background context, accurately swapping object identities, and maintaining overall image quality. Furthermore, \modelname{} exhibits promising results in text-based swapping and object insertion tasks.

LF

The authors acknowledge limitations in reconstructing intricate details within the masked area and handling objects with high degrees of freedom. Future work will focus on addressing these limitations by incorporating explicit alignment mechanisms and extending the framework to 3D/video object swapping.

Abstract

Effective editing of personal content holds a pivotal role in enabling individuals to express their creativity, weaving captivating narratives within their visual stories, and elevate the overall quality and impact of their visual content. Therefore, in this work, we introduce SwapAnything, a novel framework that can swap any objects in an image with personalized concepts given by the reference, while keeping the context unchanged. Compared with existing methods for personalized subject swapping, SwapAnything has three unique advantages: (1) precise control of arbitrary objects and parts rather than the main subject, (2) more faithful preservation of context pixels, (3) better adaptation of the personalized concept to the image. First, we propose targeted variable swapping to apply region control over latent feature maps and swap masked variables for faithful context preservation and initial semantic concept swapping. Then, we introduce appearance adaptation, to seamlessly adapt the semantic concept into the original image in terms of target location, shape, style, and content during the image generation process. Extensive results on both human and automatic evaluation demonstrate significant improvements of our approach over baseline methods on personalized swapping. Furthermore, SwapAnything shows its precise and faithful swapping abilities across single object, multiple objects, partial object, and cross-domain swapping tasks. SwapAnything also achieves great performance on text-based swapping and tasks beyond swapping such as object insertion.