Toward effective protection against diffusion based mimicry through score distillation

Authors: Haotian Xue, Chumeng Liang, Xiaoyu Wu, Yongxin Chen

What

This paper investigates the vulnerability of Latent Diffusion Models (LDMs) to adversarial attacks, particularly in the context of protecting images from unauthorized mimicry.

Why

The paper is important because it addresses the growing concern of malicious use of LDMs for creating unauthorized digital replicas, and it proposes more efficient and effective methods for protecting images from such misuse.

How

The authors analyze the bottleneck in attacking LDMs, revealing the encoder as the vulnerable component. They introduce Score Distillation Sampling (SDS) to accelerate protection, explore the effectiveness of minimizing semantic loss, and conduct extensive experiments on various mimicry scenarios (SDEdit, inpainting, textual inversion) to evaluate their proposed strategies.

Result

Key findings include: (1) The encoder of an LDM is significantly more vulnerable to attacks than the denoiser module. (2) Minimizing semantic loss can be an effective protection strategy, producing more natural perturbations compared to maximizing it. (3) SDS accelerates protection by 50% without sacrificing effectiveness. (4) The proposed strategies outperform existing methods in terms of protection strength, perturbation naturalness, and computational efficiency.

LF

The paper mainly focuses on LDMs and future work could explore attacks on pixel-based diffusion models. Additionally, investigating the robustness of the proposed protections against various defense methods is crucial for real-world deployment.

Abstract

While generative diffusion models excel in producing high-quality images, they can also be misused to mimic authorized images, posing a significant threat to AI systems. Efforts have been made to add calibrated perturbations to protect images from diffusion-based mimicry pipelines. However, most of the existing methods are too ineffective and even impractical to be used by individual users due to their high computation and memory requirements. In this work, we present novel findings on attacking latent diffusion models (LDM) and propose new plug-and-play strategies for more effective protection. In particular, we explore the bottleneck in attacking an LDM, discovering that the encoder module rather than the denoiser module is the vulnerable point. Based on this insight, we present our strategy using Score Distillation Sampling (SDS) to double the speed of protection and reduce memory occupation by half without compromising its strength. Additionally, we provide a robust protection strategy by counterintuitively minimizing the semantic loss, which can assist in generating more natural perturbations. Finally, we conduct extensive experiments to substantiate our findings and comprehensively evaluate our newly proposed strategies. We hope our insights and protective measures can contribute to better defense against malicious diffusion-based mimicry, advancing the development of secure AI systems. The code is available in https://github.com/xavihart/Diff-Protect