Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

Authors: Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, Seungryong Kim

What

This paper introduces Perturbed-Attention Guidance (PAG), a novel sampling guidance method for diffusion models that enhances sample quality by perturbing self-attention maps during the denoising process, eliminating the need for additional training or external modules.

Why

The paper addresses the limitations of existing guidance techniques like Classifier Guidance (CG) and Classifier-Free Guidance (CFG), which often lack applicability in unconditional generation or specific downstream tasks. PAG offers a more versatile approach, enhancing sample quality in both conditional and unconditional scenarios without requiring extra training or external components.

How

PAG leverages the observation that self-attention maps in diffusion U-Nets capture structural information. The method perturbs these maps by replacing them with identity matrices, creating intermediate samples with degraded structures. This ‘undesirable’ path guides the denoising process towards generating samples with superior structural coherence and realism.

Result

PAG significantly improves sample quality in both ADM and Stable Diffusion, evident in enhanced FID and IS scores, particularly in unconditional generation where CFG is inapplicable. PAG also complements CFG, leading to further quality improvements when used in conjunction. The method’s efficacy extends to downstream tasks like image restoration (PSLD) and spatially conditioned generation (ControlNet), demonstrating its versatility.

LF

The authors acknowledge limitations such as potential over-saturation at high guidance scales and the computational overhead of two forward passes per generation step. Future work could focus on mitigating these limitations by exploring techniques for efficient guidance computation and hyperparameter optimization.

Abstract

Recent studies have demonstrated that diffusion models are capable of generating high-quality samples, but their quality heavily depends on sampling guidance techniques, such as classifier guidance (CG) and classifier-free guidance (CFG). These techniques are often not applicable in unconditional generation or in various downstream tasks such as image restoration. In this paper, we propose a novel sampling guidance, called Perturbed-Attention Guidance (PAG), which improves diffusion sample quality across both unconditional and conditional settings, achieving this without requiring additional training or the integration of external modules. PAG is designed to progressively enhance the structure of samples throughout the denoising process. It involves generating intermediate samples with degraded structure by substituting selected self-attention maps in diffusion U-Net with an identity matrix, by considering the self-attention mechanisms’ ability to capture structural information, and guiding the denoising process away from these degraded samples. In both ADM and Stable Diffusion, PAG surprisingly improves sample quality in conditional and even unconditional scenarios. Moreover, PAG significantly improves the baseline performance in various downstream tasks where existing guidances such as CG or CFG cannot be fully utilized, including ControlNet with empty prompts and image restoration such as inpainting and deblurring.