Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation

Authors: Lanqing Guo, Yingqing He, Haoxin Chen, Menghan Xia, Xiaodong Cun, Yufei Wang, Siyu Huang, Yong Zhang, Xintao Wang, Qifeng Chen, Ying Shan, Bihan Wen

What

This paper introduces a novel self-cascade diffusion model that leverages a pre-trained low-resolution model to efficiently adapt to higher-resolution image and video generation tasks.

Why

This paper addresses the challenge of computationally expensive fine-tuning required to adapt pre-trained diffusion models for higher-resolution generation. It proposes an efficient method that achieves significant training speed-up while maintaining generation quality, enabling wider application of diffusion models in high-resolution settings.

How

The authors propose two versions of their self-cascade diffusion model: a tuning-free version that utilizes a pivot-guided noise re-scheduling strategy to leverage the low-resolution model’s knowledge, and a tuning version that incorporates learnable time-aware feature upsampler modules for improved detail with minimal fine-tuning on a small high-resolution dataset. They evaluate their method on both image and video generation tasks, comparing it to full fine-tuning and other adaptation techniques.

Result

The self-cascade diffusion model demonstrates significant training speed-up (5x) compared to full fine-tuning, requiring minimal additional trainable parameters (0.002M) and negligible extra inference time. Experiments on image and video generation tasks show that it achieves state-of-the-art performance in both tuning-free and tuning settings, effectively adapting to higher resolutions while preserving the original model’s generation capabilities and outperforming competing methods in terms of quality and efficiency.

LF

The authors acknowledge that the limited capacity of the lightweight upsampler modules may pose limitations, especially for very large scale gaps. Future work may involve exploring the trade-off between adaptation efficiency and generalization ability, potentially by incorporating more sophisticated upsampling mechanisms or investigating alternative methods for knowledge transfer from the low-resolution model.

Abstract

Diffusion models have proven to be highly effective in image and video generation; however, they still face composition challenges when generating images of varying sizes due to single-scale training data. Adapting large pre-trained diffusion models for higher resolution demands substantial computational and optimization resources, yet achieving a generation capability comparable to low-resolution models remains elusive. This paper proposes a novel self-cascade diffusion model that leverages the rich knowledge gained from a well-trained low-resolution model for rapid adaptation to higher-resolution image and video generation, employing either tuning-free or cheap upsampler tuning paradigms. Integrating a sequence of multi-scale upsampler modules, the self-cascade diffusion model can efficiently adapt to a higher resolution, preserving the original composition and generation capabilities. We further propose a pivot-guided noise re-schedule strategy to speed up the inference process and improve local structural details. Compared to full fine-tuning, our approach achieves a 5X training speed-up and requires only an additional 0.002M tuning parameters. Extensive experiments demonstrate that our approach can quickly adapt to higher resolution image and video synthesis by fine-tuning for just 10k steps, with virtually no additional inference time.