Quality Diversity through Human Feedback

Authors: Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, Joel Lehman

What

This paper introduces Quality Diversity through Human Feedback (QDHF), a novel approach that integrates human feedback into Quality Diversity (QD) algorithms to automatically learn diversity metrics for optimizing the generation of diverse and high-quality solutions.

Why

This paper is important because it addresses the limitations of existing QD algorithms that rely on manually crafted diversity metrics, which restricts their applicability in complex and open-ended tasks where defining such metrics is challenging. QDHF offers a more flexible and adaptable approach by leveraging human feedback to learn diversity metrics, potentially leading to improved exploration and diversity in various domains.

How

The authors propose an implementation of QDHF using latent space projection and contrastive learning. They first train a latent projection model to map solutions into a latent space, where each dimension represents a learned diversity metric. Then, they use human judgments on the similarity of solutions to fine-tune the latent projection model via contrastive learning, ensuring the learned diversity metrics align with human perception. They evaluate QDHF on three benchmark tasks: robotic arm control, maze navigation, and latent space illumination for image generation, comparing it against existing QD algorithms with unsupervised diversity discovery and ground truth metrics.

Result

Experimental results demonstrate that QDHF significantly outperforms unsupervised diversity discovery methods in QD, achieving both higher quality and diversity in the generated solutions. Notably, in the latent space illumination task, QDHF successfully generates more diverse images while maintaining high quality compared to baseline methods. User studies further confirm that QDHF-generated images are perceived as more diverse and preferred by humans.

LF

The authors acknowledge that the performance of QDHF relies on the accuracy of the learned latent projection model and the quality of human feedback. They suggest future work focusing on improving the generalization of the preference model used to collect human feedback, exploring strategies for efficient and diverse data collection, and applying QDHF to more complex and open-ended tasks in robotics, reinforcement learning, and generative modeling.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has shown potential in qualitative tasks where clear objectives are lacking. However, its effectiveness is not fully realized when it is conceptualized merely as a tool to optimize average human preferences, especially in generative tasks that demand diverse model responses. Meanwhile, Quality Diversity (QD) algorithms excel at identifying diverse and high-quality solutions but often rely on manually crafted diversity metrics. This paper introduces Quality Diversity through Human Feedback (QDHF), a novel approach integrating human feedback into the QD framework. QDHF infers diversity metrics from human judgments of similarity among solutions, thereby enhancing the applicability and effectiveness of QD algorithms. Our empirical studies show that QDHF significantly outperforms state-of-the-art methods in automatic diversity discovery and matches the efficacy of using manually crafted metrics for QD on standard benchmarks in robotics and reinforcement learning. Notably, in a latent space illumination task, QDHF substantially enhances the diversity in images generated by a diffusion model and was more favorably received in user studies. We conclude by analyzing QDHF’s scalability and the quality of its derived diversity metrics, emphasizing its potential to improve exploration and diversity in complex, open-ended optimization tasks. Source code is available on GitHub: https://github.com/ld-ing/qdhf.