Generative Escher Meshes

Authors: Noam Aigerman, Thibault Groueix

What

This paper presents a novel method for generating tileable, non-square 2D art, similar to the works of M.C. Escher, by combining mesh deformation, texture optimization, and text-guided diffusion models.

Why

The ability to automatically generate appealing and complex tiling patterns has significant implications for various fields, including art, design, and architecture, while also offering a new approach to exploring the space of tileable shapes.

How

The authors represent the tile as a textured 2D mesh and leverage Orbifold Tutte Embeddings (OTE) to ensure tileability while optimizing mesh vertices. They use a differentiable renderer to generate an image of the tile and apply Score Distillation Sampling (SDS) with a pre-trained diffusion model to guide the optimization towards matching a user-provided text prompt.

Result

The method successfully produces a wide variety of compelling tileable shapes with different symmetries, demonstrating its ability to generate complex and plausible imagery from text prompts while adhering to strict geometric constraints.

LF

Limitations include the restriction to wallpaper group tilings, difficulty in generating complex multi-object scenes, and reliance on SDS, which has limitations in speed, color saturation, and controllability. Future work could explore extensions to aperiodic tilings, multi-object tile generation, and integration with more advanced text-guided image generation techniques.

Abstract

This paper proposes a fully-automatic, text-guided generative method for producing periodic, repeating, tile-able 2D art, such as the one seen on floors, mosaics, ceramics, and the work of M.C. Escher. In contrast to the standard concept of a seamless texture, i.e., square images that are seamless when tiled, our method generates non-square tilings which comprise solely of repeating copies of the same object. It achieves this by optimizing both geometry and color of a 2D mesh, in order to generate a non-square tile in the shape and appearance of the desired object, with close to no additional background details. We enable geometric optimization of tilings by our key technical contribution: an unconstrained, differentiable parameterization of the space of all possible tileable shapes for a given symmetry group. Namely, we prove that modifying the laplacian used in a 2D mesh-mapping technique - Orbifold Tutte Embedding - can achieve all possible tiling configurations for a chosen planar symmetry group. We thus consider both the mesh’s tile-shape and its texture as optimizable parameters, rendering the textured mesh via a differentiable renderer. We leverage a trained image diffusion model to define a loss on the resulting image, thereby updating the mesh’s parameters based on its appearance matching the text prompt. We show our method is able to produce plausible, appealing results, with non-trivial tiles, for a variety of different periodic tiling patterns.