Visual Layout-to-Image Generation (VL2I). Text-to-Image models show remarkable ability in generating image from text prompt~\textcite{ramesh2022hierarchical, robalin2022photorealistic, Rombach_2022_CVPR, sahariah2022photorealistic, podsell, podinpaint}. However, controlling these models to generate images that follow the spatial and layout information specified by the user remains a challenging task. To address this, layout-to-image methods have been developed, with layout conditions, such as bounding boxes with category labels, effectively controlling the spatial location of the objects during the generation process. Most layout-to-image models inject the layout information into the cross-attention or self-attention layers in the U-Net. Layout-Diffusion~\textcite{zheng2023layoutdiffusion} constructs a layout-fusion module, which introduces-object-aware attention to model the Relationship between layout and image. Layout-Guidance~\textcite{chen2024layout} proposes a training-free technique to manipulate the self-attention maps with the layout, increasing the attention value of the region-object pair. Reconstruct-and-Generate~\textcite{lian2023LLMgrounded} proposes an LLM-grounded layout-to-image generation method, where the LLM is used to generate the layout from the text prompt, and the layout-to-image model is used to generate the image from the layout. Similar to LLM-grounded layout-to-image, Control-Net~\textcite{zhang2023adding} and T2I-Adapter~\textcite{mou2024t2i} add a trainable copy of the encoding layers of the U-Net to process the condition inputs (e.g., bounding boxes, sketches, depth, etc.) and add the output to the skip connections of the U-Net, which enables_text-to-image models to support layout-to-image.
Visual Text Generation. Generating images with text, which is also called visual text generation, is an important and challenging task. Early works, such as TextDiffuser~\textcite{chen2023textdiffuser}, use a segment-anything model (SAM) to segment the text in the image and train a layout-to-image model, then train a text-to-image model with the layout-to-image model to generate the text. GlyphDraw~\textcite{ma2023glyphdraw} and AnyText~\textcite{tuo2023anytext} render the text on a canvas with a specific font, and then use a text-to-image model to generate the image based on the rendered text. TextDiffuser-2~\textcite{chen2023textdiffuser2} uses a language model to predict the bounding boxes of the text and then uses a layout-to-image model to generate the text. These methods have shown great performance, however, the text layout is determined by either the user-specified bounding boxes or the LLM-predicted layout, which lacks the flexibility to adapt to the image and may lead to unnatural text placement and degradation of image quality. Our method, on the other hand, considers both the image layout and the text layout, and generates the image layout before the text layout, which allows the text to be placed in a more natural way.
To evaluate the efficacy of the proposed model, we compare our model with other SOTA visual text generation models, including TextDiffuser-2~\textcite{chen2023textdiffuser2}, AnyText~\textcite{tuo2023anytext} and GlyphControl~\textcite{yang2023glyphcontrol}. The details of the implementation, evaluation metrics, and comparison with SOTA models are as follows.
Implementation Details. Our pipeline contains three main components: layout-from-text model, text layout generator, and image generator. The training and inference details are as follows: