Share E-Book

Build a Text-to-Image Generator (from Scratch) With transformers and diffusions (Mark Liu)(Z-Library)

Author

Rating No ratings yet

Log in to rate

Technology
Language English

This book takes you step-by-step through creating your own AI models that can generate images from text. You’ll explore two methods of image generation—vision transformers and diffusion models—and learn vital AI development techniques as you go. Dive into the powerful models behind AI image generators. The best way to learn is to build something from scratch, and in this book you’ll build your very own diffusion model and vision transformer. As you work through each stage of development, you’ll develop an understanding of how these models can be customized, applied, and integrated for impressive multimodal AI. Build a Text-to-Image Generator (from Scratch) teaches you how to • Build and train models to generate high resolution images based on text descriptions • Edit an existing image based on text prompts • Build and train a model to add captions to images • Build and train a vision transformer to classify images • Fine-tune LLMs for downstream tasks such as classification, text or image generation • Better differentiate real images from deepfakes About the technology AI-generated images appear everywhere from high-end advertising to casual social media feeds. Text-to-image tools like Dall-e, Midjourney, and Flux make it easy to create AI art, but how do they work? In this book, you’ll find out by building your own text-to-image generator! About the book Build a Text-to-Image Generator (from Scratch) explores both transformer-based image generation and diffusion models. You’ll work hands-on to build a pair of simple generation models that can classify images, automatically add captions, reconstruct images, and enhance existing graphics. Author Mark Liu guides you every step of the way with clear explanations, informative diagrams, and eye-opening examples you can build on your own laptop. What's inside • Build a vision transformer to classify images • Edit images using text prompts • Fine-tune image models

Format PDF
Size 19.9 MB
9
Views
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
M A N N I N G Mark Liu FROMSCRATCH with transformers and diffusions BUILD A
Page 2
Text prompt: e.g., Banana riding a motorcyle, with sunglasses and a straw hat CLIP Text Encoder Text embedding Conditional Latent U-Net Latent seed: Random noise Latent image (4 x 64 x 64) Denoised latent image (4 x 64 x 64) Repeat N inference steps VAE Decoder Final output 3 x 512 x 512 The text prompt shown at the top-left corner is first encoded into a text embedding. To generate an image in the lower-dimensional space (latent space), we start with an image of pure noise (bottom left). We use the trained U-Net to iteratively denoise the image, conditional on the text embedding so the generated image matches the text prompt, with the guidance of a trained CLIP model. The generated latent image (bottom right) is presented to a trained VAE to convert it to a high-resolution image, which is the final output (top right). How a latent diffusion model (e.g., Stable Diffusion) generates an image based on a text prompt Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 3
Build a Text-to-Image Generator (from Scratch) Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 4
Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 5
MANN I NG Shelter ISland Mark Liu Build a Text-to-Image Generator (from Scratch) With transformers and diffusions Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 6
For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com © 2026 Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid- free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. ∞ Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 ISBN 9781633435421 Printed in the United States of America The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Development editor: Doug Rudder Technical editor: Nathan Crocker Review editor: Kishor Rit Production editor: Kathy Rossland Copy editor: Julie McNamee Proofreader: Melody Dolab Typesetter: Tamara ŠveliÊ SabljiÊ Cover designer: Marija Tudor Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 7
To all AI enthusiasts! Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 8
vi brief contents Part 1 Understanding attention and transformers .........1 1 ■ A tale of two models: Transformers and diffusions 3 2 ■ Build a transformer 22 3 ■ Classify images with a vision transformer 52 4 ■ Add captions to images 79 Part 2 Introduction to diffusion models ........................ 103 5 ■ Generate images with diffusion models 105 6 ■ Control what images to generate in diffusion models 126 7 ■ Generate high-resolution images with diffusion models 152 Part 3 Text-to-image generation with diffusion models ...173 8 ■ CLIP: A model to measure the similarity between image and text 175 9 ■ Text-to-image generation with latent diffusion 201 10 ■ A deep dive into Stable Diffusion 225 Part 4 Text-to-image generation with transformers .... 243 11 ■ VQGAN: Convert images into sequences of integers 245 12 ■ A minimal implementation of DALL-E 268 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 9
viibrief contents Part 5 New developments and challenges ...................... 287 13 ■ New developments and challenges in text-to-image generation 289 appendix ■ Installing PyTorch and enabling GPU training locally and in Colab 315 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 10
viii contents preface xv acknowledgments xvii about this book xix about the author xxii about the cover illustration xxiii Part 1 Understanding attention and ..................... transformers ................................................1 1 A tale of two models: Transformers and diffusions 3 1.1 What is a text-to-image generation model? 4 Unimodal vs. multimodal models 5  ■  Practical use cases of text- to-image models 6 1.2 Transformer-based text-to-image generation 7 Converting an image into a sequence of integers and then back 8 Training and using a transformer-based text-to-image model 10 1.3 Text-to-image generation with diffusion models 11 Forward and reverse diffusions 12  ■  Latent diffusion models and Stable Diffusion 13 1.4 How to build text-to-image models from scratch 15 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 11
ixcontents 1.5 Challenges for text-to-image generation models 18 Are generative AI models stealing from artists? 18 The geometric inconsistency problem 19 1.6 Social, environmental, and ethical concerns 19 2 Build a transformer 22 2.1 An overview of attention and transformers 24 How the attention mechanism works 24  ■  How to create a transformer 28 2.2 Word embedding and positional encoding 33 Word tokenization with the Spacy library 34  ■  A sequence padding function 39  ■  Input embedding from word embedding and positional encoding 41 2.3 Creating an encoder–decoder transformer 43 Coding the attention mechanism 43  ■  Defining the Transformer() class 44  ■  Creating a language translator 46 2.4 Training and using the German-to-English translator 47 Training the encoder–decoder transformer 47 Translating German to English with the trained model 48 3 Classify images with a vision transformer 52 3.1 The blueprint to train a ViT 54 Converting images to sequences 54  ■  Training a ViT for classification 56 3.2 The CIFAR-10 dataset 57 Downloading and visualizing CIFAR-10 images 58 Preparing datasets for training and testing 59 3.3 Building a ViT from scratch 60 Dividing images into patches 61  ■  Modeling the positions of different patches in an image 63  ■  Using the multi-head self-attention mechanism 64  ■  Building an encoder-only transformer 65  ■  Using the ViT to create a classifier 67 3.4 Training and using the ViT to classify images 68 Choosing the optimizer and the loss function 69  ■  Training the ViT for image classification 70  ■  Classifying images using the trained ViT 72 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 12
x contents 4 Add captions to images 79 4.1 Training and using a transformer to add captions 81 Preparing data and the causal attention mask 81 Creating and training a transformer 84 4.2 Preparing the training dataset 86 Downloading and visualizing Flickr 8k images 87  ■  Building a vocabulary of tokens 88  ■  Preparing the training dataset 89 4.3 Creating a multimodal transformer to add captions 91 Defining a ViT as the image encoder 91  ■  Creating the decoder to generate text 94 4.4 Training and using the image-to-text transformer 96 Training the encoder–decoder transformer 96  ■  Adding captions to images with the trained model 99 Part 2 Introduction to diffusion models .......... 103 5 Generate images with diffusion models 105 5.1 The forward diffusion process 107 How diffusion models work 107  ■  Visualizing the forward diffusion process 109  ■  Different diffusion schezdules 111 5.2 The reverse diffusion process 115 5.3 A blueprint to train the U-Net model 117 Steps in training a denoising U-Net model 117  ■  Preprocessing the training data 118 5.4 Training and using the diffusion model 120 The Denoising Diffusion Probabilistic Model noise scheduler 120 Inference using the U-Net denoising model 121  ■  Training and using the denoising U-Net model 122 6 Control what images to generate in diffusion models 126 6.1 Classifier-free guidance in diffusion models 128 An overview of classifier-free guidance 128  ■  A blueprint to implement CFG 129 6.2 Different components of a denoising U-Net model 131 Time step embedding and label embedding 132  ■  The U-Net denoising model architecture 135  ■  Down blocks and up blocks in the U-Net 137 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 13
xicontents 6.3 Building and training the denoising U-Net model 142 Building the denoising U-Net 142  ■  The Denoising Diffusion Probabilistic Model 144  ■  Training the diffusion model 145 6.4 Generating images with the trained diffusion model 147 Visualizing generated images 148  ■  How the guidance parameter affects generated images 149 7 Generate high-resolution images with diffusion models 152 7.1 Attention in U-Net, DDIM, and image interpolation 154 Incorporating the attention mechanism in the U-Net model 154 Denoising Diffusion Implicit Models 156  ■  Image interpolation in diffusion models 156 7.2 High-resolution flower images as training data 158 Visualizing images in the training dataset 159 Applying forward diffusion on flower images 161 7.3 Building and training a U-Net for high-resolution images 163 Building the denoising U-Net model 163  ■  Training the denoising U-Net model 165 7.4 Image generation and interpolation 166 Using the trained denoising U-Net to generate images 166 Transition from one image to another 168 Part 3 Text-to-image generation with diffusion ... models ........................................................ 173 8 CLIP: A model to measure the similarity between image and text 175 8.1 The CLIP model 177 How the CLIP model works 177  ■  Selecting an image from Flickr 8k based on a text description 178 8.2 Preparing the training dataset 180 Image–caption pairs in Flickr 8k 180  ■  The DistilBERT tokenizer 183  ■  Preprocess captions and images for training 184 8.3 Creating a CLIP model 187 Creating a text encoder 187  ■  Creating an image encoder 189 Building a CLIP model 190 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 14
xii contents 8.4 Training and using the CLIP model 192 Training the CLIP model 193  ■  Using the trained CLIP model to select images 195  ■  Using the OpenAI pretrained CLIP model to select images 197 9 Text-to-image generation with latent diffusion 201 9.1 What is a latent diffusion model? 203 How variational autoencoders work 203  ■  Combining a latent diffusion model with a variational autoencoder 205 9.2 Compressing and reconstructing images with VAEs 207 Downloading the pretrained VAE 207  ■  Encoding and decoding images with the pretrained VAE 209 9.3 Text-to-image generation with latent diffusion 213 Guidance by the CLIP model 213  ■  Diffusion in the latent space 215  ■  Converting latent images to high-resolution ones 216 9.4 Modifying existing images with text prompts 219 10 A deep dive into Stable Diffusion 225 10.1 Generating images with Stable Diffusion 227 10.2 The Stable Diffusion architecture 230 Generating images from text with Stable Diffusion 231 Text embedding interpolation 232 10.3 Creating text embeddings 234 10.4 Image generation in the latent space 236 10.5 Converting latent images to high-resolution ones 238 Part 4 Text-to-image generation with .................... transformers ............................................ 243 11 VQGAN: Convert images into sequences of integers 245 11.1 Converting images into sequences of integers and back 247 11.2 Variational autoencoders 249 What is an autoencoder? 249  ■  The need for VAEs and their training methodology 251 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 15
xiiicontents 11.3 Vector quantized variational autoencoders 252 The need for VQ-VAEs 252  ■  The VQ-VAE model architecture and training process 253  ■   11.4 Vector quantized generative adversarial networks 256 Generative adversarial networks 256  ■  VQGAN: A GAN with a VQ-VAE generator 257 11.5 A pretrained VQGAN model 259 Reconstructing images with the pretrained VQGAN 259 Converting images into sequences of integers 263 12 A minimal implementation of DALL-E 268 12.1 How min-DALL-E works 269 Training min-DALL-E 270  ■  From prompt to pixels: Image generation at inference time 272 12.2 Tokenizing and encoding the text prompt 273 Tokenizing the text prompt 273  ■  Encoding the text prompt 277 12.3 Iterative prediction of image tokens 278 Loading the pretrained BART decoder 279  ■  Predicting image tokens using the BART decoder 280 12.4 Converting image tokens to high-resolution images 282 Loading the pretrained VQGAN detokenizer 282  ■  Visualizing the intermediate and final high-resolution outputs 283 Part 5 New developments and challenges ....... 287 13 New developments and challenges in text-to-image generation 289 13.1 State-of-the-art text-to-image generators 290 DALL-E series 290  ■  Google’s Imagen 291  ■  Latent diffusion models: Stable Diffusion and Midjourney 292 13.2 Challenges and concerns 293 13.3 A blueprint to fine-tune ResNet50 295 The history and architecture of ResNet50 295  ■  A plan to fine- tune ResNet50 for classification 297  ■  Using ResNet50 to classify images 299 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 16
xiv contents 13.4 Fine-tuning ResNet50 to detect fake images 304 Downloading and preprocessing real and fake face images 304 Fine-tuning ResNet50 308  ■  Detecting deepfakes using the fine-tuned ResNet50 311 appendix Installing PyTorch and enabling GPU training locally and in Colab 315 references 323 index 327 Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 17
xv preface This book begins with my curiosity about how machines could create images from nothing more than words. When I first encountered DALL-E and Stable Diffusion, the results seemed magical: type a prompt, and out came a lifelike image that matched the description perfectly. But behind the magic were mathematics, code, and a long line of ideas in machine learning. I wanted to demystify those ideas, not just for myself, but for anyone who learns best by building things from scratch. Generative AI is advancing at a pace few of us could have predicted, reshaping not only the way we work but also how we create, design, and communicate. Text-to- image models in particular are among the most visible and transformative of these technol- ogies. They embody the leap from unimodal to multimodal AI, systems that reason across different types of data. While the headlines focused on their impressive outputs, I found myself drawn to this question: How do they really work? The only satisfying answer, I decided, was to build one myself. This book is the result of that journey. It’s not a collection of high-level explanations or black box demonstrations. Instead, it’s a hands-on guide to re-creating the funda- mental building blocks of text-to-image generation: transformers, vision models, dif- fusion processes, and latent representations. By reconstructing these systems piece by piece, readers like you gain a deeper understanding of both their power and their lim- itations. As Richard Feynman once put it, “What I cannot create, I do not understand.” That spirit guides every chapter. Writing this book also came from a desire to bridge two communities: the machine learning researchers pushing the frontier of generative models and the developers, designers, and enthusiasts who are eager to harness these tools but uncertain where to begin. My hope is that by working through code, experiments, and projects, you’ll see Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 18
xvi preface that these models aren’t impenetrable black boxes, but accessible systems built from understandable components. The examples in this book are intentionally playful (pandas in top hats, bananas on motorcycles) because creativity should be joyful. But the lessons carry serious value— from rapid prototyping in design and marketing, to aiding education, to enabling new forms of artistic expression. By the end of this book, I hope you not only understand how text-to-image models function but also feel comfortable extending, adapting, and imagining new applications of your own. Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 19
xvii acknowledgments Many people have helped bring this book from idea to reality, and I’m deeply grateful for their contributions. First, I owe special thanks to Jonathan Gennick, my acquisition editor at Manning. Jonathan not only believed in this project from the very beginning but also guided it with a clear vision of what readers most want to learn. His thoughtful feedback helped me shape the structure of the book in a way that makes the learning journey approach- able and rewarding. I’m equally indebted to my development editor, Doug Rudder, whose insistence on clarity pushed me to become a better writer. Doug’s ability to spot where explana- tions could be simplified or examples expanded has left a lasting mark on the book. His encouragement to always aim for clarity, even when discussing the most technical details, has greatly enhanced the accessibility of the book. My sincere gratitude also goes to my technical editor, Nathan Crocker, author of AI-Powered Developer (Manning, 2024) and cofounder and CTO of Checker, the global liquidity network for stablecoins. Nathan’s rigorous technical review and insightful comments were invaluable. His careful attention to detail caught subtle issues, and his practical suggestions improved the book tremendously. One memorable example is from chapter 11, where I initially instructed readers to clone and manually update an outdated GitHub repository. Nathan wisely proposed that I instead provide a forked and updated repository so readers could dive directly into experimentation without unnecessary manual updating. It’s improvements like these, both large and small, that make this book far more accessible to readers. I would also like to thank the broader Manning team, including the production edi- tor, copyeditor, and reviewers, who worked behind the scenes to polish the manuscript Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
Page 20
xviii acknowledgments and ensure its accuracy. Their efforts often go unseen but are crucial to turning a draft into a finished book. To all of the following reviewers, your suggestions helped make this a better book: Abhilash Babu, Adil Patel, Akinwale Habib, Alejandro Cuevas Riv- ero, Amaresh Rajasekharan, Arun Lakhera, Atilla Özgür, Curtis Bates, Devavrat Sabnis, Dieter Späth, Donald Bleyl, Eli Richmond Hini, Gautham K, Giovanni Alzetta, Har- dev Ranglani, Ifiok Moses, Ijem Ofili, Ishita Verma, Jahred Love, Jan Goyvaerts, José Salavert-Torres, Khashayar Baghizadeh, Mariia Bulycheva, Matheus Antônio Nogueira, Mouhamed Klank, Naga Sai Abhinay Devarinti, Oscar Peña, Pavan Kumar Adepu, Praveen Nair, Ravikumar Sanapala, Raymond Cheung, Sergio Arbeo, Shabie Iqbal, Mohammad Shahnawaz Akhter, Simeon Leyzerzon, Sukanya Konatam, Tathagata Dasgupta, Thomas M. Seeber, Trinnawat Charoenpradubsilp, Vikram Shibad, Vishal Gandhi, Xin Hu, and Yuanyuan Chen. Finally, I want to express my deepest gratitude to my family. To my wife, Ivey Zhang, thank you for your patience, encouragement, and constant belief in this project, even during the long evenings and weekends I spent writing. To my son, Andrew Liu, thank you for reminding me of the joy of curiosity, the same spirit that inspired me to write this book. This journey wouldn’t have been possible without your love and support. Licensed to Ibrahim Mohammed Mohammed <ibrahim@dotnetebusiness.com>
The above is a preview of the first 20 pages. Register to read the complete e-book.

Recommended for You

Loading recommended books...
Failed to load, please try again later

Tip the Site

Scan the WeChat Pay or Alipay code to tip. No login required.

WeChat Pay
Alipay
← Back to List