Video Generation with AI Working with Diffusion Transformers and Multimodal Learning (Joseph Enochs) (z-library.sk, 1lib.sk, z-lib.sk)

Praise for Video Generation with AI With practical, industry-relevant examples throughout, this book makes modern video AI approachable and actionable. Readers will gain the confidence to build, fine-tune, and deploy cutting-edge video generation systems while mastering diffusion transformers, multimodal learning, and production-ready workflows. —Maxime Dehaut, head of product and application development, Luxembourg Stock Exchange A definitive technical standard. This text masterfully bridges theoretical foundations and cutting-edge multimodal research with the rigorous, distributed system design necessary for production-scale generative deployments, offering practitioners a masterful, hands-on roadmap to modern video generation. —Debanshu Das, senior technical lead for AI video generation and recommendation Open-sourcing LTX was about empowering builders. Seeing it taught in a book is exactly the kind of impact we hoped for. —Dr. Zeev Farbman, cofounder and CEO, LTX

Video Generation with AI Working with Diffusion Transformers and Multimodal Learning Joseph Enochs

Video Generation with AI by Joseph Enochs Copyright © 2026 Joseph Enochs. All rights reserved. Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Nicole Butterfield Development Editor: Corbin Collins Production Editor: Ashley Stussy Copyeditor: nSight, Inc. Proofreader: Dwight Ramsey Indexer: nSight, Inc. Cover Designer: Karen Montgomery Cover Illustrator: José Marzan Jr. Interior Designer: David Futato Interior Illustrator: Kate Dullea August 2026: First Edition Revision History for the First Edition

2026-07-21: First Release See https://oreilly.com/catalog/errata.csp?isbn=9798341653344 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Video Generation with AI, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 979-8-341-65334-4 [LSI]

Preface Video generation is advancing at a remarkable pace. New models and techniques appear almost monthly, each improving realism, consistency, and multimodal reasoning. Yet often the difficult part is not using these models. It is understanding why they work. I wrote this book because I believe that real understanding comes from connecting theory with implementation. Reading a paper is valuable. Building a model is valuable. The greatest insight comes from understanding how the architecture, training process, and engineering decisions fit together to produce the final result. This is not a catalog of every video generation model or a survey of every research paper. The field is moving too quickly for that. Instead, this book focuses on the core ideas that appear across modern systems and explains them through practical examples and working implementations. My hope is that this book helps the next generation of builders move beyond treating these models as black boxes and gives them the foundation to build what comes next. Who Should Use This Book This book is intended for software developers, machine learning engineers, AI researchers, data scientists, and technical architects who want to understand how modern video generation systems are designed, trained, and deployed. It assumes a basic familiarity with Python and machine learning concepts but does not require prior experience with video generation. By connecting theory with implementation, the book aims to help you understand not only how these systems work, but why they work.

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context. TIP This element signifies a tip or suggestion. NOTE This element signifies a general note.

WARNING This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/jenochs/video-ai-book-companion. If you have a technical question or a problem using the code examples, please send email to support@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Video Generation with AI by Joseph Enochs (O’Reilly). Copyright 2026 Joseph Enochs, 979-8-341-65334-4.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning

NOTE For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 141 Stony Circle, Suite 195 Santa Rosa, CA 95401 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://oreilly.com/about/contact.html

We have a web page for this book, where we list errata and any additional information. You can access this page at https://oreil.ly/video-generation- with-ai-1e. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments No book is written alone, and this one is no exception. I would like to thank the team at O’Reilly Media for their guidance and support throughout this project. My sincere thanks go to Nicole Butterfield, Corbin Collins, Ashley Stussy, Adam Lawrence, and Kristen Brown for helping shape and bring this manuscript to publication. I am especially grateful to Michael Demers, Seb de Greef, Jordan Harvey, Maxime Dehaut, and Debanshu Das for their collaboration, technical insight, thoughtful feedback, and encouragement throughout the writing process. I also want to recognize the researchers, engineers, and open source communities whose work forms the foundation of modern AI video generation. Their willingness to publish research, share ideas, and contribute code has accelerated innovation across the industry and made books like this one possible. Finally, I want to thank my family, especially my children. Your patience, encouragement, and unwavering support gave me the time and space to complete this book. My greatest hope is that the world you inherit is one where curiosity, creativity, and technology are used to make life better for everyone.

Chapter 1. Foundations of Video Generation Learning objective: In this chapter, you’ll learn the foundational principles and emerging methods used to generate video with AI. Video generation represents one of the most challenging frontiers in AI. While generating a single image requires understanding spatial relationships within a frame, video generation introduces challenges that exponentially multiply this complexity. Models must account not only for the content of each frame but also for how those elements change logically and coherently over time. This chapter guides you from understanding these challenges to creating high-resolution videos with advanced generative models. By the end, you’ll realize why diffusion transformers (DiTs) have transformed the field and you’ll gain practical skills for working with cutting-edge video AI systems. We begin with immediate, hands-on success—generating a sample video in minutes—then build a deeper understanding of the technology that makes it possible. Hands-On: Your First AI Video Let’s start by generating a video. You’ll use CogVideoX, one of the most advanced open source models, selected for its accessibility and compatibility with a wide range of consumer-grade hardware and cloud platforms, such as Google Colab. While not a replacement for full production pipelines, CogVideoX can produce high-resolution, temporally coherent video outputs suitable for prototyping, experimentation, and creative development. This initial

success sets the stage for a more in-depth exploration of the architecture and techniques behind modern generative video models. Environment Setup First, you’ll need to prepare your environment with modern AI tooling. Please open the Chapter 1 Notebook and create a copy for your experimentation. The tooling selected here is based on the foundational requirements for video generation. You’ll mainly need the torch library at this point: import torch import numpy as np import time import matplotlib.pyplot as plt from IPython.display import Video, display, HTML Next, verify your computational environment, since video generation requires significant resources. I have chosen a model with moderate resource requirements for you to use for the demonstration exercises in this chapter: print("Video Generation AI Environment (2025 Edition)") print(f"PyTorch: {torch.__version__}") print(f"GPU Available: {torch.cuda.is_available()}") if torch.cuda.is_available(): print(f"GPU: {torch.cuda.get_device_name(0)}") print( f"VRAM: " f"{torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f}GB" ) else: print("No GPU detected - video generation will be slower") print("\n Environment ready for state-of-the-art video AI!") Next, install the necessary libraries for video generation. If you’re running this process for the first time locally, you may need to install the Diffusers library and other required dependencies. If you’re using Google Colab, the

notebooks provided by Google are likely preconfigured with compatible versions: # Install the video AI stack # Reference environment (known-good configuration used in this book) !pip install diffusers==0.33.1 transformers==4.52.4 accelerate safetensors \ --quiet !pip install imageio-ffmpeg opencv-python scikit-image --quiet # Import tools required for video generation from diffusers import CogVideoXPipeline from diffusers.utils import export_to_video import diffusers Loading the Model You’ll now load the CogVideoX model: # Start the timer start_time = time.time() # Load the CogVideoX Pipeline pipe = CogVideoXPipeline.from_pretrained( "THUDM/CogVideoX-2b", # Accessible model # "THUDM/CogVideoX-5b", # Larger model optional torch_dtype=torch.float16, # float 16 precision ) Now you can apply model optimizations, which can have a significant impact on whether your model runs successfully on your selected platform. The main components to be concerned about are VRAM on the GPU and system memory. These resources can become overloaded during the video generation process, so applying available optimizations is crucial to a successful generation: # Apply optimizations pipe.enable_model_cpu_offload() # Move unused comps to CPU. pipe.vae.enable_tiling() # Process in tiles pipe.vae.enable_slicing() # VAE optimizations # Track our load time and display results load_time = time.time() - start_time print(f"\n CogVideoX loaded and optimized in {load_time:.1f}s!")

Generate Your First Video With the model loaded and optimizations set, you’re ready to begin generating your first AI-generated video. The foundation of successful AI video is the prompt. A well-structured natural-language prompt can guide models toward generating outputs with strong visual impact. At the same time, prompts often need a “negative” counterpart to reduce common generation artifacts. These artifacts refer to visual flaws that arise during synthesis, such as distorted body parts, unnatural movements, or inconsistent backgrounds. By pairing positive and negative prompts, we can guide the model to produce imagery that is both compelling and technically clean: # Craft a prompt and negative prompt prompt = ( "A majestic golden retriever running through a sunlit " "meadow, cinematic quality, slow motion" ) negative_prompt = "blurry, low quality, distorted, text" Now run the generation process: # Track the generation process start_time = time.time() # Generate with settings optimized for quality and success video_frames = pipe( prompt=prompt, negative_prompt=negative_prompt, num_frames=49, # ~2 seconds at 24fps height=480, # Balanced quality/memory width=720, # Standard aspect ratio num_inference_steps=50, # 50 refinement steps guidance_scale=6.0, # Text conditioning generator=torch.Generator().manual_seed(42) ).frames[0] generation_time = time.time() - start_time print(f"\n Generation completed successfully!") print(f"Time: {generation_time:.1f} seconds") print( f"Generated: {len(video_frames)} frames "

f"({len(video_frames)/24:.1f}s of video)" ) Save and Display Your Achievement Once your video is generated, you can save and preview it so you can review your progress, share your work, and use it as a reference in future experiments and evaluations: # Save and display your achievement output_filename = "my_first_ai_video.mp4" export_to_video(video_frames, output_filename, fps=24) print(f"\n Video saved as: {output_filename}") print("\n Your First AI-Generated Video:") display(Video(output_filename, width=720, height=480, embed=True)) print("\n Congratulations! You've just created your first video using AI.") Congratulations! You’ve just created your first video using AI. Figure 1-1 shows the output I got.

Figure 1-1. A majestic golden retriever running through a sunlit park, generated by CogVideoX, July 2025 Introduction to Generative Video Models The video you just created represents the culmination of years of research in AI. To understand why this achievement is so significant, we must examine what makes video generation fundamentally distinct from other AI tasks. Video generative models represent a groundbreaking collaboration between artist and algorithm. Imagine a paintbrush that not only understands your creative vision but evolves with you, suggesting possibilities while remaining firmly guided by your direction. These emerging AI systems represent a growing component of creative collaboration, engaging in dynamic dialogue with professionals to push the boundaries of what’s possible in video creation while preserving the essence of artistic vision.

These models are emerging as intelligent, creative partners that learn from and build upon vast repositories of visual knowledge. They use advanced machine learning (ML) techniques to predict how images should flow from one frame sequence to another. What Makes Video Unique in AI Video generation has unique challenges that distinguish it from static image generation. While image models focus on spatial relationships within a single frame, video models must also account for how those relationships evolve across time. This introduces a temporal dimension to the problem, which requires consistency in motion, character identity, lighting dynamics, and visual perspective across multiple frames. Temporal consistency The first and perhaps most critical challenge is temporal consistency. Unlike static images, videos require maintaining coherent motion and scene continuity across multiple frames. Objects must move naturally, lighting conditions must remain consistent, and camera movements must follow realistic physics. Consider a simple scene of a person walking: each frame must show the walker in a slightly different position, with clothing that moves naturally, shadows that shift appropriately, and background elements that maintain their relative positions. Higher computational demands This leads directly to the second major challenge: exponentially higher computational demands. Generating multiple frames in sequence requires significantly more computational resources than producing single images. A 2-second video at 24 frames per second contains 48 individual photos, each of which must maintain coherent relationships with all others—a 48x multiplication in raw data complexity. To put this in perspective, a single frame at 720 × 480 resolution with three color channels contains 1,036,800 values to model. The same content, as a 2-second video, balloons to 49,766,400 values, creating an exponential increase in complexity as we

extend the video generation sequence. (See Figure 1-2 for an illustration of this increase.) Sequence of events The third dimension of complexity involves how videos tell stories through sequences of events. Videos convey the narrative of the story and require the AI models to understand the relationships between action sequences, along with the natural flow of events. This temporal storytelling dimension adds layers of complexity beyond spatial composition. A video of a glass falling and breaking must show not just the individual frames but the physics of acceleration, the moment of impact, and the fragments scattering realistically. These massive data requirements explain why video generation requires revolutionary architectural breakthroughs, particularly the development of models with billions of parameters. Models have to keep track of how objects and scenes should look within each frame, ensure temporal coherence across frames, and maintain physical realism in motion patterns and semantic understanding of text prompts, all while ensuring smooth motion and narrative flow throughout the sequence. Figure 1-2 illustrates the exponential complexity of video generation, along with the time steps required for your video pipeline. Figure 1-2. Video generation comparison and processing pipeline

Key Use Cases for Generative Video Video generative models are becoming more than theoretical concepts. In recent times, they have become transformative in industries, driving innovation across entertainment, advertising, healthcare, and beyond. Entertainment and media Google’s Veo series of models represents a significant leap forward among the first video generation models to support native audio, enabling creators to produce synchronized sound effects, ambient noise, and dialogue along with visuals in a single pass. This advancement could significantly reduce the manual effort required for separate audio production. Meta’s Movie Gen, introduced in late 2024, brings generative video capabilities into the social media ecosystem. Although, as of mid-2025, its full integration with platforms like Instagram has not been publicly confirmed, Movie Gen is positioned for short-form content creation. It offers 1080p resolution and synchronized audio output, supporting experimental workflows for creators and marketing teams. Early collaborations with filmmakers and studios, including Blumhouse Productions and Casey Affleck, are exploring its potential. Runway has also introduced many notable improvements in character and object consistency across scenes, a longstanding challenge with video generation. Runway leverages reference objects and tags to condition the output and maintain character consistency, to maintain coherent identities throughout multishot sequences, which is paramount for narrative storytelling. Advertising and marketing In advertising, generative video is driving a shift toward hyperpersonalized, scalable content production. Kling AI, with over 22 million users globally, enables brands to experiment with localized video variations, using features such as multi-image reference, a localization tool that helps advertisers maintain brand identity while dynamically adapting visuals for different

regions and demographics. For example, a global sportswear brand could generate localized ads featuring the same athlete running through different cities worldwide, adapting their clothing styles to regional preferences and inserting local landmarks. The cost efficiency could be remarkable. Meanwhile, Open-Sora introduces a new level of cost efficiency in video AI. Trained for just $200,000—10% to 20% of the cost of comparable models—it enables small and midsized businesses to access high-quality video generation tools that were once limited to large organizations. Healthcare and medical training Video AI is also transforming medical education by creating comprehensive, repeatable training libraries that offer cost-effective training for critical medical resources, making high-quality simulations more attainable. Modern simulation models can generate medically accurate content with sufficient visual fidelity for training purposes. Integrated audio features strive to replicate the surgical environment—including heart monitors and realistic machine sounds—and virtual patient interactions, enhancing the training experience. Cultural preservation and accessibility AI video tools are also helping cultural institutions reimagine preservation and accessibility. With native audio generation, historians can now re-create period-appropriate soundscapes for silent films or historical footage, including ambient sounds, music, and dialogue inspired by archival records. Modern models also allow museums to create immersive, accessible virtual experiences, generating multiple viewing angles of historical events from a single archival photograph. With the support of these emerging technologies, archives can generate audio descriptions for visually impaired visitors, create multiple language versions with culturally appropriate narration, and even produce educational content that offers “behind-the-scenes” perspectives on historical events.