Generative AI on Kubernetes Operationalizing Large Language Models (Roland Huß, Daniele Zonca)（Z-Library）

【One-Line Pitch】 A hands-on field guide for turning large language models into reliable production services on Kubernetes, covering everything from GPU scheduling and model formats to inference tuning, evaluation, and agentic applications. Best suited to MLOps engineers, platform/Kubernetes administrators, and developers who already know the cloud native basics and now need to run GenAI workloads at scale. 【Book Arc】 - **Opening (~0%–10%)**: Frames why Kubernetes is the natural home for GenAI workloads—its ability to co-schedule LLMs alongside databases, microservices, and traditional apps—and sketches the historical arc of generative AI from early sequence prediction to the deep learning era. - **Early (~10%–35%)**: Builds the conceptual and practical foundation: tokenization, embeddings, prompts and system prompts, Transformer attention, and the local-to-cluster jump (memory sizing for 7B vs. 70B models, GPU requirements). Introduces model servers like Hugging Face TGI and the shift from raw Deployments to model server controllers. - **Middle (~35%–55%)**: Dives into model data plumbing—file formats (GGUF/GGML, ONNX, TensorFlow SavedModel, Safetensors), the trade-offs between self-contained and multi-artifact layouts, and model registries (Hugging Face Hub, MLflow) that index metadata while pointing to external object storage. - **Late (~55%–85%)**: Moves into production operations: GPU scheduling with the NVIDIA GPU Operator, Dynamic Resource Allocation, sub-GPU and multi-GPU inference, data vs. model parallelism, single-node vs. multinode serving, and runtime tuning (vLLM parameters, startup time, autoscaling, LLM-aware routing). - **Ending (~85%–100%)**: Closes on the operational frontier—model evaluation and benchmarking before committing GPU resources, compression techniques, and agentic applications with secure tool integration, identity management, and persistent state. 【Key Takeaways】 - **Kubernetes wins by composition, not specialization** (Opening): Unlike Ray or Spark, which are purpose-built for ML, Kubernetes can run LLMs alongside ordinary business workloads, simplifying operations for complex multi-workload applications. - **Subword tokenization is the quiet enabler of modern LLMs** (Early): Breaking text into subword units (e.g., "regular" + "ization") eliminates the unknown-word problem while keeping vocabularies manageable—a prerequisite for handling rare terms and names. - **Prompts are more than user input** (Early): System prompts strongly shape model behavior, and prompt engineering—adding context to steer generation—is a first-class skill, not an afterthought. - **Model size dictates hardware reality** (Early): A 7B model needs roughly 15 GB of GPU memory; a 70B model needs about 140 GB. This single fact drives most deployment topology decisions. - **Model formats are a compatibility minefield** (Middle): GGUF and Safetensors are the practical choices today; true standardization (à la OCI for containers) is still distant, so format choice ties you to specific runtimes. - **Registries manage metadata, not weights** (Middle): Production registries live inside the cluster, track versions and governance, and reference external object stores like S3—keeping large model artifacts out of the registry itself. - **Multinode serving makes deployments stateful** (Late): Distributed inference requires coordination components and changes your topology; MLOps engineers must understand parallelism settings before scaling out. - **Evaluate before you spend GPU hours** (Ending): Standardized benchmarks and compression trade-offs should be resolved during model development, so inference-time configuration arrives ready to deploy. 【Reading Tips】 - **Skim the history, deep-read the mechanics.** The early GenAI timeline is context; the tokenization, memory-sizing, and format chapters are where decisions get made. - **Treat the YAML examples as reference, not tutorial.** The LLMInferenceServiceConfig/LLMInferenceService snippets reward readers who already know Kubernetes primitives—pause on the router, scheduler, and parallelism sections. - **Watch for the recurring "complexity multiplies" theme.** Each new model adds manifests, PVCs, and resource coordination; use this as your mental checklist when planning your own deployments. - **Bookmark the format comparison and registry sections.** These are the chapters you will return to when choosing runtimes or setting up governance. - **Read the ending chapters before you provision GPUs.** Evaluation and benchmarking guidance is most valuable before, not after, you commit hardware. 【Coverage Limits】 This guide is synthesized from stratified excerpts covering roughly the first half of the book in detail, with lighter coverage of the production and agentic chapters. Specific chapter titles, benchmark names, and later-chapter examples are not fully represented in the excerpts.