Generative AI on Kubernetes Operationalizing Large Language Models (Roland Huss, Daniele Zonca)（Z-Library）

【One-Line Pitch】 A practical, hands-on guide for MLOps engineers, Kubernetes administrators, and AI professionals who want to deploy, scale, and operate large language models (LLMs) on Kubernetes, covering everything from GPU scheduling to model evaluation and agentic applications. 【Book Arc】 - **Opening (~0%–9%)**: Introduces the core premise—Kubernetes as the ideal platform for GenAI workloads—and provides a brief history of generative AI, from early concepts like the Turing Test to the deep learning revolution. Sets up the book's scope: training, fine-tuning, deploying, and scaling LLMs. - **Early (~9%–25%)**: Explains the fundamentals of LLMs, including tokenization, prompt engineering, and the Transformer architecture. Covers how to run models locally with Hugging Face Transformers, highlighting hardware requirements (e.g., GPU memory for 7B vs. 70B models) and the role of runtimes like TGI. - **Early (~25%–38%)**: Moves into Kubernetes-specific deployment, contrasting manual approaches (with taints, tolerations, PVCs) against higher-level abstractions like model server controllers (e.g., KServe's LLMInferenceService). Discusses distributed inference patterns for large models (70B+) and the trade-offs of single-node vs. multinode serving. - **Middle (~38%–53%)**: Focuses on model data management—storage formats (GGUF, Safetensors, ONNX), model registries (like MLflow), and how to organize, version, and serve model artifacts within a cluster. Emphasizes the separation of metadata from actual model weights for flexibility. - **Late (~53%–end)**: Covers production concerns: GPU resource scheduling (including NVIDIA GPU Operator and dynamic allocation), autoscaling, LLM-specific metrics (Time to First Token, token throughput), model compression, evaluation benchmarks, and advanced topics like agentic applications with secure tool integration and persistent state. 【Key Takeaways】 - **Kubernetes is the right platform for GenAI** (Early): Unlike specialized tools like Ray or Spark, Kubernetes can manage AI models alongside traditional applications, databases, and microservices, simplifying operations and enabling complex multi-workload compositions. - **Subword tokenization is essential for LLMs** (Early): Breaking text into subword units (e.g., "regularization" → "regular" + "ization") eliminates the unknown-word problem while keeping vocabulary sizes manageable, allowing models to handle rare terms and technical jargon. - **GPU memory scales with model size** (Early): A 7B-parameter model requires ~15 GB of GPU memory, while a 70B model needs ~140 GB. This drives decisions about hardware, quantization, and whether to use single-node or distributed serving. - **Model server controllers abstract Kubernetes complexity** (Early): Manual deployment of LLMs involves managing taints, tolerations, storage, secrets, and model-specific parameters—complexity that multiplies with each model. Controllers like KServe's LLMInferenceService hide this behind higher-level APIs focused on model deployment. - **Model formats are still evolving** (Middle): GGUF and Safetensors are currently the most practical choices for balancing performance, compatibility, and flexibility, but true standardization (like OCI for containers) is still far off. Safetensors' multifile structure works well with OCI artifacts for efficient caching and parallel downloads. - **Model registries separate metadata from weights** (Middle): Registries like MLflow manage model versions, governance, and metadata while referencing external object stores (e.g., S3) for actual weights. This separation enables flexibility and keeps metadata accessible within the cluster. - **LLM-specific metrics matter for production** (Late): Monitoring Time to First Token and token throughput is critical for optimizing user experience and resource utilization, requiring rethinking traditional autoscaling and traffic management for long-running GPU workloads. 【Reading Tips】 - **Skim the history and fundamentals** (Early): The opening chapters on GenAI history and Transformer architecture are useful context but not the core value. Focus on the practical deployment examples and code snippets. - **Deep-read the GPU scheduling and model server controller sections** (Early–Middle): These are the heart of the book—understanding taints/tolerations, dynamic resource allocation, and KServe's LLMInferenceService will save you hours of trial-and-error in production. - **Pay attention to model format comparisons** (Middle): The discussion of GGUF vs. Safetensors vs. ONNX is practical and decision-oriented. Use it to choose the right format for your runtime and storage strategy. - **Treat the production chapter as a checklist** (Late): Autoscaling, vLLM tuning, and evaluation benchmarks are actionable items you'll want to revisit when deploying your own models. Consider bookmarking the vLLM runtime parameters section. - **Note the gaps**: The excerpts don't cover agentic applications in depth, nor do they detail specific security implementations. If those are your focus, you may need supplementary resources. 【Coverage Limits】 This guide synthesizes the first ~53% of the book (through model registries). Later sections on production tuning, autoscaling, and agentic applications are only partially covered based on available excerpts.