Supercomputers for Linux SysAdmins Managing Modern HPC Clusters and Supercomputers from Software to Hardware (Sergey Zhumatiy) (Z-Library)

Supercomputers for Linux SysAdmins Managing Modern HPC Clusters and Supercomputers from Software to Hardware Sergey Zhumatiy

Supercomputers for Linux SysAdmins: Managing Modern HPC Clusters and Supercomputers from Software to Hardware ISBN-13 (pbk): 979-8-8688-1599-7 ISBN-13 (electronic): 979-8-8688-1600-0 https://doi.org/10.1007/979-8-8688-1600-0 Copyright © 2025 by Sergey Zhumatiy This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: James Robinson-Prior Coordinating Editor: Gryffin Winkler Cover image designed by Freepik (www.freepik.com) Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza, New York, NY 10004. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a Delaware LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail booktranslations@springernature.com; for reprint, paperback, or audio rights, please e-mail bookpermissions@springernature.com. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub. For more detailed information, please visit www.apress.com/gp/services/source-code. The initial version of this book was originally written in the Russian language and was translated in English with the help of deepl.com. The initial version was updated and edited later. If disposing of this product, please recycle the paper Sergey Zhumatiy Santa Clara, CA, USA

xv About the Author Sergey Zhumatiy has been managing supercomputers since 1999, starting out with building and managing HPC clusters at Moscow State University, and holds a PhD in Computer Science. Several supercomputers under his supervision, like Chebyshev, Lomonosov, and Lomonosov-2, achieved top rankings in the top 500 supercomputers list and dominated the Russian top 50 supercomputers list. Now he works as an HPC architect and SysAdmin at NVIDIA.

xvii John Roberts spent over a decade as an HPC systems administrator, supporting large-scale scientific computing and advanced research initiatives. He recently transitioned to a new role where he continues to focus on supporting and optimizing HPC environments. John holds a bachelor's degree in computer science and brings a passion for scalable systems, innovation, and advancing scientific discovery.    About the Technical Reviewer

xix Acknowledgments I would like to express my sincere gratitude to the following people: Vladimir Voevodin for the ideas and criticism Konstantin Stefanov, Alexander Naumov, Anton Korzh, Ilya Fateev, Ben Evans, and Caio Davi for the provided material and consultations Hui Li, Paniz Karbasi, Oksana Korzh, and Mark Moe for their patience and valuable advice and ideas Viktor Datsyuk, Pavel Kostenetsky, Alexei Latsis, and Yuri Khrebtov for important comments John Roberts, the technical reviewer of this book, for the great work and tons of thoughtful corrections and suggestions

xxi Glossary of Terms Backup: A copy of data (files, databases) stored on a separate media or a group of media. Data can be restored from the backup copy to the original files, databases, or to other directories, databases. Cable organizer: A design that allows cables to be stacked within a dedicated space. Communication network: Used to exchange data by computing tasks. Compute field: A set of all compute nodes of the complex that are available for user tasks. DAPL: Direct Access Programming Library, a library for using direct access to remote computer memory without having to explicitly describe a particular type of hardware. D-BUS: System Message Bus, a server and protocol that allows any programs to communicate within the same server. A program can register as a service and publish messages (events) and as a client and subscribe to certain events. DHCP: Dynamic Host Configuration Protocol, a protocol that allows a computer to obtain information such as its IP address, network name, etc., from a server at the OS boot stage or later. File server: A computer that makes part of its file system available to other computers over a network. File storage: Equipment that provides disk space over a network or locally, such as for a file server. Form factor: Standard dimensions for computer cases, like mini-tower, 1U, 2U, etc. FPGA: Field Programmable Gate Array, a device, which can be programmatically reconfigured into various combinations of "gates" – logical units – and form specialized compute devices. FTP: File Transfer Protocol, a protocol for transferring files over a network. GPGPU: General-Purpose GPU, a GPU, which is used for computations.

xxii GPU: Graphics Processing Unit, device for generating and displaying (via separate display) graphics. Today, any video card. Host: A node on a network – a server, a computer. HPC: High-Performance Computing, the supercomputing industry. Interconnect: A jargonism, usually referring to a communications network or just a fast network. Journaling: Writing messages to a log. Many programs support logging to a file. In UNIX- like systems, there is a standard syslog service, which is used by many programs and services. KVM: Keyboard and Video Monitor, a device that allows you to connect multiple computers to a single monitor and keyboard. Latency: The time spent when a packet is transmitted through a network regardless of its length. Linpack: A test for some estimation of real performance of parallel computing complexes. Most often, the High Performance Linpack (HPL) version is used. Logging: See journaling. LVM: Logical Volume Manager, a technology for building logical disks using multiple physical disks and/or RAIDs. MAC address: The unique address of a network card in the Ethernet standard. MPI: Message Passing Interface, an open library standard intended for message passing inside a parallel application. There are many implementations of this standard (mpich, lam, openmpi, etc.). NIS: Network Information System, a technology that allows user accounts, computer names, and other system information to be stored on a server and retrieved from any computer on the network. Node (of a cluster): a computer designed for certain tasks in the cluster (computing, controlling, I/O, etc.). NTP: Network Time Protocol, a protocol for synchronizing time over a network. Rack: See telecommunication cabinet. Glossary of Terms

xxiii RAID: Redundant Array of Independent/Inexpensive Disks, an array of multiple hard disks logically combined for greater fault tolerance, speed, and/or capacity. RAID-0 (stripe): RAID, which disks are combined in such a way that logical blocks of disks alternate: block1 of the first disk, block1 of the second, … block2 of the first disk, block2 of the second disk, etc. RAID-1 (mirror): RAID, the disks of which are combined into a "mirror" to increase reliability. Information is written simultaneously to all disks in blocks with the same numbers. RAID-5: A RAID whose disks are combined into parity groups. When writing to a logical block, the written data is added by XOR method with other blocks in the group and the resulting information is written to a separate block. When reading, the correctness of the data is checked, and if one of the blocks is corrupted, the information is automatically restored. RAID controller: A device that combines multiple hard disks into a RAID. RDMA: Remote Direct Memory Access, a protocol for direct memory access to a remote computer. Register: Internal memory of the processor, works very fast, all arithmetic and logical operations are typically performed with registers only. Samba: A software package that implements the SMB and CIFS protocols used in MS Windows for network disks. It allows accessing Windows network disks from Linux, as well as creating network disks under Linux so that they can be used by Windows clients. SCI: Scalable Coherence Interface, an old standard for high-speed data transmission equipment. It means connecting network cards directly to each other in a ring or torus (two- or three-dimensional). Service network: Used to monitor and manage the state of compute nodes. SNMP: Simple Network Management Protocol, a protocol designed to monitor and manage equipment on a network. Software: In addition to the actual set of programs, this includes configuration and other files necessary for its operation. SSH: Secure Shell, a protocol for remote access to computers on a network, involving the use of an encrypted connection. Glossary of Terms