Page
1
Synthesis Lectures on Visualization Enamul Hoque Prince Large Language Models for Data Visualization Hoque Prince Large Language M odels for Data Visualization
Page
2
Synthesis Lec tures on Visualization Series Editors David Ebert, Department of Electronics and Communication Engineering, University of Arizona, Tucson, AZ, USA Niklas Elmqvist, Aarhus University, Aarhus, Denmark
Page
3
This series publishes on topics pertaining to scientific visualization, information visual- ization, and visual analytics. Potential topics include, but are not limited to scientific, information, and medical visualization; visual analytics, applications of visualization and analysis; mathematical foundations of visualization and analytics; interaction, cognition, and perception related to visualization and analytics; data integration, analysis, and visual- ization; new applications of visualization and analysis; knowledge discovery management and representation; systems, and evaluation; distributed and collaborative visualization and analysis.
Page
4
Enamul Hoque Pr ince Large Language M odels for Data V isualization
Page
5
Enamul Hoque Prince School of Information Technology York University Toronto, ON, Canada ISSN 2159-516X ISSN 2159-5178 (electronic) Synthesis Lectures on Visualization ISBN 978-3-032-37872-9 ISBN 978-3-032-37873-6 (eBook) https://doi.org/10.1007/978-3-032-37873-6 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2027 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Unless otherwise expressly agreed by the Publisher in writing, the Publisher does not authorize use or reproduction of all or any part of this work in any manner for the purpose of training, fine-tuning, evaluating, or developing artificial intelligence models, technologies, or systems. In accordance with Article 4(3) of the EU Directive 2019/790 on copyright and related rights in the Digital Single Market, the Publisher expressly reserves all rights with respect to this work under the text and data mining exception of the directive. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland If disposing of this product, please recycle the paper.
Page
6
To my sons, Rayan and Safwan, whose curiosity in seeing and understanding the world inspires my own pursuit of meaning in data.
Page
7
Preface This book emerged from a simple but profound question: how can we make data not only visible, but truly understandable through language? Over the past few years, the rapid evolution of large language models (LLMs) has transformed the landscape of artificial intelligence and rekindled old dreams in visualization research—of making data commu- nication conversational, adaptive, and accessible. Yet, despite the explosion of research at this intersection, there has been no comprehensive book dedicated to this timely and rapidly developing field. This volume aims to fill that gap. My journey toward this work spans more than 15 years of exploring the intersec- tion of natural language processing (NLP) and information visualization. During this period, I have witnessed both disciplines mature—NLP expanding from rule-based sys- tems to large-scale generative models, and visualization evolving from static graphics to interactive and personalized experiences. This book represents the synthesis of those developments: an attempt to articulate how language and visualization, when combined, can reshape how we reason with data, tell stories, and design tools that augment human intelligence. The motivation for this book arose from a growing recognition that visualization research is no longer confined to visual encodings and perceptual optimization, nor is NLP limited to text generation and comprehension. When language becomes the interface to data, and visualization becomes the medium of reasoning, both fields expand—intersecting with human computer interaction, cognitive science, accessibility, and design. This book brings these threads together, presenting a coherent account of the emerging landscape of LLM-driven visualization systems. The book is intended for a broad audience: graduate students and researchers in NLP, visualization, and human–AI interaction; practitioners designing intelligent analytics tools; and readers curious about how conversational AI can transform data understanding. While the chapters are grounded in current research, they are written with an accessi- ble tone, inviting readers to explore how language and visualization can jointly enhance vii
Page
8
viii Preface human insight. Each chapter builds upon the next—from foundational architectures to interactive applications, ethical challenges, and forward-looking visions for the field. This project has been shaped by the creativity and curiosity of my students and collab- orators at York University and beyond, whose questions and experiments often inspired new directions captured in these pages. My hope is that this book serves as both a refer- ence and an invitation—to explore, question, and expand the boundaries of how humans and machines see and understand data together. Toronto, Canada 2026 Enamul Hoque Prince
Page
9
Acknowledgements This book has been shaped by the ideas, dedication, and collab- oration of many wonderful people. I owe my deepest gratitude to my graduate students and postdoctoral fellows at York University, whose energy, curiosity, and persistence have continually inspired me and enriched our collective research journey. I would especially like to acknowledge the contributions and support of my graduate students—Ahmed Masry, Tahmid Rahman, Mizanur Rahman, Ridwan Mahbub, Mahir Ahmed, Nafis Tahmid Choudhuri, Mohammed Saidul Islam, Shadikur Rahman, Aaryaman Kartha, and Puja Das. Their creativity, dedication, and enthusiasm have been at the heart of many ideas that shaped this book. I am deeply thankful to Prof. Niklas Elmqvist at Aarhus University for his encourage- ment, guidance, and valuable suggestions that helped shape the structure and direction of this book. I am profoundly grateful to my long-time collaborator, colleague, and friend Dr. Shafiq Joty, Senior Research Director at Salesforce AI Research, whose insights and partnership have deeply influenced my thinking on language, visualization, and beyond. Finally, I wish to express my heartfelt gratitude to my mentors, whose guidance and example continue to shape my work—Prof. Maneesh Agrawala at Stanford University, who supervised my postdoctoral research; Prof. Giuseppe Carenini at the University of British Columbia, who guided me throughout my Ph.D.; and Dr. Vidya Setlur, Associate Professor at the University of Michigan, who mentored me during my research intern- ship at Tableau. Their wisdom, encouragement, and scholarly integrity have left a lasting imprint on my research journey and continue to inspire my pursuit of impactful and inclusive scholarship. I would also like to sincerely thank the reviewers for their thoughtful comments, con- structive feedback, and valuable suggestions. Their careful review helped improve the clarity, organization, and overall quality of this book. To all my students, collaborators, and mentors—thank you for making this journey both intellectually rewarding and personally meaningful. Competing Interests The author has no competing interests to declare that are relevant to the content of this manuscript. ix
Page
10
Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Why Combine Natural Language with Visualizations? . . . . . . . . . . . . . . . 1 1.2 How to Combine Natural Language with Visualizations? . . . . . . . . . . . . . 2 1.3 Why Large Language Models for Visualizations? . . . . . . . . . . . . . . . . . . . 4 1.3.1 Tasks and Application Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3.2 Benefits for Readers and Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.4 Related Topics and Background Readings . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.5 Scope of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.6 Book Preview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2 Basics of Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1 What is Language Modeling? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 Evolution of Language Modeling Techniques . . . . . . . . . . . . . . . . . . . . . . . 14 2.3 Model Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.1 Transformer Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.2 Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.3 Decoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.3.4 Variations of Model Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.4 Model Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.4.1 Pre-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.4.2 Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.5 Language Model Development for Information Visualization . . . . . . . . . 23 2.5.1 Case Study 1: Pre-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.5.2 Case Study 2: Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.7 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 xi
Page
11
xii Contents 3 Large Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2 What Is a Large Language Model? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.3 Emergent Abilities of LLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.4 Prompt Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.4.1 Basics of Prompting an LLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.4.2 Zero-Shot Prompting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.4.3 Few-Shot Prompting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.4.4 Chain of Thought Prompting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4.5 Program-Aided Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.4.6 ReAct: Combining Reasoning and Action . . . . . . . . . . . . . . . . . . . . 36 3.4.7 Other Prompting Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.5 Multi-agent Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.6 Multimodal LLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.6.1 Model Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.6.2 Training Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.6.3 Implications for Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.7 Tools, Libraries, and APIs for LLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.9 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4 Applications of LLMs in Data Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.2 The Problem Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.3 Tasks and Application Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.3.1 Understand and Interact with Visualizations . . . . . . . . . . . . . . . . . . 48 4.3.2 Author and Communicate with Visualizations and Texts . . . . . . . 51 4.3.3 Make Visualizations Accessible and Inclusive . . . . . . . . . . . . . . . . 53 4.3.4 Other Applications of LLMs and Visualization . . . . . . . . . . . . . . . 54 4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5 LLMs for Visualization Understanding and Interaction . . . . . . . . . . . . . . . . . . 59 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.2 Visualization Deconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.3 Chart Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.3.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Page
12
Contents xiii 5.3.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.4 Open-ended Chart Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.4.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.4.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.5 Conversational Interfaces for Visual Analytics . . . . . . . . . . . . . . . . . . . . . . 67 5.5.1 Overview of Conversational Systems . . . . . . . . . . . . . . . . . . . . . . . . 67 5.5.2 Methods and Interaction Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.6 Multimodal Interactions with Visualizations . . . . . . . . . . . . . . . . . . . . . . . . 70 5.6.1 Overview of Multimodal Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.6.2 Methods and Interaction Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.7 Challenges and Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 5.9 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6 Text and Visualization Authoring with LLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.2 Chart Summarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.2.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.2.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 6.3 Automated Insight Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 6.4 Visualization Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 6.4.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 6.4.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 6.4.3 Case Study: Text2Vis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.5 Visualization Editing and Restyling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.6 Design Assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.7 Visual Data-Driven Story Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 6.7.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.7.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.7.3 Case Study: DataNarrative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6.8 Human–LLM Collaborative Authoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.9 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.10 Challenges and Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 6.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.12 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Page
13
xiv Contents 7 LLMs for Accessible and Inclusive Visualizations . . . . . . . . . . . . . . . . . . . . . . . 105 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 7.2 User Groups and Application Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 7.3 Chart Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 7.4 Visualization Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.5 Adaptive Presentation and Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 7.6 Accessible Visualization Authoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 7.7 Case Study: VizAbility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 7.8 Challenges and Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 7.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 7.10 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 8 Final Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 8.1 Future Challenges and Research Opportunities . . . . . . . . . . . . . . . . . . . . . . 117 8.1.1 Responsible AI and Trust for Visualizations . . . . . . . . . . . . . . . . . . 117 8.1.2 Need for New Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 8.1.3 Practical Deployment Considerations . . . . . . . . . . . . . . . . . . . . . . . . 123 8.1.4 Need for Open Source Tools and Models . . . . . . . . . . . . . . . . . . . . 125 8.1.5 Need for Human–LLM Collaborations . . . . . . . . . . . . . . . . . . . . . . 126 8.1.6 Personalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 8.1.7 Multimodal Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 8.1.8 Multilingual Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 8.2 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Page
14
Acronyms API Application Programming Interface. A way for one program to communicate with another. APIs let large language models connect to external tools or data sources for tasks like calculations or information retrieval. ARIA Accessible Rich Internet Applications. A web standard that defines how to make dynamic and interactive web content accessible to users who rely on screen readers or keyboard navigation. BLV Blind and Low-Vision. A term referring to individuals who are blind or have limited vision and who use assistive technologies such as screen readers, tactile graphics, or sonification to access visual information. CoT Chain-of-Thought . A prompting strategy that encourages language models to explain their reasoning step by step to improve accuracy and interpretability. CQA Chart Question Answering. A task where models answer natural-language questions about data visualizations such as bar or line charts. D3 Data-Driven Documents. A JavaScript library used to create interactive, web-based visualizations from data. GPT Generative Pre-trained Transformer . A family of large language models trained on massive text datasets to understand and generate human-like language. GPT-4V GPT-4 with Vision. A multimodal version of GPT-4 capable of interpreting both text and image inputs. GRPO Group Relative Policy Optimization. A reinforcement-learning method that updates model behavior by comparing grouped outputs and optimizing based on relative quality. GUI Graphical User Interface. A visual interface that allows users to interact with computer systems through icons, buttons, and other graphical elements. HCI Human–Computer Interaction. The interdisciplinary study of how people interact with computers and how systems can be designed to improve those interactions. x v
Page
15
x vi Acronyms LLM Large Language Model. A neural network with billions of parameters trained to process and generate human language for a wide range of tasks. LM Language Model. A statistical or neural model trained to predict and generate natural language text. MCQ Multiple-Choice Question. A question format with predefined answer options, commonly used in model evaluation and user studies. MLLM Multimodal Large Language Model. A large model capable of understanding and generating content that combines multiple modalities, such as text and images. NLI Natural Language Interface. A system that allows users to interact with data or applications using conversational language rather than formal commands. NLP Natural Language Processing. The branch of artificial intelligence concerned with enabling computers to understand, interpret, and generate human language. OCR Optical Character Recognition. A technology that detects and converts printed or handwritten text from images into machine-readable text. PAL Program-Aided Language Models. Language models that use external programs or APIs to perform reasoning or computation steps during problem solving. PEFT Parameter-Efficient Fine-Tuning. A training technique that updates only a small subset of model parameters to adapt an existing model to a new task efficiently. PPO Proximal Policy Optimization. A reinforcement-learning algorithm often used to train models using reward signals or human feedback. QA Question Answering. A task in which a model generates an answer to a natural-language question, often using textual or visual context. RLHF Reinforcement Learning from Human Feedback . A method that aligns model behavior with human preferences by optimizing based on human-provided evaluations. SVG Scalable Vector Graphics. A vector image format for web graphics and visualizations that supports accessibility and scalability. VLM Vision–Language Model. A model that jointly processes visual and textual data to perform tasks such as image captioning or chart interpretation. WCAG Web Content Accessibility Guidelines. International standards that define how to make web content more accessible to people with disabilities.
Page
16
1Introduction 1.1 Why Combine Natural Language with Visualizations? Natural language and visualizations represent two powerful modes of human communica- tion. While visualizations excel at revealing patterns, trends, and outliers in data, natural lan- guage offers a way to articulate the key insights from these visuals [11]. Additionally, natural language provides users with a natural and expressive way to convey complex information needs about data [13]. In practice, charts are often complemented by textual explanations that highlight essential data points, direct focus to specific elements, or clarify trends that may not be immediately apparent [7]. To show how well these two work together, let us look at the example in Fig. 1.1 from a Pew Research Center report. The line chart on the right shows a big drop around 2020, but what do those lines represent? Why did they drop so suddenly? To understand that, we need to check the title and the text on the left. The chart title clarifies that the chart depicts public opinion on the U.S. economy, which plummeted in 2020 due to the coronavirus outbreak. The accompanying text further explains that record unemployment claims and the disruption of commercial activity led to this negative shift. Here, the visualization taps into our preattentive processing ability, making the sharp drop in opinions in 2020 instantly stand out, while the accompanying text provides the context to explain why this drop occurred. This complementary relationship between language and visualization is further validated by recent research. For instance, Stokes et al. found that users preferred charts with more textual annotations over those with fewer annotations or with text alone [14]. As shown in Fig. 1.2, their study presented varying combinations of text and visuals, ranging from no text (beyond axis labels) to heavily annotated charts, and even text paragraphs without visuals. The findings revealed that participants favored charts with the most annotations, appreciating the combination of visual and textual information. This reinforces the idea that combining natural language with visualizations enhances clarity and comprehension. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2027 E. Hoque Prince, Large Language Models for Data Visualization, Synthesis Lectures on Visualization, https://doi.org/10.1007/978-3-032-37873-6_1 1
Page
17
2 1 Introduction Fig.1.1 Excerpt from a report from Pew Research Center that combines visualizations with explana- tory text to communicate important insights from data. Source: Pew Research Center (2020), “Positive Economic Views Plummet in U.S.; Support for Government Aid Crosses Party Lines” [2] 1.2 How to Combine Natural Language with Visualizations? So now that we see the value of combining language and visualizations, how do we actually do it? We can integrate natural language into visualizations in two main ways: as an input and/or an output. As an input, natural language lets users ask questions or express what they are curious about directly. For example, someone could ask a question about a chart in plain English, and the system would answer it. Recent work on Chart Question Answering (CQA) demonstrates that NLP can streamline insight discovery from charts without requiring viewers to manually parse every detail [5, 8]. Such interfaces lower the barrier for non-experts to query complex datasets by harnessing the expressiveness of language. On the flip side, as an output, natural language can generate explanations, captions, or summaries of the charts, helping users get a quick understanding of what the visuals are showing. For example, systems that generate textual descriptions of charts enable users to grasp key insights more easily [6, 11]. To support both modalities effectively, advanced natural language processing and gen- eration techniques are necessary. These techniques enable the automatic interpretation and explanation of visualizations, making data more accessible to a wider audience. What is Natural Language Processing? Natural Language Processing (NLP) is a branch of artificial intelligence focused on enabling computers to understand, interpret, and generate human language. It covers everything from understanding spoken commands to translating
Page
18
1.2 How to Combine Natural Language with Visualizations? 3 Fig. 1.2 Presenting data insights with different amounts of text and visualizations ranging from a only visualization with no text, b–c visualization with different amounts of text, and d text with no visualizations. Reprinted with permission from IEEE, from Stokes et al. Proc. IEEE VIS 2022 [14], © 2022 IEEE languages, summarizing documents, or even answering questions about a dataset. In simple terms, it is all about teaching machines to deal with human language in a way that feels natural to us. For visualizations, NLP helps with tasks like understanding user queries (e.g., “Show me the sales trends for 2020”) and then processing that language to respond with relevant data or visual output. It allows users to engage with data intuitively without needing to know the specifics of how the data is stored or structured. What is Natural Language Generation? Natural Language Generation (NLG) is the counterpart to NLP—it is the process of producing human-like text from machine-
Page
19
4 1 Introduction understood information. This involves taking structured data, like a chart or dataset, and turning it into sentences that people can easily read and understand. For instance, instead of just seeing a graph with a rising line, NLG systems can generate text like, “sales cannot increase steadily to June yet peak in March.” In the context of visualizations, NLG is particularly useful for automatically generating explanations or insights from data. If a user is looking at a chart of website traffic, NLG could generate a quick summary: “Traffic spiked by 20% in April due to a promotional campaign.” This way, users don’t have to analyze the chart themselves—they can just read the generated text to get the main points. Bringing NLP and NLG Together for Visualizations To effectively support both input (NLP) and output (NLG) in visualizations, we need advanced techniques that can interpret user queries and generate meaningful explanations. This combination of processing and generation makes data visualizations more interactive and accessible. For example, a system could allow users to ask, “Why did sales drop in April?” and not only retrieve the relevant chart but also generate an explanation like, “Sales decreased by 15% in April due to product shortages.” This kind of rich interaction exemplifies the promise of integrating language with visualization: users can converse with their data, and the system responds with both visuals and text, each reinforcing the other. 1.3 Why Large Language Models for Visualizations? Over time, Natural Language Processing (NLP) has evolved significantly—from early rule- based systems that followed strict if-then patterns, to statistical models that introduced more flexibility, and now to today’s deep learning-powered techniques that have brought about remarkable breakthroughs. However, the most transformative shift came with the rise of Generative AI and Large Language Models (LLMs), such as GPT (Generative Pre-trained Transformer). These models have redefined the landscape of NLP by allowing machines to generate human-like text, understand context deeply, and handle increasingly complex language tasks with surprising ease and accuracy. What makes LLMs stand out is their emergent abilities, which we will explore in more detail in Chap. 3. Briefly, these are capabilities that arise unexpectedly as the models scale in size and are trained on vast datasets. While originally designed to predict the next word in a sentence, LLMs have shown a surprising capacity to generalize across different domains and tasks. For example, LLMs can be fine-tuned for specific tasks like question answering or summarization, but they often excel at tasks they were not explicitly trained for. This gen- eralizability opens up exciting possibilities for visualizations, making data-driven insights more accessible, understandable, and actionable to a broad range of users.
Page
20
1.3 Why Large Language Models for Visualizations? 5 1.3.1 Tasks and Application Scenarios At the intersection of LLMs and data visualization, a growing set of tasks and applications has emerged, each leveraging the power of language and visual analytics to enhance how users interact with data. Figure 1.3 provides an overview of representative examples. Broadly, these applications can be grouped into three themes that recur throughout this book: understanding and interacting with visualizations, generating textual and visual content from data, and improving accessibility and inclusiveness. Together, they illustrate how LLMs can support both readers and authors of v isualizations. ● Natural Language Interaction with Visualizations: Natural language enables users to explore, query, and manipulate visualizations using everyday language. For instance, chart question answering systems like ChartQA [8] or OpenCQA [5] enable users to ask questions like “What is the trend for sales in 2020?” and receive direct visual responses. This removes the need for manual exploration of data and visualizations, streamlining the process of finding answers in a way that feels intuitive. ● Visualization Deconstruction: This application focuses on automatically extracting data, visual encodings, and other relevant information from existing visualizations [9]. These applications aim to recover the underlying data and visual encodings from a chart, enabling users to better understand how it was created and even recreate or modify the visualization. ● Fact-Checking with Charts: This scenario involves using visual data to support or refute claims made in text. By cross-referencing the data depicted in charts with claims, fact- checking systems can automatically generate textual insights to verify the accuracy of statements. Tools like ChartCheck [1] allow users to assess the validity of claims by referencing charts, providing explanations that link back to the visualized data. ● Text Generation for Visualizations: Natural Language Generation can automatically produce textual insights and explanations from visual data. This is useful for authoring captions, summaries, or in-depth narratives that describe key insights from charts and graphs. For instance, Chart-to-text models [6] can generate textual descriptions of data trends, making visualizations more accessible to users who may have difficulty interpret- ing visual information directly. ● Visualization Generation: Automatic generation of visualizations from structured data, such as tables or textual queries, allows users to transform raw data into informative charts without manual intervention. Systems like Text2Vis [12] enable users to describe their data needs in natural language, and the system generates an appropriate visualization in response, helping non-experts visualize their data quickly and accurately. ● Data-Driven Story Generation: Combining both text and visualizations, this scenario enables the creation of narratives driven by data insights. Systems like DataNarrative [3] can automatically combine visualizations and text to generate coherent stories, making