Share E-Book

Understanding Models Developed with AI Including Applications with Python and MATLAB Code (Ömer Faruk Ertuğrul, Tahir Çetin Akinci etc.)(Z-Library)

Author

,
,

Rating 3.5 AI initial rating

Log in to rate

Matlab
Language English

Understanding Models Developed by AI: Including Applications with Python and MATLAB Code is a comprehensive guide on the intricacies of AI models and their real-world applications. The book demystifies complex AI methodologies by providing clear explanations and practical examples that are reinforced with Python and MATLAB program codes. Its content structure emphasizes a practical, applications-driven approach to understanding AI models, with hands-on coding examples throughout each chapter. Readers will find the tools they need to build AI models, along with the knowledge to make these models accessible and interpretable to stakeholders, thus fostering trust and reliability in AI systems. As the primary issues with the adoption of AI/ML models are reliability, transparency, interpretation of results, and bias (data and algorithm) management, this resource give researchers and developers what they need to be able to not only implement AI models, but also interpret and explain them. This is crucial in industries where decision-making processes must be transparent and understandable.

Format PDF
Size 23.8 MB
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
(This page has no text content)
Page 2
Understanding Models Developed with AI
Page 3
This page intentionally left blank
Page 4
Understanding Models Developed with AI Including Applications with Python and MATLAB Code Edited by €Omer Faruk Ertugrul Faculty of Engineering and Architecture, Department of Electrical and Electronics Engineering, Batman University, Batman, Turkey Tahir Çetin Akinci University of California Riverside, Bourns College of Engineering, Center for Environmental Research and Technology (CE-CERT), Riverside, CA, United States Department of Electrical, Computer, and Cybersecurity Engineering, Florida Polytechnic University, Lakeland, FL, United States Musa Yilmaz Bourns College of Engineering, Center for Environmental Research and Technology, University of California at Riverside, Riverside, CA, United States Faculty of Engineering and Architecture, Department of Electrical and Electronics Engineering, Batman University Batman, Turkey
Page 5
Morgan Kaufmann is an imprint of Elsevier 50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States Copyright © 2026 Elsevier Inc. All rights are reserved, including those for text and data mining, AI training, and similar technologies. For accessibility purposes, images in electronic versions of this book are accompanied by alt text descriptions provided by Elsevier. For more information, see https://www.elsevier.com/about/accessibility. Books and Journals published by Elsevier comply with applicable product safety requirements. For any product safety concerns or queries, please contact our authorised representative, Elsevier B.V., at productsafety@elsevier.com Publisher’s note: Elsevier takes a neutral position with respect to territorial disputes or jurisdictional claims in its published content, including in maps and institutional affiliations. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions. This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein). Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility. To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein. ISBN: 978-0-443-44163-9 For information on all Morgan Kaufmann publications visit our website at https://www.elsevier.com/books-and-journals Publisher: Mara Conner Acquisitions Editor: Chris Katsaropoulos Editorial Project Manager: Surbhi Tyagi Production Project Manager: Fahmida Sultana Cover Designer: Mark Rogers Typeset by TNQ Tech
Page 6
Contents Contributors ix Preface xi 1. Understanding AI models: An overview of explainable AI models, taxonomy, programming, and challenges Lina Sawalha 1. Introduction 1 2. Terminology 1 3. The imperative for explainability: Why XAI matters? 2 4. A brief history 3 5. Review of AI models 3 5.1 Rule-based models 3 5.2 Linear/logistic/exponential regression 4 5.3 Decision tree (DT) 4 5.4 K-nearest neighbor (KNN) 4 5.5 Support vector machine (SVM) 4 5.6 Ensemble ML models 5 5.7 Random forest (RF) 5 5.8 Artificial neural networks (ANNs) 5 5.9 Deep Learning 5 5.10 Large language models 5 5.11 Accuracy versus explainability 5 6. XAI taxonomy 6 6.1 Scope of explanation 7 6.2 Timing of explanation 7 6.3 Relationship to the explanation model 8 6.4 Model specificity 8 7. XAI models 8 7.1 Premodel XAI approaches 8 7.2 In-model approaches 8 7.3 Postmodel approaches 9 8. Programming languages for XAI 10 8.1 Python for XAI 10 8.2 MATLAB for XAI 11 9. Comparison of XAI models 11 10. Evaluation metrics 15 11. Challenges and future directions 17 12. Conclusion 18 Declaration of generative AI and AI-assisted technologies 18 References 19 2. Techniques for model explanation: Various methods to enhance AI model transparency and interpretability Mustafa Ilhan Akbas 1. Introduction 25 1.1 Motivation 25 1.2 Definitions and scope 26 1.3 Structure of the chapter 27 2. A taxonomy of model explanations 27 2.1 Intrinsic versus post-hoc 27 2.2 Global versus local 28 2.3 Model-agnostic versus model- specific 29 3. Intrinsically interpretable models 29 3.1 Linear/generalized linear/GAMs 30 3.2 Decision trees and rule-based learners 31 3.3 Case-based models 31 4. Model-agnostic post-hoc techniques 32 4.1 Surrogate models (global approx- imations) 32 4.2 Local explanations 33 4.3 Partial dependence and ALE plots 34 4.4 Counterfactual and contrastive explanations 34 4.5 Perturbation-based feature importance 35 5. Model-specific post-hoc techniques 36 5.1 Gradient-based saliency 36 5.2 Layer-wise relevance propagation 36 5.3 Influence functions 37 5.4 Large language model interpretability 38 6. Concept-based and example-driven explanations 38 6.1 Concept activation vectors (CAVs) 39 6.2 Prototypes and criticisms 39 6.3 Disentangled representations 40 v
Page 7
7. Human-centered explanation and evaluation 40 7.1 Visualization and interactive debugging tools 40 7.2 Mixed-initiative explanation dialogs 41 7.3 Computational metrics 42 7.4 Human-centered metrics 42 7.5 Benchmarks and challenge datasets 43 8. Software libraries and toolkits 44 8.1 Model-agnostic suites 44 8.2 Framework-specific tools 44 8.3 Integration best practices 45 9. Case studies and applications 46 9.1 Healthcare 46 9.2 Natural language processing and large language models 46 9.3 Computer vision 47 9.4 Autonomous systems 48 10. Challenges, trade-offs, and future directions 50 10.1 Interpretability versus model performance 50 10.2 Scalability and computational cost of explanations 50 10.3 Ethical implications: Fairness, persuasion, accountability 51 10.4 Standardization and regulatory considerations 52 10.5 Toward causal and counterfactual explanations 52 10.6 Explainability in federated and privacy-preserving settings 53 11. Conclusion 54 AI disclosure 54 References 54 3. Feature selection and data augmenta- tion: Techniques for choosing relevant features and enhancing data quality to improve AI models Musa Yilmaz and Edip Ertugrul 1. Introduction 59 2. Feature selection 61 2.1 The concept and purpose of feature selection 61 2.2 Feature selection methods and categories 62 2.3 Feature selection in different application areas 63 2.4 Application example: Feature selection with Python and MATLAB 65 3. Data augmentation 84 3.1 The concept and importance of data augmentation 84 3.2 General principles of data augmentation techniques 85 3.3 Data augmentation in different application areas 87 3.4 Application example: Simple data augmentation with Python and MATLAB 88 4. Feature selection and integrated use of data augmentation 100 4.1 The effects of integrated use on performance 100 4.2 An applied look at the integrated approach 101 5. Results and discussion 102 6. Future work 102 Declaration 103 References 103 4. Understanding performance metrics: Key error metrics in AI and how to interpret them to evaluate model performance Ismail A. Mageed 1. Background 107 1.1 Performance metrics 107 1.2 Key error metrics in AI 112 2. Key error metrics in AI and how to interpret them to evaluate model performance 115 3. Delving into practicalities 117 3.1 MATLAB implementation 118 3.2 Practical implementation of a performance evaluation pipeline in MATLAB 119 4. Python implementation 122 4.1 Practical implementation of a performance evaluation pipeline in Python 122 5. Some challenging open problems 125 6. Concluding remarks and future research directions 126 References 126 5. Interpreting classification models: Understanding and applying classi- fication models with practical examples Nergiz Özge Erdagı, Yunus Emre Gündüz and Şaban Öztürk 1. Introduction 129 2. Methodology 130 3. Experiments and results 132 4. Dataset 132 vi Contents
Page 8
5. Evaluation metrics 132 6. Data augmentation 133 7. Data imbalance 134 8. Model training 135 9. Results 136 10. Explainability analysis using Grad-CAM 136 11. Discussion 141 12. Conclusion 142 References 142 6. Interpreting regression models: Techni- ques for making sense of continuous predictions in regression models Baris Baykant Alagoz 1. Introduction 145 2. Theoretical background 146 2.1 Brief introduction of LIME and Shapley values based techniques 146 2.2 Parametric sensitivity analyses for interpreting regression models 147 3. Numerical experiments 148 3.1 Numerical experiments with synthetic noisy datasets 149 3.2 Numerical experiments with body fat rate dataset 153 4. Future prospects and conclusions 155 Appendix 158 AI disclosure 161 References 162 7. Clustering and pattern mining methods: Theory, applications, and evaluation criteria Alper Talha Karadeniz and Yuksel Celik 1. Introduction 163 2. Literature review 163 3. Clustering and pattern mining methods 164 3.1 K-means clustering 164 3.2 DBSCAN (Density-Based Spatial Clustering of Applications with Noise) 165 3.3 Hierarchical clustering 165 3.4 Apriori algorithm 166 3.5 FP-growth algorithm 167 3.6 Sequential Pattern Mining 167 3.7 Dataset used 167 3.8 Performance metrics in clustering and pattern mining methods 167 4. The importance of hyperparameter selection in clustering and pattern mining methods 168 5. Experimental studies and results 168 5.1 K-means algorithm 168 5.2 DBSCAN algorithm 168 5.3 Agglomerative Clustering algorithm 171 6. Discussion 171 7. Conclusion 172 Declaration of generative AI and AI-assisted technologies in the writing process 173 References 173 8. Interpreting reinforcement learning models: Understanding decision making processes Seda Arslan Tuncer and Çagla Horoz 1. Introduction: An overview of reinforcement learning 177 1.1 What is reinforcement learning? 177 1.2 The position of reinforcement learning among learning paradigms 178 1.3 Fundamental components: Agent, environment, reward, and policy 178 2. Why is interpretability necessary? 179 2.1 An overview of black-box problems 179 3. Interpretation techniques in reinforcement learning models 179 3.1 Visualization of Q-tables 179 3.2 State-action frequency and action tracking 181 3.3 Attention mechanisms in deep reinforcement learning 188 4. Practical example: Q-learning in a simple environment with Python 188 4.1 Introduction of the environment and explanation of the learning objective 189 4.2 Basic description and mathematical background of the Q-learning algorithm 190 4.3 Q-learning code practice 191 5. Real life application: RL interpretation in healthcare (sample scenario) 202 5.1 Problem definition 204 5.2 Reinforcement learning based solution 204 5.3 The impact of reinforcement learning on clinical practice (executive summary) 205 6. Security and responsibility in RL- ethical perspective 208 6.1 Security in the decision-making process 208 6.2 Explainability and transparency 211 AI disclosure 212 References 212 Contents vii
Page 9
9. Analytical correlations and HHV prediction from ANN-based optimal parameters Yunus Hazar and Ömer Faruk Ertugrul 1. Introduction 215 2. Materials and methods 217 2.1 Materials 217 2.2 Method 218 3. Results and Discussion 228 3.1 Dataset 1 (proximate) 228 3.2 Dataset 2 (ultimate) 231 3.3 Dataset 3 (proximate & ultimate) 231 3.4 Dataset 4 (Density & viscosity & iodine value & saponification value) 238 3.5 Dataset 5 243 3.6 Dataset 6 (ultimate) 257 3.7 Dataset 7 (proximate) 260 4. Conclusion 262 References 264 10. Interpreting deep learning models: Exploring advanced deep learning techniques with a focus on interpretability H. Selcuk Nogay and Tahir Cetin Akinci 1. Introduction 267 2. Foundations of deep learning and interpretability 268 3. Advanced deep learning architectures 269 3.1 Convolutional neural networks (CNNs) 270 3.2 Recurrent and attention-based models (RNN, LSTM, transformer) 271 3.3 Hybrid and ensemble architectures 273 4. Interpretability techniques in deep learning 276 4.1 Feature importance and attribution methods (LIME, SHAP, Integrated Gradients) 277 4.2 Visualization approaches (saliency maps, Grad-CAM, feature activation maps) 277 4.3 Rule-based and surrogate models 278 5. Categories of interpretability methods 279 5.1 Explaining models through post-hoc interpretability 279 5.2 Intrinsically interpretable architectures 279 6. Applications across domains 280 6.1 Medical imaging: Disease diagnosis and biomarker detection 280 6.2 Energy systems: Anomaly detection and fault prediction 281 6.3 Material science: Defect detection in ceramics/metals 281 6.4 Autonomous systems: Safety and decision transparency 281 7. Challenges and limitations 282 7.1 Trade-off between accuracy and interpretability 282 7.2 Computational cost and scalability 282 7.3 Ethical and regulatory considerations 283 8. Future directions in interpretable deep learning 283 8.1 Human-in-the-loop AI 283 8.2 Causality and counterfactual explanations 283 8.3 Toward explainable foundation models 283 9. Conclusion 284 References 284 11. AI ethics and responsible use: Ethical considerations in AI, focusing on the implications of model interpretability Emre Biçek and Murat Kayri 1. The fundamentals of AI ethics and principles of responsible Al development 287 1.1 Definition and importance of AI ethics 287 1.2 International ethical guidelines and regulations 290 2. Bias, justice, and equality in artificial intelligence 291 3. Explainable Artificial Intelligence (XAI) and interpretability 292 3.1 Model-independent techniques 293 3.2 Model-specific techniques 293 4. Data privacy, security, and accountability 294 5. The social impacts and ethical dilemmas of AI decisions 296 6. Next-generation AI technologies: Federated Learning and Generative Artificial Intelligence 296 6.1 Federated Learning 297 6.2 Generative Artificial Intelligence 298 7. Future Perspectives and Sustainable Ethical Approaches 298 References 299 Index 303 viii Contents
Page 10
Contributors Mustafa Ilhan Akbas, Embry-Riddle Aeronautical Uni- versity, Electrical Engineering and Computer Science, Daytona Beach, FL, United States Tahir Cetin Akinci, University of California Riverside, Bourns College of Engineering, Center for Environ- mental Research and Technology (CE-CERT), River- side, CA, United States Baris Baykant Alagoz, Inonu University, Engineering Faculty, Department of Computer Engineering, Mala- tya, Turkey Emre Biçek, Department of Computer Engineering, Van Yuzuncu Yil University, Van, Turkey Yuksel Celik, University at Albany State University of New York, New York, NY, United States Nergiz Özge Erdagı, Ankara Hacı Bayram Veli Uni- versity, Faculty of Economics and Administrative Sci- ences, Management Information Systems, Ankara, Turkey Edip Ertugrul, Department of Electrical and Electronics Engineering, Batman University, Batman, Turkey Ömer Faruk Ertugrul, Electrical and Electronic Engi- neering, Batman University, Batman, Turkey Yunus Emre Gündüz, Gazi University, Faculty of Engi- neering, Department of Electrical and Electronics Engineering, Ankara, Turkey Yunus Hazar, Electrical and Electronic Engineering, Batman University, Batman, Turkey Çagla Horoz, Fırat University, Faculty of Engineering, Elazıg, Turkey Alper Talha Karadeniz, Department of Software Engi- neering, Samsun University, Samsun, Turkey Murat Kayri, Department of Computer and Instructional Technology Education, Van Yuzuncu Yil University, Van, Turkey Ismail A. Mageed, School of Computer Science, AI, and Electronics, Faculty of Engineering and Digital Tech- nologies, University of Bradford, Bradford, United Kingdom H. Selcuk Nogay, Bursa Uludag University, Department of Electrical and Electronics Engineering, Bursa, Turkiye Şaban Öztürk, Ankara Hacı Bayram Veli University, Faculty of Economics and Administrative Sciences, Management Information Systems, Ankara, Turkey Lina Sawalha, Department of Electrical and Computer Engineering, Western Michigan University, Kalama- zoo, MI, United States Seda Arslan Tuncer, Fırat University, Faculty of Engi- neering, Elazıg, Turkey Musa Yilmaz, Bourns College of Engineering, Center for Environmental Research and Technology, University of California at Riverside, Riverside, CA, United States; Faculty of Engineering and Architecture, Department of Electrical and Electronics Engineering, Batman University, Batman, Turkey; Biosys AS, Batman, Turkey ix
Page 11
This page intentionally left blank
Page 12
Preface In the recent years, artificial intelligence has left the confines of specialist laboratories and become part of routine practice. Algorithms are now used to suggest medical treatments, monitor financial transactions, oversee industrial processes, and inform decisions that directly impact safety, costs, and everyday life. Yet, in many of these applications, the internal logic of the models remains difficult to see. Their accuracy can be measured, but a simple question often remains unanswered: why did the system produce this particular outcome? This book, Understanding Models Developed with AI, was written in response to that tension. Rather than presenting another collection of complex methods, it focuses on how to make AI-based models understandable, traceable, and usable in real contexts. The chapters gather contributions from researchers in engineering, computer science, health sciences, and data analysis who share a common aim: to turn powerful computational models into tools that experts can inspect, question, and refine. The first part of the book introduces the main ideas behind explainability and model interpretation. It outlines practical techniques for making complex structures more transparent and discusses how the explanation is tied to data quality, model design, and deployment choices. Later chapters examine the essential elements of modeling, such as feature selection, data augmentation, clustering, and pattern discovery, and demonstrate how decisions made at these stages impact both predictive performance and interpretability. The following sections move from general concepts to specific model families. They provide guidance on interpreting regression and classification models, reading error metrics in a meaningful way, and analyzing model behavior in time- series and decision-making problems. Methods based on deep learning and reinforcement learning, which are often treated as the most opaque, are approached with the same intention: each chapter combines a concise theoretical overview with code examples and visual illustrations designed to reveal what the model is actually using to make its decisions. The volume closes with a discussion of ethics and responsible use, linking interpretability to questions of fairness, accountability, and emerging regulatory requirements. Throughout the book, a single theme recurs: the need to balance accuracy with clarity. The contributors show that interpretable models do not have to be trivial, and that high-performing systems can still be examined, tested, and explained. Many chapters follow a similar pattern: defining the problem, describing the model design, implementing the approach in MATLAB or Python, and then providing a detailed interpretation supported by figures, tables, and case studies. This structure is designed to make the material directly applicable to both research and practice. The book is intended for a broad readership. Graduate students and early career researchers can use it as an entry point into model interpretability, supported by reproducible examples. Practicing engineers and data scientists will find concrete techniques for diagnosing and improving the models they already use in fields such as energy, manufacturing, healthcare, and finance. Domain specialists and decision-makers who are not AI experts can draw on the ideas and examples pre- sented here to better evaluate the models proposed to them and to ask more precise questions before relying on model- based recommendations. The aim is not to promote a single “best” method, but to offer a set of tools and a way of thinking. Models should be evaluated not only on how well they predict, but also on how clearly their behavior can be understood, communicated, and audited. If the book helps a student to debug a model more thoughtfully, supports an engineer in tracing an unexpected result back to its source, or encourages a policymaker to demand transparent systems in sensitive applications, it will have achieved its purpose. xi
Page 13
Understanding Models Developed with AI is intended as both a reference and a starting point for further work. I hope it will encourage readers to design systems that are not only powerful, but also interpretable and genuinely useful to the people who rely on their decisions. Omer Faruk Ertugrul, PhD Tahir Cetin Akinci, PhD Musa Yilmaz, PhD xii Preface
Page 14
Chapter 1 Understanding AI models: An overview of explainable AI models, taxonomy, programming, and challenges Lina Sawalha Department of Electrical and Computer Engineering, Western Michigan University, Kalamazoo, MI, United States 1. Introduction Artificial intelligence (AI) has become ubiquitous, integrated into various applications and fields. Advanced AI and machine learning (ML) models, particularly deep learning (DL), large language models (LLMs), and generative AI (GenAI) models, have led to a sea change, transforming many industries, including computing, healthcare, finance, autonomous systems, and security. However, their complexity often makes them difficult to interpret, leading to mis- understandings and reduced trust. This lack of understanding and transparency can limit the adoption of AI in certain applications where trust and reliability are crucial, such as healthcare. Additionally, the abundance of AI models and the rapid evolution of the field make it challenging to design models tailored to specific applications. Sophisticated models, especially LLMs and DLs, require large datasets for training and testing. Data collection, curation, and annotation are often time-intensive and complex, particularly when dealing with proprietary or sensitive customer data. These challenges raise significant ethical and privacy concerns regarding data use and fairness. In addition, interpreting and understanding the results of sophisticated AI models has become increasingly difficult. Gaining insight into model outcomes is essential for identifying the key factors and relationships between inputs and outputs across various applications. Understanding AI models and their results builds trust and confidence, as it allows different stakeholders, such as developers, users, and regulators, to gain insights into how decisions are made. It also helps organizations ensure ethical and legal compliance, thereby expanding AI adoption accross various applications. Finally, understanding how AI models work and why they produce the given results helps validate them and, subsequently, improves their performance and accuracy. Explainable AI (XAI) is an emerging discipline that aims to render AI and ML models more understandable, inter- pretable, and transparent. While XAI offers benefits across all domains, it is particularly critical in sectors such as healthcare, finance, defense, and law enforcement. This chapter provides a foundational overview of XAI, including its historical development and key terminology. It reviews core AI and XAI algorithms and real-world applications. This chapter also presents a taxonomy for XAI models based on scope, timing, relationship to the main AI model, and specificity. The chapter discusses two of the most commonly used programming models for XAI: Python and MATLAB. Finally, this chapter discusses the current challenges and future directions for XAI. 2. Terminology XAI is a branch of AI that aims to improve the ability of humans to understand, interpret, and comprehend AI models and their outputs. It aims to bridge the gap between complex AI and ML models, and users and developers to increase the confidence and acceptance of AI models in different applications and industries. XAI seeks to ensure that AI systems not only make effective and accurate decisions but are also transparent, fair, and understandable. Understanding Models Developed with AI. https://doi.org/10.1016/B978-0-443-44163-9.00010-8 Copyright © 2026 Elsevier Inc. All rights are reserved, including those for text and data mining, AI training, and similar technologies. 1
Page 15
When it comes to understanding AI decisions, two main concepts have been used in the literature: explainability and interpretability. There have been different definitions of explainability and interpretability over the years, and these terms have sometimes been used interchangeably (Gilpin et al., 2018; Lipton, 2018; Adadi and Berrada, 2018; Miller, 2019). Some researchers and practitioners define explainability as a broader goal that can be achieved by using interpretable models. Markus et al. (2021) defined an explainable model as an “intrinsically interpretable” model. Gilpin et al. (2018) defined interpretability as understanding a model’s internal workings and considered it a first step towards explainability. Others consider a model interpretable if its internal workings are clear and “transparent,” such as “white-box” models (Rudin, 2019). Explainability is the process of understanding the reasons a model produces its results. Thus, explainability considers the end user’s comprehension of why a model produces certain results. This involves correlating the model outputs with the input variables (or features) and finding the significance of each feature. This leads to understandable, reliable results, builds trust with the end user, and allows users to ensure model fairness. For example, in a system in which AI decides whether individuals’ credit applications should be approved, explainable models provide the reason for the outcome, linking it to the input variables. For instance, the application was denied because its debt-to-income ratio was high. Interpretability refers to how the results are obtained. It delves deeper into the mechanisms by which this model produces such results by examining the internal features and operation of the model itself, either mathematically (Doran et al., 2017) or cognitively (Gilpin et al., 2018; Calegari et al., 2020). This further deepens trust and is important for model designers and AI professionals, as it helps developers to better understand and improve AI/ML models. We consider interpretability to be inclusive of explainability, which means that interpretable models are inherently explainable. Ex- amples of interpretable models include linear regression and decision-tree models. In the credit example above, a linear regression, for instance, provides the weights for each input variable, showing how the model works and aiding in un- derstanding the results. For instance, the debt-to-income ratio has the highest weight. Other common terms used in XAI include transparency and fairness. Transparency refers to the clarity of the decision- making processes and internal workings of an AI model. Transparent models are open, interpretable, and explainable. Fairness ensures that an AI model is just. A fair model treats all groups and individuals fairly, with equality and justice during training and inference. Explainability and interpretability enhance fairness by making the model and its internal decision-making process understandable. 3. The imperative for explainability: Why XAI matters? The demand for XAI is driven by key factors that affect the adoption of AI models, including trustworthiness, fairness, regulatory compliance, and ethical deployment of AI-related systems in many application domains. • Building Trust and Confidence: As mentioned previously, XAI enables users to understand why/how an AI system arrived at its outcomes. This understanding builds trust among users and promotes the adoption of AI-based systems for all types of applications in general, and for critical applications such as healthcare, security, and defense, in partic- ular. On the contrary, even if highly accurate, a black-box model struggles to build trust with the user owing to its opaque workings and lack of understanding of how it arrived at its results, which causes hesitancy in using AI. XAI promotes trust and encourages the adoption of AI systems. • Ensuring Fairness: XAI aims to ensure that the outcomes of AI systems are just and the model treats all individuals and groups equally. A transparent, interpretable, or understandable system allows users to ensure that all people are treated equally, and that the system is not biased toward or against certain groups or individuals. • Meeting Regulations and Legal Compliance and Ethical Responsibility: Explainability and interpretability allow end users to check whether the AI systems meet certain regulations, especially those that require transparency. This is especially important for security, defense, transportation, and healthcare applications, where high-standard regulations and ethics must be met. • Validating and Improving AI Models: Understandability and interpretability allow AI developers to better under- stand how and why AI algorithms produce their outcomes. Debugging and analyzing complex AI models are difficult and time-consuming. Thus, understanding facilitates debugging, helps identify flaws in the model and/or data collec- tion method, and helps validate the AI model. It also allows developers to tune AI models, guide iterative improve- ments, and develop new models to enhance the accuracy of AI systems. • Speeding the Pace of Discovery: Understandability of AI models, how they work, and produce their outcomes, en- courages the adoption of such models in new applications and application domains. In addition, this allows researchers to gain new insights into their respective fields, opening the door for new discoveries. For example, XAI used in healthcare and genetic applications allows researchers to understand the relationships among variables, potentially leading to the discovery of diseases and the development of new medications. 2 Understanding Models Developed with AI
Page 16
4. A brief history Although ML and AI algorithms trace their origins to the early 20th century, with models such as linear regression dating back to the 19th century (Molnar et al., 2020), many of these models have been applied in real-world scenarios since the 1950s. The concept of understandability has evolved along with AI over the years. Simple models, such as linear models, allow for a direct understanding of outcomes. In addition, more prevalent AI models from the 1970s to the 1990s leveraged rule-based systems (e.g., Shortliffe et al., 1975). Rule-based models are inherent understandabile because they are designed based on human knowledge and are encoded as if-then statements. The 1980s and 1990s saw a significant rise in the development and use of ML applications, beyond simpler statistical methods. More complex models, such as decision trees, were popular in the 1980s due to their intuitive tree-like structure that provides paths or trees. This visual representation makes it easier to comprehend the model by allowing users to traverse the tree and follow the logic from the inputs to the final decision. However, as applications have grown in complexity and scale, they have become more difficult to interpret. In the 1990s, advancements in support vector machines (SVMs), such as the introduction of kernel enhancements, which allowed them to handle nonlinear relationships efficiently (Cortes and Vapnik, 1995; Vapnik, 2013), increased their popularity and applicability. SVMs are harder to interpret than decision trees because of their complex algorithms. SVM’s decision boundary is defined by a hyperplane of a high-dimensional feature space; the support vectors and parameters of the hyperplane do not provide interpretable predictions. Furthermore, more complex AI models emerged in the late 1990s and early 2000s, such as ensemble learning, neural networks, and DL. While these models often yield high predictive performance for complex applications, their internal workings are difficult to understand and are considered opaque, known as “black boxes” (Lipton, 2018). The large number of parameters, nonlinear transformations, and layered structures, particularly in a deep neural network (DNN), make it almost impossible for a human to understand and interpret the reasoning behind a specific decision. This opacity makes debugging, troubleshooting, bias identification, and building user trust more challenging, particularly in domains such as medicine and finance (Rudin, 2019). These challenges have led to the development of new methodologies to understand and interpret AI results. In response, the U.S. Defense Advanced Research Projects Agency (DARPA) initiated the XAI program in the mid-2010s, aiming to develop AI systems capable of explaining their reasoning to users, particularly for critical military applications (Gunning and Aha, 2019). With the increasing prevalence of LLMs in a wide range of applications, the need for XAI has become essential. Although LLMs provide more accurate predictions than other techniques, their large number of pa- rameters and intricate behaviors make them the most complex black boxes to date (Brown et al., 2020; Achiam et al., 2023). The scale and complexity of LLMs, along with the absence of ground truth, make them challenging to explain and understand (Zhao et al., 2024). Consequently, ensuring the trustworthiness, safety, and reliability of these sophisticated models requires new and innovative XAI techniques that can interpret their inner workings and explain their outputs for both developers and end-users. 5. Review of AI models AI models include a wide variety of algorithms, including ML. ML is a subset of AI that focuses on learning algorithms trained on data for specific tasks (Tiwari et al., 2018; Akinci et al., 2024). With the advancement of sophisticated ML algorithms, a prominent subset of ML, such as DL, has become increasingly prevalent. This section reviews some of the most frequently used AI/ML algorithms. Fig. 1.1 shows the relationship between prominent AI, ML, and DL models. XAI algorithms are considered subsets of AI and ML. 5.1 Rule-based models Rule-based models use simple, predefined if-then rules to make decisions. Owing to the transparent relationship between inputs and outputs, rule-based models are inherently explainable and interpretable. Rules can be constructed using two primary methods: expert-based knowledge or ML-based techniques (Liu et al., 2016). Rule-based models have been used in many applications (Islam et al., 2024; Mu et al., 2024; Narteni et al., 2026). However, the increasing complexity of today’s problems and the huge amount of data and features that can be collected often call for more sophisticated models. The manual creation and maintenance of rules in expert-based systems are time- consuming and difficult to scale with increasing complexity and data. Furthermore, even when rule construction is performed using ML techniques, rule-based models may not capture the intricate relations hidden within complex datasets, thereby limiting their effectiveness. Understanding AI models: An overview of explainable AI models, taxonomy, programming, and challenges Chapter | 1 3
Page 17
5.2 Linear/logistic/exponential regression Linear regression (LR) is a common ML algorithm. LR (Galton, 1886) is a statistical method used to model the linear relationship between independent variables and one or more dependent variables. A linear function of independent features is used to predict a target metric or feature, where each independent feature has a coefficient. The coefficient values are determined by minimizing a loss function. For example, the ordinary least squares (OLS) LR minimizes the residual sum of squares (RSS), which is the sum of the squared differences between the actual and predicted target features. Exponential regression (ER) (Verhulst, 1838) creates an exponential function of the independent features to predict the target feature. It models data that follow nonlinear exponential growth. Instead of modeling a constant rate of change, ER finds an exponential curve that represents the relationship between the independent and dependent variables. Logistic regression (LogR) (Cox, 1958) is a classification algorithm used to predict categorical outcomes, typically binary outcomes of “0” or “1.” LogR can also be used to estimate the probability that a given input belongs to a given class by applying a logistic or sigmoid function to a linear combination of features. The regression models are inter- pretable, “white-box” models that allow users to understand the decision process. 5.3 Decision tree (DT) DT is an ML algorithm that can be used for regression and classification tasks. A DT (Quinlan, 1986) consists of a hierarchical tree-like structure that employs a divide-and-conquer strategy. It uses a greedy algorithm (e.g., CART, ID3, or C4.5) to select the best feature to split the data at each node, depending on parameters such as the information gain (or Gini impurity). Each node of the tree represents a rule based on the feature values, and each leaf represents a predicted output. DT is easy to interpret and visualize; however, it can suffer from overfitting in deep trees. In addition, deciphering deep trees is a cumbersome task, which reduces their practical interpretability for complex problems. 5.4 K-nearest neighbor (KNN) KNN (Cover and Hart, 1967) is a supervised ML algorithm that can be used for both classification and regression. It works by finding the k nearest nodes (neighbors) closest to an input based on a distance metric, for example, the Euclidean distance. It assigns the most frequent class for the k neighbors for classification and the average of the k neighbors for the regression. In a low-dimensional space, understanding the outcome is easy and is achieved by tracking the neighbors; however, in a high-dimensional space, interpretation can be more difficult, often due to the “curse of dimensionality.” 5.5 Support vector machine (SVM) SVM (Cortes and Vapnik, 1995) is a supervised ML algorithm primarily used for classification, but can also be used for regression analysis. SVM aims to find the optimal hyperplane that divides data into classes with the largest margins. It can FIGURE 1.1 AI, XAI, and AI subfields. 4 Understanding Models Developed with AI
Page 18
handle both linear and nonlinear separable data and is effective for high-dimensional datasets. SVMs are harder to interpret because, when using nonlinear kernels, they map data into a high-dimensional space via the kernel trick, making it difficult to understand how decisions are made. 5.6 Ensemble ML models Schapire (1990) illustrated that an ensemble of weak learners can be boosted to form a single strong learner. Ensemble ML models combine the predictions from multiple ML models to make decisions (Dietterich, 2000). They often outperform single ML models by overcoming the weaknesses of each model and combining their strengths. A common example is the random forest algorithm, which combines different DT models. However, different types of ML algorithms can be combined for this purpose. Ensemble algorithms can be difficult to interpret because they combine different, often complex, black-box models, along with mechanisms used to produce the final outcome (weighting, averaging, or voting). 5.7 Random forest (RF) RF, originally proposed by Ho (1995) and later refined by Breiman (2001), is a widely used ensemble learning algorithm that generates a forest of uncorrelated trees. By utilizing bagging and feature randomness, it produces an averaged prediction that improves the model’s accuracy. A small RF model can be easily interpreted because of its limited number of trees and simpler structure. However, larger RF models with many trees become increasingly difficult to interpret due to their complexity and the ensemble nature of aggregating multiple DTs. 5.8 Artificial neural networks (ANNs) An ANN is a multilayer perceptron algorithm inspired by the human brain (Agatonovic-Kustrin and Beresford, 2000). It consists of interconnected layers of nodes (neurons) designed to learn complex patterns, with at least one input layer, one output layer, and one hidden layer. It uses an activation function that decides whether a node (neuron) should be activated. The activation function takes the weighted sum of the inputs to the node and a bias and passes them through nonlinear trans- formations to generate an output. Examples of activation functions include the Sigmoid, Tanh, ReLU, and Softmax functions. 5.9 Deep Learning DL (LeCun et al., 2015) refers to a class of algorithms that uses neural networks with many hidden layers. DL algorithms are used in many domains of applications and are essential for those involving high-dimensional data, such as audio, image processing, and natural language processing (NLP). Common DL techniques include recurrent neural networks (RNNs), convolutional neural networks (CNNs), graph neural networks (GNNs), and transformers. However, DL models are often considered opaque and difficult to understand and interpret due to their deep architectures, large number of parameters, and complex feature representations. 5.10 Large language models LLMs leverage DL architectures, primarily transformer-based networks (Vaswani et al., 2017) to generate human-like text. Recently, LLMs have evolved into multimodal systems capable of processing and predicting various data types beyond text, making them suitable for many tasks. Furthermore, with the development of agentic AI, LLMs can perform a variety of tasks with autonomy. Examples of LLM models include OpenAI’s GPT, Microsoft’s Copilot, Meta’s LLaMA, Google's Gemini, Anthropic's Claude, and DeepSeek. While these models are trained on massive datasets to learn intricate data patterns, they are prone to hallucinations and biases that can be difficult to detect. The complexity of their archi- tecturesmakes understanding LLMs’ outputs a significant challenge. 5.11 Accuracy versus explainability The prediction accuracy and explainability of ML and AI models exhibit an inverse relationship and vary based on their applications. Fig. 1.2 illustrates the general trade-off for the models described, assuming a relatively complex problem. Rule-based, LR/ER/LogR, naive Bayes (NB), generalized additive models (GAMs), and DT are intrinsically interpretable, with rule-based methods being the most interpretable and the least accurate in general. While DT and GAMs offer more flexibility and often higher accuracy than pure linear models, they retain high transparency, although a very deep DT can become complex to trace. KNN is also highly explainable for small applications, but its interpretability decreases rapidly for larger applications with a higher number of nodes. Conversely, DL and ensemble methods are among the least Understanding AI models: An overview of explainable AI models, taxonomy, programming, and challenges Chapter | 1 5
Page 19
explainable owing to their opaque nature; however, they typically provide high accuracy for sophisticated problems involving high-dimensional data. LLMs are the least interpretable but most accurate for complex applications, such as NLP and healthcare. As this figure shows the general relationship among the AI models, the actual trade-off can differ based on the specific application and method used for quantifying explainability (Herm et al., 2023). 6. XAI taxonomy XAI techniques can be classified based on the scope of explanation, the timing of explanation, the relationship between the explanation model and the AI model, and the specificity of AI models. Fig. 1.3 shows the taxonomy used to classify XAI models. Scope-based explanations are divided into local and global explanations (Barredo Arrieta et al., 2020). Timing-based explanations are further classified into premodel, in-model, and postmodel explanations (Rajabi and Etminani, 2024). The relationship with the AI model is divided into ante- and post-hoc models (Speith, 2022; Retzlaff et al., 2024), with some referring to ante-hoc as intrinsic or transparent (Retzlaff et al., 2024). Finally, XAI models can be classified into model-specific and model-agnostic categories based on their specificity to AI models. Some of these classifications have been overloaded and defined differently in the literature. For example, some call the in-model as FIGURE 1.2 Explainability versus accuracy of AI/ML models (inspired by Gunning and Aha, 2019; Jovic et al., 2025). FIGURE 1.3 XAI taxonomy: classification of XAI techniques. 6 Understanding Models Developed with AI
Page 20
“intrinsic,” referring to transparent models with inherent interpretability (Chinnaraju, 2025; Ribeiro et al., 2016b). Others refer to intrinsic models as transparent models that are attached to and modify AI models to learn their internal workings and weights during training. These classifications are not mutually exclusive, meaning that an XAI model can be classified into more than one category. This section clarifies the main categories of XAI. Other categorizations of XAI models can be based on the explainability mechanism and type of the XAI model. For example, Cheng et al. (2025) divided XAI models into attribution, activation, perturbation, and transformer-based models. Carvalho et al. (2024) categorized XAI based on the explanation model used: rule-based, grammar-based, feature-based, map-based, surrogate, generative-based, neural symbolic, and example-based models. 6.1 Scope of explanation XAI models are classified as either local or global methods based on what they explain. Local XAI approaches explain AI models at the level of individual decisions. For example, when using an AI model to decide whether an individual should be approved for a credit application, a local XAI model can explain the reasons for the rejection of that application. In contrast, global XAI models focus on understanding the overall behavior of a model. For example, a global model can identify the most important features for credit application prediction and how they affect the decision. Together, local and global explanations can provide a detailed overview of the decision-making process and specific reasons for an outcome. 6.2 Timing of explanation XAI models have been categorized based on their timing as premodel, in-model, or postmodel (Carvalho et al., 2019; Corti~nas-Lorenzo and Lacey, 2024). They were also classified as ante- and post-hoc models. Premodel methods were applied to understand the dataset. The in-model can be either inherent to the AI model, such as interpretable and transparent models, or other techniques applied during the model’s execution. Postmodels are applied after AI models make predictions to explain the outcomes. 6.2.1 Premodel explanation Premodel explanations aim to understand how a model would behave before the training phase. While premodel tech- niques are related to exploratory data analysis and feature selection, which are performed prior to training, they aim to explain how features affect the behavior and decisions of the AI model (Pawlicki, 2024). It is important to understand the model’s data to identify the relationships in the data and boost confidence in the model. For example, understanding the important features that affect credit application results, such as income and debt, can be a pre-model step toward un- derstanding the reasons behind the prediction. Premodel techniques include exploratory data analysis techniques and “model cards.” Data analysis include statistical analysis and visualization techniques to help understand the data patterns, abnormalities, and potential biases before training the model. Model cards are benchmarked evaluations of pretrained models under different conditions, intended uses, and potential pitfalls. This helps better understand the models and determine inclusion and possible biases (Mitchell et al., 2019). 6.2.2 In-model explanation While in-model explanations have been referred to as “intrinsic” or “white-box” transparent models in the literature, we differentiate between AI algorithms that are inherently interpretable, such as DT and LR, and XAI models that are attached to an opaque model as an in-model approach to explain it. For example, REverse Time AttentIoN (RETAIN) is an in-model attention-based technique added to RNNs, which becomes part of the algorithm and changes the way RNNs work to retain information. Thus, we further classify in-model techniques into inherently transparent intrinsic and attache techniques. 6.2.3 Postmodel explanation Postmodel explanations, also known as post-hoc models, are XAI models applied after ML inference. They are used to explain a trained model after making predictions to understand its outcomes and how the model produces such results (Retzlaff et al., 2024). These techniques are applied to understand black-box models such as DNNs, transformers, and ensemble algorithms. Understanding AI models: An overview of explainable AI models, taxonomy, programming, and challenges Chapter | 1 7
The above is a preview of the first 20 pages. Register to read the complete e-book.

Recommended for You

Loading recommended books...
Failed to load, please try again later

Tip the Site

Scan the WeChat Pay or Alipay code to tip. No login required.

WeChat Pay
Alipay
← Back to List