Digital Library

Domain-Specific Small Language Models (MEAP, all 15 chapters) (Guglielmo Iozzia) (z-library.sk, 1lib.sk, z-lib.sk)

Share E-Book
Read Without Registration

Domain-Specific Small Language Models (MEAP, all 15 chapters) (Guglielmo Iozzia) (z-library.sk, 1lib.sk, z-lib.sk)

Author

AI
Language English

Bigger isn’t always better. Train and tune highly focused language models optimized for domain specific tasks. When you need a language model to respond accurately and quickly about a specific field of knowledge, the sprawling capacity of a LLM may hurt more than it helps. Domain-Specific Small Language Models teaches you to build generative AI models optimized for specific fields. In Domain-Specific Small Language Models you’ll discover: Model sizing best practices Open source libraries, frameworks, utilities and runtimes Fine-tuning techniques for custom datasets Hugging Face’s libraries for SLMs Running SLMs on commodity hardware Model optimization or quantization Perfect for cost- or hardware-constrained environments, Small Language Models (SLMs) train on domain specific data for high-quality results in specific tasks. In Domain-Specific Small Language Models you’ll develop SLMs that can generate everything from Python code to protein structures and antibody sequences—all on commodity hardware. about the reader For machine learning engineers familiar with Python. about the author Guglielmo Iozzia is a Director, ML/AI and Applied Mathematics at MSD. He studied Electronic and Biomedical Engineering at the University of Bologna, has an extensive background in Software and ML/AI Engineering applied to real-life use cases across different industries, such as Biotech Manufacturing, Healthcare, Cloud Operations, and Cyber Security.

Format PDF
Size 9.4 MB
5
Views
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
(This page has no text content)
Page 2
(This page has no text content)
Page 3
Domain-Specific Small Language Models 1. welcome 2. 1_Large_Language_Models 3. 2_Tuning_for_a_Specific_Domain 4. 3_End-to-end_Transformer_Fine_Tuning 5. 4_Running_Inference 6. 5_Exploring_ONNX 7. 6_Quantizing_for_Your_Production_Environment 8. 7_Generating_Python_Code 9. 8_Generating_Protein_Structures 10. 9_Advanced_Quantization_Techniques 11. 10_Profiling_Insights 12. 11_Deployment_and_Serving 13. 12_Running_on_Your_Laptop 14. 13_Creating_End-to-end_LLM_applications 15. 14_Advanced_Components_for_LLM_Applications 16. 15_Test-time_Compute_and_Small_Language_Models
Page 4
welcome Thank you for purchasing the MEAP for Domain-Specific Small Language Models. To get the most benefit from this book, you’ll want to have some established skills in Python programming, with experience in PyTorch and knowledge about training/fine tuning Transformers-based language models, and willing to learn more about higher level and more user friendly API, optimizing such models, to make them ready for serving and inference on different hardware, also when on a tight infrastructure budget, or when the destination deployment system is a device, such as a laptop or a smartphone. I started experimenting with Transformer models in early 2022, after reading DeepMind's original paper “Attention is All You Need” and did a first local implementation of a coding assistant based on a fine tuned version of the CodeGen Open Source model from SalesForce. That started first as a pet project, to learn more about this language model architecture, but then I spotted the potential for small language models in other more challenging and industry specific tasks and data (and still being of use in many projects in my specific field of application, biotech/pharma, since then). The discovery and understanding of optimization and quantization techniques and libraries contributed to the spark to move in this direction. This looks like the right time to release this book, as, after the release of OpenAI’s ChatGPT in November 2022 and the followup closed source and large language models from many other competitors, a lot of hype has been produced, due to my opinion mostly on general misunderstanding of the way these models work and which real capabilities and limitations they have. Also, a false narrative from diverse big tech companies and consulting firms is masking the existing alternative and valid Open Source ecosystem and the limits of the third-party commercial modes to tackle industry specific tasks, because of a lack of domain specific knowledge and data and other concerns about safety, security, privacy and budget. This book is then my attempt to produce an as much as possible exhaustive, highly informative, deep dive into the available Open Source libraries, papers, techniques and best practices to make domain-specific and local models working on your own challenges
Page 5
and private data. I have tried to cover those topics that are more mature, deep diving into the concepts behind the used API and providing several working code examples that cover everything discussed across the chapters of this book. Along with all that, I also wanted to focus on two specific and often oversought areas: SMLs quantization and inference speed. Two key areas that make relying on SLMs possible for industry specific tasks. Finally, this book provides insights and best practices about the design and implementation of complex systems that use SLMs. While this book covers many topics and aspects about SLMs keeping production in mind, it isn’t comprehensive of all the possible alternatives and use cases (it cannot, being the the available Open Source ecosystem too vast and the areas for application of SLMs wider than the same for larger language models), but it has been written with the goal to give the readers all the building blocks to tackle any project in this field. I hope you find it as useful to read as I did to write it. Please be sure to post any questions, comments, or suggestions you have about the book in the liveBook discussion forum. Your feedback is essential in developing the best book possible. Thanks. —Guglielmo Iozzia In this book welcome 1 Large Language Models 2 Tuning for a Specific Domain 3 End- to-end Transformer Fine Tuning 4 Running Inference 5 Exploring ONNX 6 Quantizing for Your Production Environment 7 Generating Python Code 8 Generating Protein Structures 9 Advanced Quantization Techniques 10 Profiling Insights 11 Deployment and Serving 12 Running on Your Laptop 13 Creating End-to-end LLM applications 14 Advanced Components for LLM Applications 15 Test-time Compute and Small Language Models
Page 6
1 Large Language Models Whenever a new, potentially disruptive technology emerges—or an existing one undergoes a radical change—there's an inevitable hype about it. This hype can result in a swift drop in interest if expectations aren't met or, conversely, lead to a rapid increase in investments and projects before its true business potential and associated risks are fully understood. Either way, nothing good would come out from a new big thing. You can observe the same happening nowadays to Large Language Models (LLMs): huge hype about them which makes it difficult to understand what they really can do (and cannot do) and if they really could be a good fit for some of your own business needs. But hype is the only thing you wouldn’t find in this book. This chapter aims to clarify the LLM world by highlighting pros, risks and challenges of both generalist and domain-specific models. 1.1 10000 feet overview The release of ChatGPT by OpenAI on November 30th 2022, is widely regarded as the beginning of LLM revolution, marking a significant moment in our shared technological narrative. Actually, that should be seen as the starting date of the commercialization of LLMs, as the underlying technology behind has been around since more than 5 years before. Everything started in 2017, when researchers at DeepMind published the “Attention is All You Need” paper [1], where they introduced for the first time the concept of Transformer, which is the parent architecture of other LLMs that have been implemented lately, including GPT-3.5, upon which the OpenAI’s ChatGPT first release was based. But why the Transformers started a revolution in the Artificial Intelligence (AI) field? Haven’t Natural Language Process (NLP) tasks been already accomplished before the DeepMind’s aforementioned paper release? That’s correct: the idea to make a computer able to understand text in natural language has been around in the past decades and major advancements in tasks such as text classification, completion or translation have been achieved through other Deep Learning (DL) architectures, trained mostly in a supervised way, also before 2017. But preliminary approaches
Page 7
didn’t give the possibility to shift towards semi-supervised or possibly unsupervised learning and weren’t suitable to work at a larger scale, to be trained on a vast corpus of unlabeled text. To provide you with a simplified explanation of different types of learning, here is a brief description: Supervised Learning: Models are trained with labeled data, learning to predict outputs from inputs, useful for tasks like text categorization. Unsupervised Learning: Works on unlabeled data to identify hidden structures, ideal for clustering and topic modeling. Semi-Supervised Learning: Mixes a small amount of labeled data with a larger set of unlabeled data, helping to improve learning when labels are limited. Reinforcement Learning: Models learn through trial and error using rewards, applied in areas such as conversational AI. Self-Supervised Learning: Generates its own supervision from the data, often employed in language models for tasks like filling in blanks within texts. In supervised learning many pairs of inputs and related labels are used to train a model. This paradigm is very powerful but also very laborious and costly, as it requires humans labelling all the training data. In order to make a model more powerful you need more data, but scaling a data labelling procedure is harder than scaling a model itself. Data labelling becomes the bottleneck for supervised learning here. LLMs follow a self-supervised training paradigm: rather than having a dataset where labels have been assigned by humans, the labels are generated programmatically. It usually starts by collecting huge amounts of text (more than what is collected for supervised learning). Data is then partitioned into input and label: for example, in case of a text completion task, this is achieved by chopping off the final word and making it the label. By removing the “next word” and having the model guess it, you can compare the prediction to the actual word that we got from the original data set and teach the model to guess the proper word (which is consistent with the fact that this kind of models are trained to be used for generative purposes). The first LLMs based on the Transformer or subsequent architectures have been trained to address the same basic tasks on text as for their predecessor
Page 8
neural networks, but, because being trained on a larger corpus of data, they could capture also lot of both syntax and semantics of natural language, which developed other abilities that allow them to complete also diverse and more complex tasks, such as solving arithmetical problems, completing programming code or summarize a document. 1.2 The Transformer Architecture Prior to the Transformers, tasks on unstructured text were performed using Recurrent Neural Network (RNN) models. Trained on labelled data, these architectures use their internal state to process variable-length sequences of text inputs. Later evolutions, such as LTSM (Long-short Term Memory) or GRU (Gated Recurrent Unit) architectures, introduced the capability to remember and forget important information, but they didn’t solve one of the major challenges of RNNs: their slowness when training, due to the need to feed the data sequentially (with no chance then to parallelize and get the most from all the available processor cores). Some of the problems affecting RNNs were addressed with the introduction of the so-called attention mechanism, which is a technique to enhance some parts of an input and at the same time diminish the other parts. In RNN architectures the amount of information that can be propagated is limited and the window of retained information is shorter, but through the attention mechanism it can be increased. However, this didn’t solve the scalability issues. We had to wait for the coming of the Transformer for that, as this novel architecture introduced two major changes compared to the RNN models: Use the self-attention mechanism (a specialization of the attention mechanism which allows a model relating different positions of a sequence to compute a representation of the sequence itself), so that they can process the entire input all at once. No recurrent structure. This means that a Transformer allows more parallelism than a RNN and consequently requires significantly reduced training time. Not only the novel architecture: another key component of the success for the Transformers in becoming the standard choice for any task related to text is represented by the so-called word embeddings, high-dimensional vector
Page 9
representations of words to capture their semantic and syntactic properties. They enable a Transformer model to manipulate words in a numerical space, where nearby words share some relationship of meaning and/or other similarity. This allows then treating words no more as isolated entities, so that a model can understand their interplay within a given context. Transformer architectures have been proposed also for other tasks where different types or data is involved, such as Computer Vision, but those use cases are out of scope for this book, as it focuses on generative models that can handle unstructured text only. 1.3 Evolutions of Transformers The Transformer architecture proposed by DeepMind was just the start. As soon as researchers and professionals started to use it, they spotted room for improvements and/or specializations for tasks involving text. The original architecture is encoder-decoder based: the encoder part is responsible to convert the input text into a numerical representation, while the decoder part takes as input such representation and decodes it back to text. Both use the attention mechanism. In time, it has been proved that for some tasks only the encoder part is needed, while for some others a language model could rely only on the decoder part. This then led to the birth of two different families of Transformers: BERT (Bidirectional Encoder Representation from Transformers). Like its name suggests, it uses only the encoder part. This kind of architecture performs better typically in classification or prediction tasks on text. GPT (Generative Pre-trained Transformer). It uses only the decoder part. It performs better typically in generative tasks on text. At the end, the architecture choice depends on the specific tasks you want to achieve. The examples in this book cover architectures belonging to both families. Another important concept, to complete this overview of the evolution of the Transformer, is the so-called RLHF (Reinforcement Learning from Human
Page 10
Feedback), a technique which is behind the scenes for the OpenAI LLMs. As you can image from the name, it uses Reinforcement Learning so that an LLM can learn an optimal policy to maximize a reward. The model behind ChatGPT is an evolution of the GPT architecture (the first 2 versions released as Open Source, while subsequent evolutions moved toward a closed source model), towards which they started applying RLHF to do fine tuning and improve performance. 1.4 Areas of application The definition of LLM is a little bit misleading to many. While it is clear why these models are called large (as their size can span from few billions to more than one hundred billion parameters, which translates in dozen/hundred Gigabytes), the term language makes often people believe that the generated content can be text in natural language only. It is true that the first NLP applications started to address translations tasks from a natural language to another, but overcoming the limitations of RNN architectures and the emergence of new abilities made possible to train LLMs to perform diverse tasks beyond translation, such as: Language understanding Text classification Test generation Question answering Document summarization Semantic parsing Pattern recognition Basic math solving Code generation Dialogue General knowledge Logical inference chains This leads also to make possible for an LLM to handle different types of text other than natural language: basically, anything that is a symbolic representation of something, as shown in figure 1.1:
Page 11
Figure 1.1 Some examples of diverse content an LLM can generate That’s why LLMs are going to untap a great number of real-life use cases in
Page 12
diverse industries. 1.5 The Open Source revolution The introduction of the Transformer architecture wasn’t the spark to ignite the interest and efforts only of business entities, such as OpenAI, Microsoft or Google. Also, the Open Source community started their contribution of LLMs with more or less permissive licenses about models, code, and training data. Figure 1.2, taken from the paper [2], shows a timeline of the major commercial and Open Source LLMs from 2019 to mid-2023 (date of the latest update of the paper, but other models came also after or still coming in): Figure 1.2 The timeline of LLMs since 2019 (image taken from paper [2])
Page 13
Those highlighted in yellow in figure 1.2 are publicly available to use, with different license permissions (please remember to always check the usage terms before planning to use one for commercial purposes). Some of the
Page 14
proposed OS models are generalist, same as those commercials from OpenAI and Microsoft, some others are specialized in a particular domain. Paradoxically, the second spark that accelerated the OS contributions was the release and opening to final users of ChatGPT: the count of new proposals or newer releases or forks of existing models literally tripled since then. With efforts not only from research, but also from industry and passionate individuals or groups. This means that you have choices: whether for any reason you would need to shift from a proprietary LLM model to one entirely owned by your organization, you can: by picking one pre-trained OS model, you can then instruct or fine-tune it on your own data rather than develop and train from scratch. Which in terms of costs is a big saving. Figure 1.3 shows the financial impact of developing a custom LLM from scratch: Figure 1.3 Order of magnitude of costs for each phase of LLM implementation from scratch
Page 15
(This page has no text content)
Page 16
The situation represented in this figure is like what OpenAI or Google had to do since starting with their GPT-X or Bard models. The green “$” symbols indicate the magnitude of costs for each phase. Contrary to the common belief, the training phase of such large models is costly, but it isn’t the costliest. The development phase requires more resources, as it includes the collection and preparation of vast amounts of unstructured data and all the experiments needed to achieve a stable model. Once the training of the final candidate is completed and its performance have met the KPIs for it, deployment and serving for inference represent another big challenge, as ensuring scalability, execution performance, quality, monitoring, seamless user experience, etc. translate into costs not always possible to forecast upfront with good precision. But the Open Source offering can help here. Figure 1.4 shows the financial impact of implementing a custom LLM starting from a pretrained OS model: Figure 1.4 Order of magnitude of costs for each phase of LLM implementation when starting from a pretrained model
Page 17
(This page has no text content)
Page 18
Again here, the green “$” symbols indicate the magnitude of costs for each phase in this scenario. Definitely you can cut costs related to development, as you already have identified the architecture and a pretrained model that would be the baseline for your specific task. The costs for data collection and preparation are what is left here (they are non-negligible, but that’s the only financial impact left for this phase). Fine tuning has still costs, but not comparable with the training of a LLM from scratch: computational power is still needed, but you have more room for optimization in this scenario. You can notice that nothing changed about deployment and inference: same as when developing and training from scratch, challenges and costs remain the same. But this is the part where all the topic explained in this book come to the rescue: a set of techniques and best practices through which it would be possible to optimize your custom LLM to reduce computational power consumption (while keeping acceptable accuracy), in so allowing deployment and inference also in a constrained environment and cost saving. Long story short: you will learn how to make custom LLMs run inference without the same powerful infrastructure that big tech companies such Microsoft, Google or Meta have. 1.6 Risks and challenges with generalist LLMs The commercial generalist closed source LLMs available from third party vendors are powerful, can address diverse tasks, provide a good user experience for non-technical users too, have costs associated to usage only (as you don’t have to care about deployment and maintenance), but come with concrete risks that can make all the pros worthless and that are listed in this section. Deployment outside your organization network. This means that any information you put in the prompt for them potentially goes outside your company boundaries and, depending on the usage agreement terms, could be used to fine tune the model and become available to other customers. You most probably have already read of cases where employees at large corporations have passed some sensitive or IP data as input to an external service hosting the model (it happened at Samsung [3] and many others).
Page 19
Data leakage risks. Being the model hosted on an infrastructure where you don’t have any control, bad cyber security postures (unbeknownst to you, with no chance to set up a mitigation strategy then) can lead malicious actors to access private data. There is evidence that something like this happened once for ChatGPT [4]. Closeness: this means you don’t have any detail about how a LLM has been implemented and trained, then no chance for reproducibility or interpretability. The latter being a fundamental requirement for some heavy regulated sectors. You often don’t have visibility about infrastructure and/or model version changes. Transparency: data used to train closed source models aren’t publicly disclosed. No chance then to verify if they are affected by some bias or contain copyrighted data for which permission to use it hasn’t been granted from the owner. Hallucinations: the generated information could be either in conflict with the existing source (intrinsic hallucination) or cannot be verified by the available source (extrinsic hallucination). Intrinsic hallucination can happen also on custom models trained locally to an organization, on proprietary data (this is anyway one challenge with LLMs in general), but no mitigation is possible for extrinsic hallucination when no access to or knowledge about the training data is available. Coding. This emergent LLMs’ ability opens the possibility to a broader audience, having also limited coding skills, to unwillingly generate code containing malicious elements or produce malware easily. Unfortunately, third-party LLMs don’t have sophisticated guardrails for this as they can be bypassed working properly on the prompt. What an organization can do is to setup its own guardrails (which isn’t a trivial task) or shift the responsibility to do the proper validation on internal users. All these risks, along with the fact that these large models are trained on a vast number of tasks on vast but still generalist data sets, make them not always suitable for some important business purposes. 1.7 When do domain specific LLMs provide a greater business value than generalist ones?
Page 20
A question arises: is it preferable to move towards a custom domain specific LLM? There are several considerations to do if you want to make the right choice. First, let’s have a quick look at the composition of the training data for the available pre-trained LLMs. Figure 1.5, take from the paper [2], shows the ratios of the data source types used to train some popular existing LLMs: Figure 1.5 Ratios of data source types used to train some popular existing LLMs You can notice that majority of the training data (not only for the LLMs mentioned in the figure, as this is a pattern for all the commercial and OS pretrained LLMs) represents content mostly scraped from the Web or taken
The above is a preview of the first 20 pages. Register to read the complete e-book.

Recommended for You

Loading recommended books...
Failed to load, please try again later

Tip the Site

Scan the WeChat Pay or Alipay code to tip. No login required.

WeChat Pay
Alipay
← Back to List