Page
1
M A N N I N G Tommaso Teofili Foreword by Chris Mattmann
Page
2
Deep Learning for Search
Page
3
(This page has no text content)
Page
4
Deep Learning for Search TOMMASO TEOFILI FOREWORD BY CHRIS MATTMANN M A N N I N G SHELTER ISLAND
Page
5
For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com © 2019 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Frances Lefkowitz 20 Baldwin Road Review editor: Ivan Martinović PO Box 761 Production editor: Tiffany Taylor Shelter Island, NY 11964 Copy editor: Tiffany Taylor Proofreader: Katie Tennant Typesetter: Dottie Marsico Cover designer: Marija Tudor ISBN 9781617294792 Printed in the United States of America
Page
6
To Mattia, Giacomo, and Michela “Happiness is only real when shared.” —Christopher McCandless
Page
7
(This page has no text content)
Page
8
brief contents PART 1 SEARCH MEETS DEEP LEARNING......................................... 1 1 ■ Neural search 3 2 ■ Generating synonyms 36 PART 2 THROWING NEURAL NETS AT A SEARCH ENGINE ............... 73 3 ■ From plain retrieval to text generation 75 4 ■ More-sensitive query suggestions 112 5 ■ Ranking search results with word embeddings 146 6 ■ Document embeddings for rankings and recommendations 170 PART 3 ONE STEP BEYOND........................................................ 193 7 ■ Searching across languages 195 8 ■ Content-based image search 225 9 ■ A peek at performance 267vii
Page
9
(This page has no text content)
Page
10
contents foreword xiii preface xv acknowledgments xvii about this book xviii about the author xxi about the cover illustration xxii PART 1 SEARCH MEETS DEEP LEARNING..........................1 1 Neural search 3 1.1 Neural networks and deep learning 4 1.2 What is machine learning? 7 1.3 What deep learning can do for search 9 1.4 A roadmap for learning deep learning 12 1.5 Retrieving useful information 13 Text, tokens, terms, and search fundamentals 14 ■ Relevance first 22 ■ Classic retrieval models 23 ■ Precision and recall 24 1.6 Unsolved problems 24 1.7 Opening the search engine black box 25 1.8 Deep learning to the rescue 27 1.9 Index, please meet neuron 31ix
Page
11
CONTENTSx1.10 Neural network training 31 1.11 The promises of neural search 34 2 Generating synonyms 36 2.1 Introduction to synonym expansion 37 Why synonyms? 39 ■ Vocabulary-based synonym matching 40 2.2 The importance of context 49 2.3 Feed-forward neural networks 51 2.4 Using word2vec 54 Setting up word2vec in Deeplearning4j 63 ■ Word2vec-based synonym expansion 64 2.5 Evaluations and comparisons 66 2.6 Considerations for production systems 67 Synonyms vs. antonyms 69 PART 2 THROWING NEURAL NETS AT A SEARCH ENGINE........................................73 3 From plain retrieval to text generation 75 3.1 Information need vs. query: Bridging the gap 77 Generating alternative queries 77 ■ Data preparation 79 Wrap-up of generating data 86 3.2 Learning over sequences 86 3.3 Recurrent neural networks 88 RNN internals and dynamics 91 ■ Long-term dependencies 94 Long short-term memory networks 95 3.4 LSTM networks for unsupervised text generation 95 Unsupervised query expansion 103 3.5 From unsupervised to supervised text generation 106 Sequence-to-sequence modeling 107 3.6 Considerations for production systems 110 4 More-sensitive query suggestions 112 4.1 Generating query suggestions 113 Suggesting while composing queries 113 ■ Dictionary-based suggesters 114
Page
12
CONTENTS xi4.2 Lucene Lookup APIs 115 4.3 Analyzed suggesters 118 4.4 Using language models 124 4.5 Content-based suggesters 128 4.6 Neural language models 129 4.7 Character-based neural language model for suggestions 131 4.8 Tuning the LSTM language model 134 4.9 Diversifying suggestions using word embeddings 142 5 Ranking search results with word embeddings 146 5.1 The importance of ranking 147 5.2 Retrieval models 149 TF-IDF and the vector space model 151 ■ Ranking documents in Lucene 154 ■ Probabilistic models 156 5.3 Neural information retrieval 158 5.4 From word to document vectors 159 5.5 Evaluations and comparisons 165 Similarity based on averaged word embeddings 166 6 Document embeddings for rankings and recommendations 170 6.1 From word to document embeddings 171 6.2 Using paragraph vectors in ranking 174 Paragraph vector–based similarity 177 6.3 Document embeddings and related content 177 Search, recommendations, and related content 178 ■ Using frequent terms to find similar content 179 ■ Retrieving similar content with paragraph vectors 188 ■ Retrieving similar content with vectors from encoder-decoder models 191 PART 3 ONE STEP BEYOND.........................................193 7 Searching across languages 195 7.1 Serving users who speak multiple languages 196 Translating documents vs. queries 197 ■ Cross-language search 199 ■ Querying in multiple languages on top of Lucene 200
Page
13
CONTENTSxii7.2 Statistical machine translation 202 Alignment 204 ■ Phrase-based translation 205 7.3 Working with parallel corpora 206 7.4 Neural machine translation 208 Encoder-decoder models 209 ■ Encoder-decoder for MT in DL4J 212 7.5 Word and document embeddings for multiple languages 219 Linear projected monolingual embeddings 219 8 Content-based image search 225 8.1 Image contents and search 227 8.2 A look back: Text-based image retrieval 229 8.3 Understanding images 231 Image representations 233 ■ Feature extraction 235 8.4 Deep learning for image representation 243 Convolutional neural networks 245 ■ Image search 253 Locality-sensitive hashing 258 8.5 Working with unlabeled images 261 9 A peek at performance 267 9.1 Performance and the promises of deep learning 268 From model design to production 269 9.2 Indexes and neurons working together 284 9.3 Working with streams of data 287 index 295
Page
14
foreword It’s hard to quantify how commonplace terms like neural networks and deep learning have become, and, more directly, how these technologies are impacting our lives. From automating routine jobs, to replicating difficult decisions, to helping cars drive themselves (and human beings) to their destinations, the power of neural networks and deep learning as techniques for revolutionizing computing is only in its nascence. That’s why this book is so material and important. Not only are neural networks, AI, and deep learning automating routine jobs and decisions and making them easier, but they are also making search easier. Formerly, the state of the art in information retrieval and search involved complex linear algebra, including matrix multiplication to represent the matching of user queries to documents. Today, instead of using alge- braic and linear models, the state of the art involves—as an example—the application of neural networks to discern word similarity between documents after learning how to summarize the documents into words using separate networks. And that is only one area in the search process where AI and deep learning are being used. Tommaso Teofili, in Deep Learning for Search, takes a practical approach toward showing you the state of the art in using neural networks, AI, and deep learning in the development of search engines. The book is filled with examples and walks the reader through the architecture of today’s search engines—while also giving you enough background to understand how and where deep learning fits, and how it makes search better. From building your first network to find similar words in a query expan- sion, to learning word embeddings to help with search ranking, to searching across languages and images, Tommaso shows you where AI and deep learning can super- charge your code and search capability. The book is written by a true pioneer of open source. Tommaso is a former chair of the Apache Lucene project—the de facto search indexing engine that powers Elasticsearch and Apache Solr—and he has also contributed greatly to languagexiii
Page
15
FOREWORDxivunderstanding and translation on Apache OpenNLP. More recently, he is the proposed chair for the Apache Joshua (incubating) project for statistical machine translation. I know that you will learn a great deal from this book, and I commend it for find- ing a middle ground in common sense, explanations of complex theory, and real code that you can play with using the latest deep learning and search technologies. Enjoy. I know I did! —CHRIS MATTMANN ASSOCIATE CHIEF TECHNOLOGY AND INNOVATION OFFICER, NASA JPL
Page
16
preface The field of natural language processing bewitched me as soon as I came to know about it nearly 10 years ago, while studying for my master’s degree. The promise that computers could help us understand the (already, even then) vast amount of textual documents in existence sounded like magic. I still remember how exciting it was to see my first NLP programs extract even vaguely correct and useful information from a few text documents. About the same time, at work, I was asked to do some consulting for a customer on their new open source search architecture. My colleague, who was an expert in the field, was busy on another project, so I was given a copy of Lucene in Action,1 which I studied for a couple of weeks; then I was sent out on the consulting job. A couple of years after I worked on that Lucene/Solr-based project, the new search engine went live (and, as far as I know, it’s still used). I can’t tell you how many times the search engine algorithms needed to be adjusted because of this or that query or this or that fragment of indexed text, but we made it work. I could see users’ queries, and I could see the data that was there to be retrieved, but a minimal difference in spelling or omitting a certain word could cause very relevant information to not show up in the search results. So while I was very proud of my work, I kept wondering how I could have done better to avoid the many manual interventions the product managers asked me to perform in order to provide the best possible user experience. Right after this, I quite by chance found myself involved in machine learning thanks to Andrew Ng’s first machine learning online class (which originated the Cour- sera MOOC series). I was so fascinated with the concepts behind the neural networks shown in the class that I decided to try to implement a small library for neural 1 Michael McCandless, Erik Hatcher, and Otis Gospodnetić (Manning, 2010), http://www.manning.com/ books/lucene-in-action-second-edition.xv
Page
17
PREFACExvinetworks in Java myself, just for fun (http://svn.apache.org/repos/asf/labs/yay/). I started hunting for other online courses like Andrej Karpathy’s course on convolu- tional neural networks for visual recognition and Richard Socher’s course on deep neural networks for natural language processing. Since then, I have kept working on search engines, natural language processing, and deep learning, mostly in open source. A couple of years ago (!), Manning reached out to me to review a book on NLP, and I was naive enough to write at the bottom of my review that I would be interested in writing a book on search engines and neural networks. When Manning came back to me, expressing interest, I was kind of surprised, and wondered, do I really want to write a book on that? I realized that, yes, I was interested. While deep learning has revolutionized computer vision and natural language pro- cessing, there’s still a lot to uncover for its applications in search. I’m sure we can’t (yet?) rely on deep learning to automatically set up and tune search engines on our behalf, but it can help a lot in making the search engine user’s experience smoother. With deep learning, we can do things in search engines that we can’t do with other existing techniques so far, and we can use deep learning to enhance the techniques we already use in search engines. The journey toward making search engines more effective through deep neural networks has just started. I hope you enjoy it.
Page
18
xvii acknowledgments First and foremost I would like to thank my lovely wife Michela for encouraging and supporting me throughout this long journey: thanks for the love, energy, and dedica- tion during long days, nights, and weekends of writing! Thanks go to Giacomo and Mattia for helping me choose the coolest cover illustra- tion possible and for all the playing and laughs while I was trying to write. I would like to thank my father for his pride and his confidence in me. Big thanks go to my friend Federico for his tireless effort in reviewing all the book material (book, code, images, and so on) and for the enjoyable discussions and ideas shared. More huge thanks go to my friends and colleagues Antonio, Francesco, and Simone for their support, laughs, and advice. Thanks also go to my fellow Apache OpenNLP (http://opennlp.apache.org) friends Suneel, Joern, and Koji for providing feedback, advice, and ideas that helped shape the book. I thank Chris Mattmann for writing such an inspiring foreword. My thanks also go to Frances Lefkowitz, my development editor, for her patience and guidance throughout the writing process, including our discussions about Steph, KD, and the Warriors. And I thank the others at Manning who made this book possi- ble, including publisher Marjan Bace and everyone on the editorial and production teams who worked behind the scenes. In addition, I thank the technical peer review- ers led by Ivan Martinović —Abhinav Upadhyay, Al Krinker, Alberto Simões, Álvaro Falquina, Andrew Wyllie, Antonio Magnaghi, Chris Morgan, Giuliano Bertoti, Greg Zanotti, Jeroen Benckhuijsen, Krief David, Lucian Enache, Martin Beer, Michael Wall, Michal Paszkiewicz, Mirko Kämpf, Pauli Sutelainen, Simona Ruso, Srdan Dukic, and Ursin Stauss—and the forum contributors. On the technical side, thanks go to Michiel Trimpe, who served as the book’s technical editor; and Karsten Strøbaek, who served as the book’s technical proofreader. Finally I’d like to thank the Apache Lucene and Deeplearning4j communities for providing such excellent tools and for supporting users in a friendly manner.
Page
19
about this book Deep Learning for Search is a practical book about how to use (deep) neural networks to help build effective search engines. This book examines several components of a search engine, providing insights on how they work and guidance on how neural net- works can be used in each context. Emphasis is given to practical, example-driven explanations of search and deep learning techniques, most of which are accompanied by code. At the same time, references to relevant research papers are provided where applicable to encourage you to read more and deepen your knowledge on specific topics. Neural network and search-specific topics are explained throughout the book as you read about them. After reading this book, you’ll have a solid understanding of the main challenges related to search engines, how they are commonly addressed, and what deep learning can do to help. You’ll gain a solid understanding of several different deep learning techniques and where they fit in the context of search. You’ll get to know the Lucene and Deeplearning4j libraries well. In addition, you’ll develop a practical attitude toward testing the effectiveness of neural networks (rather than viewing them as magic) and measuring their costs and benefits. Who should read this book This book is intended for readers with an intermediate programming background. It will be best if you’re proficient in Java programming, with an interest or active involve- ment in developing search engines. You should read this book if you would like to make your search engine more effective at giving relevant results and therefore more useful for end users. Even if you don’t have a search background, basic concepts about search engines are introduced during the course of the book as each specific aspect of search isxviii
Page
20
ABOUT THIS BOOK xixtouched on. Similarly, you aren’t expected to already know about machine or deep learning. This book will introduce all the required machine learning and deep learn- ing basics, together with practical tips regarding the application of deep learning to search engines in production scenarios. You should be ready to get your hands on the code and extend existing open source libraries to implement deep learning algorithms to solve search problems. Roadmap This book is divided into three parts: Part 1 introduces the basic concepts of search, machine learning, and deep learning. Chapter 1 introduces the rationale for applying deep learning tech- niques to search problems by touching on problems with respect to most com- mon approaches to information retrieval. Chapter 2 gives a first example of how to use a neural network model to improve the effectiveness of a search engine by generating synonyms from the data. Part 2 deals with common search engine tasks that can be better addressed with the help of deep neural networks. Chapter 3 introduces the use of recurrent neural networks for generating queries that are alternatives to the ones entered by users. Chapter 4 addresses the task of providing better suggestions while the user is typing the query, with the help of deep neural networks. Chapter 5 focuses on ranking models: in particular, how to provide more-relevant search results using word embeddings. Chapter 6 deals with the use of document embeddings both in ranking functions and in the context of content recom- mendation. Part 3 takes up more-complex scenarios like deep learning–powered machine translation and image search. Chapter 7 guides you through giving your search engine multilanguage capabilities through neural network–based approaches. Chapter 8 deals with searching a collection of images based on their contents, empowered by deep learning models. Chapter 9 discusses production-related topics like fine-tuning deep learning models and dealing with constantly incom- ing streams of data. The complexity of the topics and concepts addressed grows over the course of the book. If you’re new to deep learning, search, or both, I highly recommend reading chapters 1 and 2 first. Otherwise, feel free to jump around and pick chapters based on your needs and interests. About the code In this book, code snippets are preferred over fully detailed code listings, in order to provide quick, easy insight into what the code is doing and how. The full source code can be found on the book’s page on the Manning website: www.manning.com/ books/deep-learning-for-search. The software will also be kept up to date on the