AI Approach to Ranking of Search Results and Query Auto-Completions for Enterprise Search

Abstract

Enterprise Search (ES) is an organisation's use of Information Retrieval (IR) technologies to find content from multiple databases (e.g. corporate directories), intranet document repositories and the organisation's web pages. A consequence of the ubiquity of commercial Web Search (WS) is that users have high expectations when interacting with their organisation's ES service. As with WS, ranking is also a major challenge for ES deployments. In many organisations, the ES service is said to be `relevance-blind'. Users or employees cannot find the information they seek within an acceptable time because of poor ranking in the search results or inappropriate `wording' of the initial query. ES is an area of enormous importance for businesses yet has attracted relatively little academic interest, partly due to researchers' difficulties in gaining access to corporate intranets and datasets. Two methods for improving ranking are Learning to Rank (LTR) and Language Modelling (LM). LTR is the application of supervised machine learning techniques for training a model (with labelled data) to provide the best ranking order of documents for a given query item. LM is a foundational component of Natural Language Processing (NLP) and, when applied to ranking tasks, can learn text features that bridge the gap between query and document vocabulary (semantic matching). LTR with LM also enables the discovery of the best candidates for Query Auto-Complete (QAC). QAC for ES can inform new staff members about the range of selections available to them and assist in narrowing that selection even before the user has finished typing a query. Query suggestions can steer searchers to use the appropriate organisational jargon/terminology and avoid submitting queries that produce no results. This thesis investigates whether LTR and LM can significantly enhance the ranking of both search results and query auto-completions in ES. Ranking models are trained and evaluated on real-world ES data from a large third-level academic institution. A custom Language Model, built using LLM techniques, detects enterprise-specific jargon and is integrated as a feature in the ranking pipeline. LTR is used to weigh feature contributions and create ranking models for a live Enterprise Search service. Offline experiments and online A/B tests show improvements in Mean Reciprocal Rank (MRR) and normalized Discounted Cumulative Gain (nDCG) scores, with up to a 15.3% improvement in MRR for QAC and an nDCG@5 gain of 5.85% for search results. The overarching contribution is a bridge of the gap between academic AI ranking models and real-world Enterprise Search implementations. Additionally, an LTR-formatted ES ranking dataset is generated and made publicly available to support reproducibility and future research.