Recommender Systems: A Study of User-Cold Start and Reinforcement Learning

Abstract

Recommender systems (RS) are used in online platforms to recommend personalised items to users. For this, it is crucial for the RS to accurately identify the user's preferences. This is particularly challenging when recommending items to new users who join the system, also known as the user-cold start problem in RS. The system initially has little to no information about a new user, so it needs to quickly learn the user's preferences. At the same time, the interests of a user can change over time, and the system needs to dynamically adapt to these changing preferences in order to provide meaningful recommendations. Therefore it is important for a recommender to continuously learn an effective representation of a user's interests. In this thesis, we study the use of Reinforcement Learning (RL) to address various item-recommendation tasks, with an emphasis on effectively and continuously learning about the users.

We first propose a novel Monte Carlo Tree Search (MCTS) recommender that quickly and accurately learns the preferences of a new user with fewer item interactions. The MCTS does this by intelligently selecting items that allows it to quickly learn about a new user, through the user's feedback to these items. We show that the MCTS achieves superior cold-start performance in a computationally lightweight manner. Although the MCTS is better suited for explicit feedback in terms of item ratings, we also show that the MCTS can be extended to diverse feedback types such as pairwise ranked comparison and missing-not-at-random.

Second, we introduce RLT4Rec, a transformer-based Reinforcement Learning recommender that is capable of handling both new and established users within a single, scalable framework. RLT4Rec only takes a sequence of prior user interactions as input and eliminates the need for an explicit user state, instead learning an effective internal representation solely from prior interaction sequences. Our findings show that RLT4Rec consistently surpasses our MCTS and several other RL-based recommender baselines in both user-cold start tasks and recommending items to established users.

Finally, we reassess the effectiveness of existing Reinforcement Learning based systems in sequential recommendation tasks. We identify significant reproducibility issues and evaluations inconsistencies in the existing RL-based recommender literature. To measure the relative progress of RL systems in recommenders, we conduct thorough evaluations of RL-enhanced recommenders against well-known supervised learning-based recommenders. Our findings show that when evaluated under standard next-item prediction protocols, existing RL-enhanced models often fail to outperform properly tuned supervised learning baselines.