Incorporating Cognitive Relevance into Dense Retrieval for Conversational Search
Fang Yubo (AY 2021)
Conversational search, where users perform information-seeking conversations with an agent using natural language, has gained substantial attention from the information retrieval research community. Existing studies of conversational search mainly focus on natural language understanding of users’ utterances to capture their information needs by asking clarifying questions. However, the available action space of a conversational search system should not be constrained to these. In this study, we aimed to develop a novel approach that improves the search performance of a conversational search system by proposing new type of questions the conversational search system could ask. The current IR systems mainly focus on retrieving documents that have algorithmic relevance, which was defined by Saracevic as “concerns the relation between a query and the texts in the documents.” Instead, we focused on incorporating cognitive relevance, which concerns the relation between the state of knowledge and cognitive information need of a user, into conversational search. Furthermore, we built on an effective state-of-the-art dense retrieval method and proposed a better training strategy that improved searching accuracy. Finally, we proposed a novel approach to combining the cognitive relevance and dense retrieval method, which trained the conversational search model to retrieve documents based on their relevance and novelty concerning users information needs. The experimental results show that the proposed system can outperform the baseline system when concerning the diversity of the response to each turn of informationseeking conversations with an acceptable level of decrease in relevance. This suggested that our proposed system could retrieve documents concerns not only their relevance but also their novelty concerning the knowledge states of the user. For future work, we will further explore balancing the algorithmic relevance and cognitive relevance trade-off to improve the search performance of conversational search systems.