The Impact of Generative AI on the Process and Outcomes of Collaborative Trip Planning Tasks
Hikaru Kumamoto (AY 2023)
Cooperative search is a type of information retrieval in which participants work together to satisfy information needs. Previous research has shown that the most common goal is trip planning. Previous research includes studies of cooperative search performance and user behavior, studies of conversational agents, and studies of utterance exchange and arousal in chats. Based on the results of these previous studies, we hypothesized that cooperative search using generative AI would be more efficient in terms of cooperative search by reducing chit-chat. Specifically, we conducted a user experiment to investigate the impact of generative AI on the process and outcome of cooperative search, and to clarify how to make cooperative search smoother.
The experiment was conducted in pairs, and the task was trip planning. The experiment was conducted under two conditions: one in which the participants made a trip plan using only the search engine, and the other in which they used both the search engine and ChatGPT. The time limit was 60 minutes. One computer was provided per pair, and participants were asked to fill out the created travel plan on the travel plan form that was distributed to them. The travel planning task was divided into four steps: determining the destination, determining the schedule, determining and reserving accommodations, and reserving transportation, and participants were instructed to work through each step. The evaluation focused on task completion time, satisfaction based on a questionnaire survey, and task completion as judged by the travel plan. The use of ChatGPT during the task was also examined.
Analysis of the representative values for each of the indicators showed that fewer respondents found the task difficult because they were able to find information more easily using ChatGPT. It was also found that the amount of conversation between subjects was lower when ChatGPT was used. In the procedure-by-procedure analysis, there were some differences in basic statistics, but no statistically significant differences were found.When we examined the use of ChatGPT, the most common use of ChatGPT was to suggest ideas. The most common use of ChatGPT was for scheduling. Furthermore, this study examined the optimal use of the tool in each step of the trip planning task, based on the participants' free responses and the experiment. As a result, for determining the destination, we suggested using ChatGPT to generate destination ideas, followed by using a search engine to collect visual data. In determining the itinerary, we propose a method of letting ChatGPT determine the general framework of the itinerary and repeatedly asking for improvements. In determining and booking accommodations, we suggest using search engines as the primary method. For booking transportation, we conclude that search engines are more appropriate for checking times and making reservations.
In this study, no clear effect of generative interactive AI on cooperative search tasks was observed, but differences were found in some indicators. Future directions include testing under different task requirements, such as extending the time limit.
(Translated by DeepL)