Understanding users' dissatisfaction with chatgpt responses: types, resolving tactics, and the effect of knowledge level

Abstract

Large language models (LLMs) with chat-based capabilities, such as ChatGPT, are widely used in various workflows. However, users often experience difficulties in using this technology and various dissatisfactions. Researchers have introduced several methods, such as prompt engineering, to improve model responses. However, they focus on crafting one prompt, and little has been investigated on how to deal with the user’s dissatisfaction during the conversation. Therefore, we examine end users’ dissatisfaction and their strategies to address it. After organizing users’ dissatisfaction with LLM into seven categories based on a literature review, with ChatGPT as a case study, we collected 511 instances of dissatisfactory ChatGPT responses from 107 users and their detailed recollections of dissatisfied experiences, which we released as a dataset. Our analysis reveals that users most frequently experience dissatisfaction with ChatGPT not grasping intent, while accuracy-related dissatisfactions are the most serious. We also identified four tactics users employ to address their dissatisfaction and their effectiveness. We found that users often do not try to address dissatisfaction, and even when they do, 72% remains unresolved, especially those with low knowledge of LLM. We also found that they tended to put minimal effort into resolving dissatisfaction. Based on these findings, we propose design implications for minimizing user dissatisfaction and enhancing the usability of chat-based LLM.