Addressing the User Experience Challenges in LLM Powered Chatbot through Prompt Engineering and Conversational

Abstract

Large Language Models (LLMs) are increasingly integrated into tools for knowledge work, yetusers often struggle with unclear expectations and ambiguous communication when interacting withthese systems. This study explores how prompt engineering and conversational strategies impact userexperience in LLM-powered chatbots. A two-part within-subject experiment using Rabobank’s AIAssistant for internal employees examined the effects of varying system prompts and response styles.Results show participants prefer the system that used moderately detailed prompt over the minimalor overly complex ones, but the user-centric metrics are statistically non-significant. Conversationalstrategy preferences were context-dependent: Immediate Answers were favored for straightforwardnessand clarity, while Clarifying Questions and Chain of Thought approaches were valued in complexor ambiguous tasks. The findings suggest further research into system prompt engineering usage as amean for developers to reduce users’ cognitive load, and highlight adaptive, context-aware interactiondesign in improving the usability of AI systems. While the study focuses on IT professionals, theoverall findings can be generalized for a wider audience.