Exploring the use of large language models to improve one-to-one communication training scenarios

Abstract

This paper explores the automation of communication training scenario writing by using AI generated statements. Two different research questions were explored, the first attempts to improve the quality of the training scenarios by replacing so-called non-functioning “Distractor” options. These options are pitfalls to make a user think more about the best practice response in a certain scenario. Distractors were first evaluated where non-functioning distractors were identified and replaced by AI generated statements. The original and generated distractors were implemented in communication training scenarios which were used in a bachelor course. The differ ences in how often these options were chosen were analyzed using a Mann Whitney Utest. There were no significant differences found, although the outcome was very close to the 0.05 significance cutoff with a p-value of 0.057. For the second research question we attempted to generate statements based on the desired parameter set tings. The output was evaluated by experts who concluded the results are promising, but not ready for automation without human evaluation. The expert grades of the AI generated options were significantly worse than the rating of the human written dis tractors meaning more work has to be done before fully automated parameter based answering option generated is viable