Towards multi-modal, multi-party virtual dialog systems

Abstract

During the last two decades, our ability to model, render, and animate realistic-looking virtual characters has increased immensely. In contrast, progress in modes of interaction with these characters remains stagnant. For an ideally realistic method of interaction, we look towards robust modeling of conversation. However, existing methods tend to suffer from three major drawbacks: they (1) support a limited number of communication modalities, (2) are heavily geared towards one-on-one type interactions, and (3) are often built to serve in narrowly specialized scenarios. To overcome these drawbacks, we propose two tools that would aid in the development of generally applicable, multi-modal, multi-party dialog systems. The first is a system for the management of communication between actors, and the second a baseline framework for the development of dialog-related agency systems. To demonstrate their effectiveness, we combine them in a sample system inspired by social practice theory, and employ it in a virtual couples-therapy scenario as a proof-of-concept.