Taxonomy Construction with Multi-Critic Reinforcement Learning

Abstract

This thesis investigates the application of multi-critic reinforcement learning to taxonomy construction. Using multiple critics in complicated environments has been shown to improve overall performance on a plethora of reinforcement learning tasks. Guiding the learning process of the critic by explicitly constraining it to a small subset of the task is an effective way to speed up learning and ensure stability during training. Combining several of these constrained critics into a centralized critic outperforms single critic methods on a variety of different tasks. Multi-critic algorithms are especially effective when there is an underlying structure of the task with clearly defined sub-tasks that can be evaluated independently. In this work, we introduce a multi-critic algorithm for taxonomy construction, where we use two critics to evaluate the choice of the parent and child words at each step. Through a series of experiments, we demonstrate that the critics have learned to effectively identify the source of error in incorrect actions, which was not possible with previous methods. We also demonstrate the robustness of our model by analyzing the consistency of the structure of its generated taxonomies.