Which natural language inference problems are hard for neural models?

Abstract

Last decade the interest in natural language inference has increased because it serves as a task to test AI models on natural language understanding. This resulted in several models with new state-of-the-art performance. While overall accuracy on different benchmarks has been increasing steadily, little research is done on specific problem types that are hard to solve. This paper explores different characteristics of the inference problems, resulting in problem types that are hard to solve for models based on certain architectures or trained on specific data set.