Automated Behavior Estimation for Pain Detection in Dogs with Computer Vision

Abstract

Studying dog behavior is a crucial aspect in determining their welfare since their behavior cues reflect their mental state. Studying behaviors in dogs can aid veterinarians in estimating their pain levels. Especially dogs experiencing chronic pain are masters at masking their pain signals. Deep learning models might be able to pick up the pain signals that even experts do not immediately notice. Our starting point is a two-stream network encompassing a ConvLSTM stream for RGB frames and an LSTM stream for processing keypoints to classify pain. We propose adding a third stream that encapsulates an ethogram, representing the dogs' behaviors in the corresponding input frames to improve the baseline two-stream pain estimation model. We established a method that uses the Multi-modal Semantic Query Network (MSQNet) to generate behavior labels for unlabeled data to deal with the problem of annotating an abundance of videos, which is a labor-intensive and time-consuming process. We processed the ethograms using two different methods: a simple linear layer and a random forest classifier, respectively. Analyzing the performance of these methods showed us that the former performed better than the latter, possibly due to the limited amount of training samples available. Additionally, explainability is a crucial aspect of pain detection in dogs, since we want to know why the dogs appear to be in pain. We tested several methods to aid explainability, including saliency heatmaps, Shapley values, Naive Bayes probabilities, confidence scores, and a decision tree. From the explainability measures, we can conclude that the models tend to classify dogs with higher activity into the ``no pain" class, and dogs with low activity into the ``pain" class.