Automatic Classification of Legal Violations in Cookie Banner Texts

Abstract

Cookie banners are designed to request consent from website visitors for their personal data. Recent research suggest that a high percentage of cookie banners violate legal regulations as defined by the General Data Protection Regulation (GDPR) and the ePrivacy Directive. In this paper, we focus on language used in these cookie banners, and whether these legal violations can be automatically detected. We make use of a small cookie banner dataset that is annotated by five experts for legal violations and test it with state-of-the-art classification models, namely BERT, LEGAL-BERT, BART in a zero-shot setting, and BERT with LIWC embeddings. Our results show that none of the models outperform the others in all classes, but in general, BERT and LEGAL-BERT provide the highest accuracy results (70%-97%). However, even these best performing models are influenced by the the unbalanced distributions in the dataset.