MemeMind framework: Leveraging Large Language Models for Meme Classification and XAI
Summary
In the digital age, social media has become a vital platform for human interaction, reflecting a broad spectrum of ideas, emotions, and perspectives. Internet memes, as a prevalent form of digital content, have evolved into sophisticated vehicles for cultural expression, social critique, and emotional resonance. However, the complex interplay of visual and textual elements in memes presents unique challenges for sentiment analysis, emotion detection, and hate speech identification.
This thesis explores the state-of-the-art in meme analysis and classification, proposing a novel framework called MemeMind. The framework incorporates a multi-modal pipeline that leverages the LLaVa model to generate detailed meme descriptions and performs emotion-related classification tasks from the Memotion 2 dataset via prompting a large language model (LLM). The pipeline serves as the foundation for implementing various explainable AI (XAI) methods using LLMs to enhance trust in the system's decisions.
The MemeMind framework is evaluated through a two-part process. First, benchmarking tests are conducted on the Memotion 2 dataset to assess the performance of the pipeline in classifying emotion-related categories and offensiveness in memes. Following this, various XAI methods are developed, including descriptions, counterfactual comparisons, and an interactive chatbot. These XAI methods, along with a control group that only displays the AI's classifications, are then evaluated in a user study utilizing a moderator interface. The user study aims to assess the effectiveness of the different XAI methods in enhancing user trust and satisfaction with the AI-assisted content moderation system for internet memes.
The analysis reveals a significant difference in the total satisfaction score between the group using the interactive assistant and the group provided with only the AI's classification, with participants using the interactive assistant reporting significantly higher satisfaction. A significant difference in wariness towards the moderation system is found among all groups, with the group provided with AI-generated descriptions reporting higher levels of wariness compared to the group using counterfactual comparisons.
Furthermore, the analysis of user deviations from the AI's advice across different XAI groups showed intriguing results. While no significant differences were found between the groups in terms of deviating from the AI's offensive meme predictions, users in both the interactive assistant group and the counterfactual comparison group were significantly more likely to deviate from the AI's non-offensive advice compared to those in the group without XAI. This suggests that both the interactive assistant and counterfactual comparison XAI methods may have encouraged users to critically evaluate the AI's decisions and exercise their own judgment more frequently when the AI predicted a meme to be non-offensive. However, the possibility that the counterfactual information (present in the interactive assistant and counterfactual comparison group) might have confused participants, leading them to misinterpret or misapply the AI's advice in the context of their task, cannot be ruled out without further investigation.