Fooling news misinformation classifiers by generating adversarial attcks without alterning the semantics of the text

Abstract

This research looks at the robustness of four different machine learning models (Naive Bayes, SVM, feed-forward neural network, and LSTM), applied to a true or false news classification task. The four classifiers were trained on the same training set but tested on both a regular and altered test set. The altered test set had words replaced with synonyms to investigate if the classifiers were susceptible to semantics-preserving alterations. Naive Bayes based true or false news classification was not able to perform well enough to say it acquired an “understanding” of the content, thus no unambiguous answer could be given, regarding Naive Bayes. SVM and the feed-forward neural network classifiers showed no differences in scores and thus are likely to be insusceptible to the few alterations made. Long short-term memory, however proofed to be susceptible to the alterations and should thus not be implemented as an automatic news classification system, for the classification can be altered without altering the semantics of the input. Even though the SVM and feed-forward neural network classifier’s scores remained unchanged, more research is needed to give definitive answers, regarding the classifiers.