Assessing Adversarial Vulnerabilities in Fake News Detection: A Comparative Study of GPT-2 and BERT Variants
Abstract
The increasing sophistication of fake news dissemination poses a growing threat to digital information integrity, demanding the deployment of robust and intelligent detection systems. Transformer-based language models—particularly BERT, RoBERTa, DistilBERT, and GPT-2—have shown promising results in detecting misinformation by leveraging deep contextual understanding. However, their vulnerability to adversarial attacks reveals a critical weakness in their deployment for real-world applications. This study conducts a comprehensive evaluation of these models under both standard and custom adversarial attack scenarios to assess their reliability in detecting manipulated or misleading content. Using the "newsmediabias/fake_news_elections_labelled_data" dataset, we fine- tune each model and subject them to a battery of adversarial techniques, including TextFooler, PWWS, BAE, DeepWordBug, TextBugger, as well as novel attack methods designed specifically for this study: Enhanced Substitution Attack (ESA) and Comprehensive Text Attack (CTA). We analyze model behavior in terms of accuracy degradation, perturbation efficiency, and computational cost. Our findings reveal stark contrasts in model robustness: while RoBERTa maintains the highest performance on clean data, it—along with other models—is significantly compromised under even subtle adversarial manipulations. The study highlights GPT-2's limitations as a generative model repurposed for classification, as it fails catastrophically under most attack conditions. These insights underscore the urgent need for adversarial resilience in fake news detection systems and pave the way for future research focused on integrating robust defense mechanisms into transformer-based architectures.
Cite this paper
Published in
