Numbers keep the spam out: Robustness of word- and character-level SMS spam filters to obfuscation and good-word attacks

Abstract

Spammers routinely disguise messages to evade statistical filters, by obfuscating words with look-alike characters, digits or inserted punctuation, or by appending words typical of legitimate messages. This study measures how six standard spam filters, namely naive Bayes, logistic regression and linear support vector machines on word or character n-gram features, withstand such evasion on the public SMS Spam Collection of 5,572 real text messages. With five random train–test splits, spam in the test set was attacked with leetspeak, homoglyph substitution, character insertion and typographical errors applied to 10–100% of its words, both randomly and targeted at the most spam-indicative words, and with good-word attacks appending 5–20 words typical of legitimate messages. On clean data the filters reached F1-scores of 95–97% with false-positive rates below 0.7%. Random obfuscation of half of the words reduced spam recall by at most 7.3 percentage points, and some obfuscations even increased the recall of character-level naive Bayes. The main reason for this robustness was not the models but the messages: 78.3% of spam but only 0.1% of legitimate messages contained a number of five or more digits, and such non-lexical cues survived word obfuscation. When the filters were restricted to alphabetic words, clean recall hardly changed (92.6–93.6%), but targeted character insertion of half of the words reduced the recall of character-level models to 10–12%. Good-word attacks were the most effective practical attack, reducing recall to 49–71% with 20 appended words. Text normalisation restored recall under character insertion (from 87.8% to 95.0% for character-level logistic regression), and adversarial training with obfuscated spam raised it to 99.9%, but neither defence helped against good-word attacks, and adversarial training made them more effective. Clean accuracy therefore hides a strong dependence on non-lexical cues, and robustness evaluations of spam filters should include both targeted obfuscation and good-word attacks.