| Sumario: | Cyber security is an arms race between the bad guys and phishing protection systems, as the former continue to seek ways to obfuscate URLs. Machine learning techniques are highly effective in detecting phishing URLs, but it remains to be seen how they will hold up under attack. In this paper we report baseline testing of phishing protection systems for URL obfuscation. We developed a quantitative set of experiments using a list of URL features from the PhiUSIIL data set. We tested four machine learning models (Logistic Regression, Random Forest, Extreme Gradient Boosting (XGBoost) and Multi-Layer Perceptron (MLP) using baseline test data (a mix of good and bad URLs) and synthetic test data (a mix of good and bad URLs with medium obfuscation, including URLs with subdomain injection, character replacement and encoding). We found all models performed extremely well on baseline (better than macro F1 score of 0.99), but less well on obfuscated. We found Random Forest and XGBoost models were more robust, attaining more than 40% of original performance, while Logistic Regression and MLP were less robust (less than 20%). This is a trade-off between performance and robustness. This work shows the importance of robust models for phishing detection systems and provides a method for evaluating model robustness to inform the development of robust models for detecting phishing under attacks.
|
|---|