Hybrid gnn–llm framework for machine learning based anomaly detection in enterprise email networks: integrating semantic, structural, and reinforcement learning techniques

Enterprise email networks face an escalating volume of sophisticated cyber attacks including spear-phishing, credential harvesting, business email compromise, covert data exfiltration, and multi-stage malware delivery. Traditional signature-based and rule-based detection systems are inadequate against modern adversaries who continuously modify content, employ multi-layer obfuscation, and exploit large language models to generate contextually convincing fraudulent messages. This paper proposes, implements, and experimentally validates a Hybrid Graph Neural Network-Large Language Model (GNN-LLM) Anomaly Detection Framework for malicious email activity in enterprise networks. The framework integrates five complementary detection pillars: (1) semantic representation via a LoRA fine-tuned ModernBERT encoder with mean pooling over all token hidden states, (2) structural modelling via a Graph Attention Network (GAT) operating on token co-occurrence graphs, (3) unsupervised anomaly detection via Isolation Forest trained exclusively on benign email patterns, (4) supervised soft-voting ensemble classification combining SVM, Random Forest, and Logistic Regression with PCA dimensionality reduction, and (5) adaptive decision fusion via a Proximal Policy Optimisation (PPO) agent with asymmetric reward structure and automatic F1-maximising threshold calibration. Evaluated on three publicly available benchmark datasets spanning corporate email, SMS spam, and academic mailing lists, the system achieves 99.90 % accuracy and 100 % recall (zero false negatives across 5,060 test samples) on the Enron corporate email corpus, with only 0.20 % false positive rate. Cross-domain evaluation confirms 95.17 % accuracy on KNUJ SMS and 98.85 % on Ling-Spam without domain-specific retraining. A key empirical finding is that mean pooling over all BERT token representations yields a 50.6 percentage-point accuracy gain over [CLS]-token extraction alone; establishing it as the decisive architectural choice for domain-shifted email classification tasks.