Context-Aware Hybrid YOLOv8–Vision Transformer Framework for Real-Time Weapon and Suspicious-Behavior Detection in Intelligent Surveillance

Abstract

Automated surveillance systems that rely exclusively on object detection or behavioral analysis can provide incomplete threat assessment because the presence of a dangerous object and the behavioral context of a scene represent distinct security cues. This study presents a hybrid intelligentsurveillance framework that integrates YOLOv8-based weapon detection with Vision Transformer
(ViT)-based suspicious-behavior analysis for real-time early threat detection. Live video is processed through two complementary branches: YOLOv8 performs object-level weapon localization, while a ViT-Base/16 model analyzes visual context to distinguish normal from anomalous or suspicious activity. The resulting evidence is integrated through the implemented decision-level OR rule, enabling an alert when either weapon evidence or suspicious behavioral evidence is detected. The behavior-analysis branch, evaluated using UCSD surveillance data, achieved a reported 88.6% accuracy, 87.9% precision, 86.4% recall, and 87.1% F1-score. The YOLOv8 weapon detector achieved 91.2% detection accuracy, 90.5% precision, and 92.1% recall, corresponding to a derived F1-score of approximately 91.3%, while operating at approximately 30 FPS. The integrated prototype reported an alert delay of less than 2 s. The results demonstrate the feasibility of combining localized physical-threat evidence with transformer-based contextual analysis in an end-to-end surveillance pipeline, while further temporal, robustness, and cross-dataset validation remains necessary before large-scale deployment.