IoT cyberattacks are becoming more frequent and complicated, threatening individuals and organizations. IoT networks are vulnerable to internal and external cyberattacks because to their openness and self-configuration. DoS attacks are particularly destructive, stopping genuine users from accessing key services. Traditional anomaly detection approaches fail to identify complex temporal correlations and are inaccurate and not robust. This study introduces a feature extraction-based VGGNet model for time series anomaly detection using the Artificial Butterfly Optimization (ABO) algorithm for feature selection and a hybrid Capsule Network (CapsNet) deep learning model for accurate attack classification. VGGNet extracts hierarchical temporal features to improve representation quality, whereas ABO effectively picks the most relevant features to reduce computing cost. The hybrid CapsNet classifier captures spatial and hierarchical connections among selected characteristics to improve anomaly detection accuracy. Experimental results on MSL and PSM time series datasets show high classification accuracy, reduced false alarms, and improved precision-recall metrics, exceeding conventional methods. This scalable, adaptable approach detects anomalies in real time, enabling deep learning-driven cybersecurity solutions.