This paper develops a detailed mathematical framework for anomaly detection using machine learning, focusing on optimization theory, statistical learning, and computational analysis. We evaluate three key approaches: Support Vector Machines (SVM), Autoencoders, and Isolation Forests, then propose a hybrid model integrating autoencoder-based feature extraction with SVM classification. Through rigorous mathematical proofs and numerical experiments, we analyze the strengths and weaknesses of each approach. Performance is evaluated using the KDD99 dataset, showcasing significant improvements in detection accuracy and efficiency. Code snippets are included for reproducibility, and results are visualized through multiple tables and graphs.