To evaluate the diagnostic performance of artificial intelligence (AI) in detecting root canal orifices using images captured with a dental operating microscope (DOM). A total of 80 human maxillary first and second molars were included in the study. After preparing traditional access cavities, root canal orifices were identified under a dental operating microscope (DOM) at 21.25x magnification. To ensure accurate identification, the number of root canal orifices was cross-verified by analyzing axial CBCT images. Following orifice identification, video recordings were obtained using the DOM, from which a total of 1,527 frames were randomly selected for analysis. The root canal orifices in these frames were manually labeled using CranioCatch labeling software (CranioCatch, Eskişehir, Turkey). A segmentation model for root canal orifice detection was developed using the YOLOv8x model and implemented with OpenCV, PyTorch, NumPy, Pandas, TensorBoard, and Seaborn libraries. A confusion matrix was employed to assess the model’s diagnostic performance by comparing predicted outcomes with actual observations. In the binary classification task, the system correctly identified 502 out of 526 root canal orifices, yielding an accuracy of 91%. There were 24 false negatives and 24 false positives. For the specific identification of the mesiobuccal 2 (MB2) canal, the algorithm detected MB2 in 63 out of 70 images, resulting in an accuracy rate of 80%. However, it missed MB2 in 7 images (7 false negatives) and misclassified 9 images, with surface irregularities mistaken for MB2 (9 false positives). The YOLO-based CNN demonstrated high accuracy and sensitivity in detecting root canal orifices from DOM images. This study highlights the potential of AI algorithms for real-time clinical assistance and their possible role in enhancing the training of dental students.