Orthodontic technology has been widely applied in clinical practice, but many potential patients don’t pay enough attention to dental health. A complete set of methods for predicting dental malocclusion based on deep learning algorithms is constructed as follows: (1) based on self-collected open-mouth facial images, dataset of mouth together with its keypoints are constructed. Mouth region can be cropped and the correspondent keypoints identified through Haar cascade classifier and YOLOv8-pose algorithm respectively; (2) three-class segmentation of teeth are achieved is improved by self-Building a teeth segmentation dataset and basing on Mask RCNN algorithm, introduced group normalization and dropout regularization, added CA attention mechanism and transfer learning, used the NAG optimizer and focal loss; (3) Extracting single tooth outer contour, main axis, etc. geometric feature parameters, combining mouth keypoint, determining the tooth sequence, single tooth inclination angle and adjacent tooth spacing parameters are used for judged the dental malocclusion state. The experimental results show that the improved instance segmentation algorithm has an accuracy of 94.96%, which is 4.28%, 4.75% and 8.22% higher than Mask RCNN, Yolact and SOLOv2 respectively. The proposed method is suitable for selfie oral image teeth segmentation and dental malocclusion prediction.