The research on fatigue damage accumulation holds significant importance for the safety and reliability of mechanical structures. This study introduces an innovative approach to fatigue damage prediction by combining machine learning (ML) with physical mechanism, aiming to improve prediction accuracy, particularly with small datasets. A novel ML framework is proposed, incorporating a customized loss function that seamlessly integrates ML techniques with physical mechanism. This method improves model performance, tackles limited data challenges, and achieves faster convergence and higher accuracy than traditional ML models. The results demonstrate that embedding physical mechanism into ML models significantly boosts the accuracy of fatigue damage predictions, even when the training dataset is reduced by 30%. This work underscores the potential of hybridizing physical knowledge with ML to improve predictive capabilities and robustness, making it a powerful strategy for accurately predicting residual fatigue damage in scenarios with limited data.