Directional data analysis plays a critical role in paleomagnetism, where observations are distributed in a spherical surface. Existing methods for analyzing directional data often fail to incorporate prior physical information about plate geodynamics, significantly constraining their potential. To address this limitation, we developed a hybrid, physics-informed machine learning model that uses a neural network to learn the underlying rotations responsible for generating directional data. We found that our method efficiently fits both synthetic and real paleomagnetic datasets. Additionally, by leveraging differentiable programming, we can incorporate physical constraints in the form of regularizations. These results could enhance future estimations of apparent polar wander paths, advancing the reconstruction of past tectonic plate motions.