We introduce a deep learning-based hybrid beamforming (HBF) strategy for millimeter-wave transmission systems, specifically addressing the challenges posed by phase noise of local oscillators. Our approach utilizes a deep neural network to optimize precoding and combining matrices based on channel state information. We incorporate the symbol index through an adaptive attention mechanism and employ a self-supervised learning approach with a phase-noise-aware loss function to mitigate the effects of phase noise.While primarily focused on phase noise, our method also accommodates other practical constraints, such as limited-resolution phase shifter and imperfect channel estimation. Simulation results demonstrate that our design outperforms traditional and deep-learning based HBF methods in terms of data rate both in scenarios impacted only by phase noise and compounded distortion scenarios including low-resolution phase shifters and channel estimation errors.