Recent machine learning models have shown great success for weather prediction tasks, suggesting that atmospheric dynamics can in principle be learned from data. However, such approaches suffer from instability or drift in long integrations and are hence usually not suited for climate simulations. Incorporating physical knowledge promises to alleviate such shortcomings. Here, we present PseudospectralNet (PSN), an architecture for a hybrid atmospheric model that combines a quasigeostrophic physics-based dynamical core with a data-driven core based on an UNet. Our architecture transforms between grid and spectral space at every time step and therefore mimics the pseudospectral solution approach many intermediate-complexity atmospheric models follow. Neural networks are separately defined in the spectral and grid space and are combined with physics-based dynamical cores in each of these spaces. We train PSN separately on data from quasigeostrophic models, primitive equation models, and reanalysis data. Adding the physics-based dynamical core to our model helps both short-term predictability and long-term numerical stability of the hybrid model.