We present a statistical study of large magnetic field vector residuals between Swarm observations and the 13th generation International Geomagnetic Reference Field (IGRF-13) model under quiet to moderate geomagnetic conditions. Since the vast majority of data are taken during these conditions, statistics of these large residuals are important for satellite operation when using IGRF-13 as reference, as well as for magnetosphere-ionosphere-thermosphere (MIT) studies. Residuals with magnitude of vector differences between observations and model estimations larger than 300 nT under Dst >-50 nT are studied from 2014 to 2020. All large residuals appear in the high-latitude auroral zone region peaking around 70° (-70°) magnetic latitude (MLAT) in the northern hemisphere (southern hemisphere) with also a secondary occurrence peak just below 80° (-80°) MLAT. However, the two hemispheres show clear asymmetries in the magnetic longitude (MLON) distribution where both hemispheres show high concentration of large residuals around the geographic poles. Since polar satellite’s orbits give rise to highly biased number of observations around the geographic poles, it results in a decrease in occurrence rate with respect to the total number of measurements. Identifying the source of large residuals under quiet to moderate geomagnetic conditions is helpful to better separate out geophysical current signatures from non-geophysical large residuals that arise due to hemispheric differences in the location of the IGRF-13 geomagnetic pole compared to the geographic pole. We suggest that near geographic pole observations of large residuals are not geophysical, but due to mapping uncertainties of the IGRF-13 geomagnetic coordinates to geographic coordinates.