Abrupt vegetation shifts indicate sudden and profound changes in ecosystem structure and function, which may lead to excessive pressures on regional water resources and thus undermine sustainability. In the context of ecological restoration and climate change, it is unclear whether large-scale abrupt vegetation shifts have occurred on the Loess Plateau and what are the differences and mechanisms associated with the effects of these vegetation changes on soil moisture content (SMC). In this study, we found that approximately 27.9% of grasslands and 24.8% of forests on the Loess Plateau experienced positive abrupt vegetation shifts from 2000 to 2020, which may be associated with the implementation of the Grassland Ecological Compensation Program (GECP) and the Natural Forest Conservation Program (NFCP). By using the climate zone method and multi-period difference-in-differences (MDID) model, we found that the effects of abrupt vegetation shifts on SMC were heavily dependent on local climate conditions. While most abrupt vegetation shifts could wet surface soil (0–7 cm), only 55.9% of grasslands and 33.9% of forests in abrupt shifts had positive effects on entire soil (0–289 cm), specifically in areas with lower precipitation and stronger evaporation conditions. In contrast, approximately 35.7% of grasslands and 37.6% of forests in abrupt shifts may exacerbate soil drying. This study fills the knowledge gap in the effects of abrupt vegetation shifts on SMC and tries to explore the underlying mechanisms, with implications for water management and ecological programs in areas with similar climate conditions.