Located at the edge of the Tibetan Plateau, the Qilian Mountains form a significant high-altitude permafrost region. Understanding the stratification characteristics of soil carbon and the response mechanisms under altitudinal variation provides important insights into the carbon pool dynamics across the Tibetan Plateau. Potentilla parvifolia is a shrub species widely distributed at moderate and high altitudes. Recently, with the intensification of global climate change, there have been signs of migration to the high latitudes. This migration affects the microbial community structure and has influence on the soil carbon pools and the carbon cycling processes of ecosystems. In the present study, the rhizosphere soil carbon components and microbial communities of P. parvifolia were evaluated along altitudinal gradients (3204, 3350, 3550, and 3650 m) in the Qilian Mountains. We found that P. parvifolia significantly increased the biomass of soil gram-positive bacteria and fungi at medium and low altitudes (3204, 3350, and 3550 m) (p < 0.05), and significantly increased the contribution of mineral-associated organic carbon (MAOC) to the soil organic carbon (SOC) content when compared to those in bare soil (p < 0.05). Furthermore, P. parvifolia mainly increased SOC content by increasing the contribution of microbial necromass carbon, especially fungal necromass carbon, to MAOC; however, there were no significant differences at high altitudes (3650 m) (p > 0.05). This research enhances our comprehension of the impact of plant migration on SOC and its microbial mechanisms and provides a basis for the development of bioprotection strategies for alpine ecosystem conservation.