The spatial pattern and community assembly of soil microbial taxa have notable meanings for biodiversity shaping and maintaining mechanisms. Rare fungal taxa may exhibit distinct patterns and assembly mechanisms compared to abundant taxa, but such information is limited, especially at large scales. Here, we investigated distance-decay patterns and underlying assembly mechanisms for abundant and rare fungal taxa in 129 soil samples collected across 4,000 km in Chinese Northern grasslands, based on high-throughput sequencing data. A total of 208 abundant OTUs (relative abundance > 0.1%, 2.73% of entire OTUs) and 5,779 rare OTUs (relative abundance < 0.01%, 75.85% of entire OTUs) were identified. Both abundant and rare fungal taxa showed significant distance-decay relationships (P < 0.001), but the turnover rate for rare taxa (0.0024 per 100 km) was nearly half that of abundant taxa (0.0054 per 100 km) based on the binary Bray-Curtis distance. The lower turnover of rare fungal taxa was likely due to their community assembly mechanism dominated by stochastic processes, which were less influenced by environmental gradients. In contrast, abundant taxa assembly was dominated by deterministic factors like soil variables and plant traits, which varied significantly along the geographic distance. Consistently, rare fungal taxa were also less sensitive to environmental changes, with a lower turnover rate by environmental distance (0.0027 vs. 0.0099) than abundant taxa. In summary, our findings revealed that rare fungal taxa, shaped mainly by stochastic processes, had lower spatial turnover compared to abundant taxa, dominated by deterministic processes, enhancing our understanding of rare microbial biogeography.