Natural fallow practice has been identified as an effective way to overcome obstacles of continuous cropping. However, how the resulting soil microbial changes impact plant fitness, and how the context-specific differences diverge from those caused by continuous cropping remain largely unknown. This study used the third-year continuous tobacco cropping soil (CCS) and natural fallow soil (FS) to cultivate Nicotiana tabacum. The influences of soil microorganisms on the fitness of N. tabacum were assessed by reassembling soil microbial communities. Then, the bacterial and fungal community assembly of the bulk soil and the rhizosphere were characterized using amplicon sequencing and statistical analysis. The results indicated that soil microorganisms play more important roles for plant fitness for N. tabacum grown in FS compared with CCS. Moreover, the abiotic context of FS exerts stronger effects compared with those of CCS for the reassembly of soil microbiomes. Comparative analysis identified the context-specific microbial clades and the differential strength of rhizosphere effects. In conclusion, this paper provides context-specific microbial evidence, which may unravel the potential mechanism underlying the different response of N. tabacum to changes of soil microbiomes induced by natural fallow and continuous cropping practices.