Defaunation of large-bodied animals threatens essential ecosystem functions, such as seed dispersal. However, the impact of this ‘downsizing’ of animal communities on plant-frugivore trait matching—the alignment of frugivory-related plant traits (e.g., fruit size) with frugivore traits (e.g., body mass)—remains poorly understood at macroecological scales. Here, we investigate how human disturbance and environmental conditions influence trait matching in plant-frugivore networks across the tropics. We compiled data on fruit size and frugivore body mass for 1,927 plant and 1,120 animal species and integrated these with 12,708 interaction records across 102 networks. Using fourth-corner analyses and structural equation models (SEMs), we assessed how human disturbance and environmental conditions affected trait matching strength across networks. SEMs revealed that these factors influenced trait matching strength through indirect pathways: human disturbance weakened trait matching by reducing the range of frugivore body masses within networks, whereas wet and productive ecosystems, where the proportion of fruit in frugivore diets is higher, supported stronger trait matching. Our synthesis shows that plant-frugivore trait matching is a widespread macroecological pattern that is weakened by human disturbance, particularly through the defaunation of large-bodied animals. Human activities are thereby decoupling the coevolved relationships between fruiting plants and their animal seed dispersers.