For decades community ecology has examined empirical relationships between ecosystem productivity and diversity. Despite this long history, tests of hypothesized mechanisms, namely the interplay between environmental filtering, biotic interactions, and dispersal, are lacking, largely due to their intractability using traditional approaches. Across a productivity gradient in a serpentine grassland in California, USA, we coupled occupancy data for four annual plants with persistence measures of paired transplants under natural conditions and reduced biotic interactions with neighbors. We found a positive relationship between productivity and biodiversity (i.e., the proportion of our four focal species found in a location) despite strong competition limiting species persistence in productive environments. Additionally, across species and for the community, we found a strong mismatch between occupancy and persistence, largely due to dispersal excess. Our results suggest that biodiversity-productivity relationships can be largely driven by dispersal and its interactive effects with local biotic and abiotic conditions.