There have been numerous calls to promote reproducible research. This growing awareness coincides with major advances in data/code sharing technologies. Yet authors, journals, institutions, and funders still need to act to advance more reproducible research. Here, we suggest to view reproducibility as a continuum that includes the 1) availability of data, models, code, and directions to use the digital artifacts, 2) replication of results, and 3) reproducibility of findings. We present a simple survey tool to assess where a peer-reviewed journal article lies on the continuum. We use the tool to assess 360 random sampled articles of the 1,989 articles published in 2017 in six well-regarded hydrology and water resources journals. 49% of sampled articles had some materials available online, but just 5.6% made available all the data, models, code, and directions. For 1.6% of articles, we generated results that replicated some or all of the published results. Assessments took 5 to 14 minutes per article to determine the availability of digital artifacts and 25 to 86 minutes to replicate results (25-75% range). The availability of data, models, code, and directions differed by journal and journal policy towards data availability. From the 360 article sample, we estimate that 0.6% to 6.8% of all articles published in the six journals in 2017 can be replicated using their published artifacts (95% confidence interval). These results suggest several practices to improve the reproducibility of published research. First, authors should provide directions to use their data, models, and code in addition to the digital artifacts. Second, on author submission, journals should use a tool like ours to assess the submission’s position on the reproducibility continuum. Third, journals should formulate policies that require authors to state the intended reproducibility of their work and place relevant information in an easy-to-find article location. Fourth, journals, institutions, and funders should highlight work whose digital artifacts, results, and findings are available, replicable, and reproducible.