Federal agencies in the US must evaluate the environmental justice implications of regulatory actions. Environmental justice analyses frequently use demographic tests to determine whether regulated projects will disproportionately impact vulnerable communities, including American Indian communities. Demographic tests typically yield negative results, which are often cited as evidence of no environmental justice implications. However, susceptibility of demographic tests to false negative errors is unknown. In these cases, false negative errors occur when a test cannot identify a vulnerable population concentrated disproportionately within a project study area. We developed a technique to evaluate the susceptibility of demographic tests to false negative errors. We used the technique to assess a test commonly used by regulators to permit fossil fuel pipelines. The Atlantic Coast Pipeline served as a case study. The demographic test did not identify disproportionately large American Indian populations under any realistic scenario, a false negative error rate of 100%. In our case study, the test did not detect a disproportionately large American Indian population until the study area contained a four times greater fraction of American Indians than the reference area. We extend the results to study the test’s performance throughout the US. The test’s inability to detect disproportionately large American Indian populations calls into question the validity of negative results and the general ability of the test to inform conclusions about environmental justice or sustainability. We recommend abandoning the test in favor of more rigorous methods.