Purpose In validation studies to assess algorithms identifying subjects’ outcome status, researchers often use a sampling method with the ”all possible cases” assumption that all true cases in the database are included in the sample. No study has quantitatively assessed how the extent of missing true cases might bias the estimated performance measures. This study aims to quantify the magnitude of biases and propose a sensitivity analysis method. Methods We first formulate the bias in each performance measure under the violation of the assumption. Using these bias formulas, we propose a sensitivity analysis method to quantify the magnitude of biases and compute bias-corrected estimates. Also, we proposed a sampling approach that is helpful for sensitivity analysis. Finally, as motivating examples, we apply our proposed method to the data from two validation studies that evaluated the performance of case-finding algorithms created by medical claims data. Results We showed that the violation of the assumption does not bias positive predictive value ( PPV ), while it leads to overestimated sensitivity ( Se ) , specificity ( Sp ), and negative predictive value ( NPV ). Our bias formula and example indicate that Se varies greatly depending on the missed true cases, while Sp and NPV are relatively robust under rare outcome situations. Conclusions The deviation from the assumption provides overestimated validation measure values except PPV . This implies a risk of misleading researchers into overestimating the performance of the algorithms. Our proposed sampling option would be useful to investigate whether the assumption is violated and reliably determine the upper limit of the sensitivity parameter.