The present study evaluated the inter-rater reliability of the Heads Up Checkup (HCU), a brief digital mental health and behavioral adaptive screening system designed for use in primary care and diverse school settings. Two independent licensed clinical psychologists reviewed a random sample of thirty (N=30) HCU clinical screening reports of 13-14 year old adolescents drawn from a larger sample (N=846) enrolled in a public middle school in California. Results showed strong inter-rater agreement (Fleiss kappa = .93) between clinician ratings and the screener’s priority risk index (HPI) in identifying students “in crisis.” In addition, clinicians’ ratings of confidence in their priority judgments were found to be significantly higher for the “in crisis” cases. Reasonable evidence of convergent validity emerged due to a strong relationship between clinician ratings of psychological distress and the HPI. Overall findings suggest that as an online universal school-based screener, the HCU has valid utility for identifying young adolescents “in crisis” which can translate into timely interventions and pragmatic real-world therapeutic solutions. Future research directions with respect to the refinement of the HCU’s measurement characteristics and its feasibility as an online screener at the population-level in schools are discussed.
