Background De-labeling suspected adverse drug reactions (ADRs) is a critical goal in clinical practice, as misdiagnosed allergies lead to unnecessary medicalization and unwarranted restrictions on available medications. In children, high-risk factors for ADR, such as immediate urticaria, often overlap with common viral rashes, complicating risk stratification. This challenge is compounded by caregiver-reported histories, which can distort symptom timing and severity. This case-control study evaluated the reliability of caregivers and the influence of non-clinical factors in predicting the risk. Methods Patients and caregivers referred to the Allergy Units of two tertiary care hospitals for suspected hypersensitivity reactions (HSRs) were included. Questionnaires were offered to caregivers and patients aged 7-17. Participants were divided into two groups: patients with suspected HSRs and controls without a history of HSR. Enrolled subjects were asked to recognize pictures of maculopapular, micropapular, and urticaria via a multiple-choice questionnaire. Results Data from 405 participants were collected between September 2023 and August 2024. Statistical analysis showed that a higher caregiver education and a positive personal or familiar history of HSR and/or urticaria significantly improved recognition accuracy. Nevertheless, over 50% of participants with suspected HSR history misidentified urticaria with micropapular rash, underscoring the low reliability of caregiver reports. In addition, rash recognition skills did not differ significantly between healthcare and non-healthcare caregivers. Conclusions These findings highlight the need for systematic approaches beyond descriptive histories. Integrating non-clinical factors into diagnostic frameworks could enhance risk stratification and support safer and more effective allergy workup for de-labeling suspected HSR in pediatric patients.