Understanding the distinction between signs in sign language and gestures has been a topic of interest among researchers over the past decade. Recent arguments emphasize that sign languages exhibit syntactic, semantic, morphological, and phonological features while gestures are considered external to language. This theoretical distinction is importantly reflected at the brain level, with distinct brain regions being involved in sign language comprehension and gesture processing. A recent EEG study using the Fast Periodic Visual Stimulation (FPVS) design, for example, has shown heightened discrimination responses to number signs in deaf signers compared to hearing non-signers. However, it remains unclear whether these findings are related to deafness or sign language knowledge. The present study aimed to address this question by comparing, with a similar FPVS design, the brain responses of hearing signer adults to number signs and number gestures with those of hearing non-signers. Results showed more salient discrimination responses to number signs in hearing signers compared to hearing controls, while similar neural responses were found for number gestures among both groups. Our findings shed light on the role of deafness and sign language knowledge in shaping neural responses to linguistic and non-linguistic numerical representations.