Cannabinoid receptor 1 (CB1R) is widely expressed in central auditory system and play important roles in synaptic plasticity and sensory processing. However, the function of CB1R in specific neuronal subtypes in the central auditory system is largely unclear. In the current study, we investigated whether CB1R deficiency in the parvalbumin (PV)-expressing interneurons, a major class of GABAergic interneurons, affect hearing function. We fist systematically examined the neuronal localization and distribution of CB1R in mice central auditory system using double-label immunofluorescence and confocal laser scanning microscopy, and found that CB1R showed a wide distribution in the central auditory system, especially highly expressed in the cochlear nucleus (CN), superior olivary complex (SOC) and lateral lemniscus (LL). Furthermore, we established a CB1R conditional knockout mice specifically in PV interneurons, and measured auditory function using the auditory brainstem response (ABR) test. Surprisingly, analysis of ABR indicated that conditional deletion of CB1R specifically from PV interneurons significantly elevated the physiological hearing threshold, prolongated the latency of I waves, and decreased the amplitudes of I–V waves. Collectively, these results indicate that CB1R is highly expressed in CN and SOC, as well as deleting CB1R specifically from PV interneurons resulted in partial hearing loss and abnormal brainstem response. Our finding provides an anatomical basis for further investigating CB1R’s function in auditory system, and suggest that CB1R expression in inhibitory PV interneurons is essential for hearing function.