Sorghum is known as camel among the crops, grown worldwide for food, fodder, and fuel. However, sorghum is highly susceptible to low temperature stress, which greatly affects seed germination, seedling vigor, root architecture, level of cyanogenic glycosides, fertility, and grain yield. Adaptation to low temperature is very crucial for achieving desirable yields under temperate conditions. Here, an association mapping study was conducted using the large global sorghum diversity germplasm accessions to delineate the genetics of early season cold (ESC) and late season frost (LSF) tolerance in sorghum. A total of 10 single nucleotide polymorphisms (SNPs) and 17 quantitative trait loci (QTLs) were identified for ESC and 40 SNPs were identified for LSF. Two ESC tolerance QTLs identified from our study were co-localized with the classical tannin genes Tan1 and Tan2. This study identified probable candidate genes: Sobic.001G157100 (NPH3), Sobic.001G156600 (Lectin receptor-like serine/threonine kinase), and Sobic.006G061100 (SnRK1 gamma sub-unit) as for ESC tolerance and Sobic.006G139900 (UDP-glucoronosyl and UDP-glucosyl transferase), Sobic.002G187400 (Serine/threonine protein kinase), and Sobic.004G333700 (Ca2+/calmodulin-dependent protein phosphatase) for LSF tolerance in sorghum. The identified candidate genes were known to play a major role in seed germinability under cold stress and involved in plant signal transduction and regulation of cold and other biotic and abiotic stresses in crop plants.