Sex-biased dispersal potentially underpins the genetic structural mismatch in genetic structure between different inherited genetic regions. However, accurately evaluating the effect of sex-biased dispersal is challenging due to other sex-biased traits and limitations in usable data. The brown bear (Ursus arctos) exhibits extreme male-biased dispersal. In the Hokkaido population of Japan, three divergent maternally inherited mitochondrial DNA lineages were distributed separately, however the paternally inherited nonrecombining region of the Y chromosome analysis did not confirm this distribution pattern. To investigate how the characteristic genetic structure has been shaped and to evaluate the influence of male-biased dispersal on the process, we conducted a reduced-representation genome sequencing of this population, and used published whole-genome resequencing data. The ratio of genetic differentiation between X-chromosomes and autosomes inferred a significant effect of male-biased dispersal both between and within populations. We observed a slight difference within subpopulations, which could be attributed to restricted gene flow, mainly for female bears because of the lowlands. We estimated a historical trend of the brown bear’s relatively suitable habitat decreasing during the Last Glacial Maximum (ca. 21,000 years ago), which could have caused a historical population decline followed by a recent rapid expansion primarily driven by male bears. These results indicate that the Hokkaido population had contracted despite the appearance of the land bridge between Eurasia and Hokkaido Island and has expanded rapidly while being limited by the lowlands. We emphasize the importance of the contribution of male-biased dispersal contributing to migration into new habitats and rapid population colonization.