The architecture underpinning genomic divergence is still a largely uncharted territory and likely case-dependent. Here, we investigated genome-wide variation in Ballan wrasse, a northeastern Atlantic fish species that displays two sympatric color morphs, spotty and plain, that have been suggested to represent subspecies. We produced a chromosome-level reference genome, and thereafter investigated genomic divergence among 152 individuals including both morphs, from two localities in Spain and Norway each, and one in France. Differences between morphs dominated in Spain in accordance with sympatric divergence, whereas in Norway geographic divergence was highest supporting allopatric differentiation. Chromosomes had very large low-recombining areas that were shared across populations and have accumulated further divergence. Within the Spanish morphs, large islands of divergence covered ~11% of the genome, showed high morph-specificity, and strong selection. The same regions showed frequent admixture in the French morphs and no differentiation in Norway. In contrast, divergent regions observed between sampling localities in Norway were scattered, shorter and found throughout the genome. High inbreeding and lower diversity were observed in the Norwegian samples, consistent with the proposed recolonization bottleneck and subsequent drift. Several genomic regions were significantly associated with morphs and contained tens of genes of diverse functions, suggesting that coloration is unlikely to be the sole driver of divergence. Our results suggest that large, polygenic divergence islands were initially formed and preserved in the south but were gradually lost and uncoupled from the phenotype of the fish further north when these areas were repopulated after the last glacial maximum.