Density dependence is a fundamental ecological process. Patterns of animal habitat selection and social behaviour are often density-dependent and density-dependent traits should affect reproduction and survival, and subsequently affect fitness and population dynamics. The Ideal Free Distribution and Optimal Foraging Theory present distinct predictions about how the effect of habitat selection on fitness differs across a population density gradient. Using a social ungulate (Rangifer tarandus) as a model system, we test competing hypotheses about how (co)variance in habitat specialization, social behaviour, and fitness vary across a population density gradient. Within a behavioural reaction norm framework, we estimated repeatability, phenotypic plasticity, and phenotypic covariance among social behaviours and habitat selection to demonstrate the adaptive value of these phenotypes across a population density gradient. In support of Optimal Foraging Theory, but not the Ideal Free Distribution, we found that at high density habitat specialists had higher fitness than generalists, but were also less social than habitat generalists, suggesting the possibility that specialists were inhibited from being social. Our findings illustrate that social strength and habitat specialization varied consistently among individuals across a density gradient, but that habitat specialists maximized fitness at high density. Taken together, our study provides preliminary support for Optimal Foraging Theory as the driving mechanism for density-dependent habitat specialization.