The ability to predict community assembly, species coexistence, and biological invasions in a changing environment remains a significant challenge. Here, by combining a functional traits, niche and fitness frameworks we derived the following hypotheses: 1) species with analogous traits occupy analogous niches; 2) within each niche, species from the same morphology-based functional group (MBFG) are interchangeable and might cooperate; and 3) successful invaders belong to the resident MBFG. Controlled experiments with phytoplankton communities under contrasting environmental scenarios and the invasion of two bloom-forming toxic cyanobacteria were used to test hypothesis. Fitness was estimated using Tilman’s R*, and functional abilities were evaluated with morphological and metagenomic traits. Environmental filters were primary determinants of community structure, individual traits reflected the species niche and defined invasion success. Warm and eutrophic treatments increased total biomass, in which light limitation allowed filamentous cyanobacteria to dominate. However, different species of the same MBFG alternatively dominated or co-dominated while fundamental functions remained and potential cooperation observed. These findings, in conjunction with the successful invasion of species from the resident MBFG, contribute to a more nuanced understanding of community assembly according to Modern Coexistence Theory in species-rich communities and improve predictions of global impacts on water quality.