The interaction between corals and algae is of great significance in maintaining the health of coral reef ecosystems. Globally, tropical coral reefs are being degraded by human activities, resulting in the decline of reef-building corals and an increase in macroalgae. The study aims to determine the ecological mechanism of coral–algal phase shifts occurring in the coastal waters of Wenchang in Hainan Province, which are affected by human activities. The field survey results revealed that 177 species of reef-forming corals belonged to 49 genera and 18 families, which were dominated by coral taxa Favites, Porites, Galaxea, and Montipora. Furthermore, there existed obvious competition between dominant corals and algae, especially Lobophora, which showed stronger competitive advantages compared to other algae. The salinity, NO2−, and NH4+ were the key environmental drivers that affected macroalgal abundance. There was a negative correlation between macroalgae and live coral cover and a positive correlation with coral mortality. Notably, the dominant algal species Lobophora had a significant negative correlation with the dominant coral taxa Favites, Galaxea, and Montipora. Further correlation analysis showed that significant interspecific competition existed among corals and algae themselves, which may function as the internal driving factor for the phase shifts of coral–algal relationships. These results will help us understand the role of different functional groups of algae in the degradation of coral reef ecosystems and lay the research foundation for the development of scientific and rational coral reef protection strategies.