The core purpose of a recommender system is to provide users with personalized recommendations, not just popular items. The current expert recommendation training paradigm in community Question Answering (CQA) scenarios, i.e., fitting a recommendation model is the loss function of the user’s behavioral data is lowered, making the model biased towards experts with higher popularity. This produces the dreaded Matthew effect where popular experts are recommended more frequently and become more popular. The problem of expert recommendation in existing CQA mainly focuses on the perspective of improving recommendation accuracy, while traditional recommender systems solve the popularity bias problem with inverse propensity weighting (IPW). However, the IPW method is highly sensitive to the weighting strategy, and the sparse interaction data in real scenarios causes estimating the propensity score to become difficult. In this work, we consider the intrinsic cause-cause relationship of the popularity bias problem in a new perspective. We find that the expert popularity bias mainly exists in the direct influence of expert nodes on the ranking scores, i.e., the intrinsic attributes of the experts are the reason why the expert recommender system gives too high ranking scores. To mitigate the popularity bias, it is an effective approach to answer the counterfactual question, i.e., what would be the ranking scores assuming that the model uses only expert attributes. To this end, we innovatively developed a causal diagram to illustrate the causal relationships among the factors in the expert recommendation process. During training, we utilize multi-task learning to quantify the contribution of each node to the final ranking score; during testing, we mitigate the popularity bias problem by constructing a counterfactual ranking score that removes the popularity of experts through counterfactual reasoning. It is important to note that our proposed scheme only achieves its purpose by modifying the traditional recommended learning process and is not specific to a specific model, so it can be easily implemented in existing methods. We demonstrated this experimentally on the latest six models in the current CQA. It is verified on the real data set of Zhihu question-answering community?, and the experiment shows that the algorithm can effectively mitigate the influence of popularity bias, while ensuring that the recommendation performance does not fluctuate greatly.