This paper investigates the distributed mean square output tracking problem of heterogeneous multi-agent systems with Markovian switching topologies and infinite transmission delays. The main challenge of the concerned problem arises from how to deal with Markovian switching topologies and infinite transmission delays simultaneously. A novel distributed observer is developed based on a newly proposed Lyapunov functional method. Then, a distributed controller based on the distributed observer is developed. It is shown that the stochastic distributed tracking problem is solved in the sense of mean square if the union graph of the underlying Markovian switching topology contains a spanning tree. A distinctive feature of the proposed controller is that the infinite delays are not required to be known. Finally, the effectiveness of the proposed controller is illustrated by two numerical examples.