In this paper, we consider the matrix-weighted consensus problems with disturbances. To this end, we firstly propose a new disturbance observer design for systems with unknown matched or mismatched disturbances representable as a linear combination of time-varying basis functions. Under some assumptions on the boundedness and persistent excitation of the regression matrix, the disturbances can be precisely estimated at an exponential rate and thus, can be compensated by a suitable compensation input. Next, disturbance-observer based consensus algorithms are proposed for matrix-weighted networks of single- and double-integrators with matched or mismatched disturbances. We show that both matched and mismatched disturbances can be estimated and actively compensated, and the system globally asymptotically converges to a fixed point in the kernel of the matrix-weighted Laplacian. Depending on the network connectivity, the system can asymptotically achieve a consensus or a cluster configuration. The disturbance-observer based consensus design is further extended for higher-order integrators with disturbances. Finally, simulation results are provided to support the mathematical analysis.