Aiming at solving the problem that the capability of direction of arrival (DOA) method for estimation of nested arrays decreases sharply under unknown mutual coupling, an iterative weighted low rank matrix reconstruction (IW-LRMR) algorithm is proposed in this paper. Firstly, the received data covariance matrix is extended, and a weighted low-rank matrix recovery problem is formulated for joint estimations of the Toeplitz matrix and mutual coupling coefficients. Next, the DOA from recovered Toeplitz matrix is retrieved by using the root multiple signal classification (Root-MUSIC) algorithm. Experimental results show that IW-LRMR algorithm can effectively avoid the influence of mutual coupling while solving the grid mismatch problem so that the accuracy and the resolution of estimation are improved.