In distributed passive localization and tracking system, the track observed by the subsystem seems like Brownian motion track, because the tracked target is non-cooperative target and its maneuver is often complex, and the localization accuracy is poor. These track characteristics will seriously disturb track association between different subsystems. In order to solve this problem, the track to track association algorithm based on empirical mode decomposition (EMD) is proposed in this letter. Each dimension data of the track from each subsystem is processed by EMD and the high frequency components are eliminated. A track motion trend (MT) vector is created by collecting the rest components of the processed data. For these vectors, the corresponding rule is constructed, in which the association threshold is self-adaptive, and the assumption of the target motion model is not required. The simulation results show that the proposed algorithm can accomplish track association effectively in passive localization systems.