In this paper, a dual closed-loop active disturbance rejection control (ADRC) scheme based on the swarm intelligent technique is investigated for quadrotors to address several challenging control problems. First, a composite control system is designed to guarantee a satisfied quadrotor flight. Then, an improved particle swarm optimization (IPSO) algorithm with constriction factors, simulated annealing (SA) and adaptive jump strategy is developed to enhance its capacity of escaping from the local optimal trap and thoroughly exploring the search space. Simultaneously, the IPSO algorithm is employed to obtain the optimal parameters of ADRC controller. Comparative simulation and platform experiments show that the proposed IPSO-based ADRC controller can satisfy the control requirements of the inner-outer loop well.