A color fusion image color harmony assessment framework based on joint features is proposed. In this framework, the fused images are firstly segmented into image patches for evaluation separately. Secondly, the memory color theory is proposed to calculate the color harmony score of image patches. Meanwhile, the topic model is improved to extract global natural statistical features. Finally, a quality assessment module is designed to fuse global natural statistical features and local spatial features to evaluate color fusion images. We compare the results of the proposed method CHA with six existing traditional image quality assessment methods and six deep learning-based methods. Experimental results demonstrate the promising performance of the proposed method.