Complex civil structures require the cooperation of many building materials. However, it is difficult to accurately monitor and evaluate the inner damage states of various material systems. Based on a convolutional neural network (CNN) and the acoustic emission (AE) time-frequency diagram, we used the transfer learning method for classifying the AE signals of different materials under external loads. The results show the CNN model can accurately classify cracks that come from different materials based on AE signals. The recognition accuracy can reach 90% just by re-training the full connection layer of the pre-trained model, and its accuracy can reach 97% after re-training the top 2 convolutional layers of this model. A realization of cracking source identification mainly depends on the differences in mineral particles in materials. This work highlights the great potential for real-time and quantitative monitoring of the health status of composite civil structures.