Specific emitter identification (SEI) techniques utilize the physical-layer fingerprints embedded in their received signals to identify the unique emitter. Fingerprints originate from hardware impairments during transmitter manufacturing. Considering the different manifestations of fingerprints in different domain perspectives, a novel neural operator with the time and frequency domain attention mechanism is proposed, which is named the complex Fourier neural operator (CFNO). And to further enhance the individual discriminability, the demodulation reconstruction waveform and the synchronized waveform are converted into a new sample format, named the paired sample. With the paired sample, the proposed network acquires prior communication knowledge and a baseline of the ideal transmitter. In this work, different network structures are investigated. And performances of the paired sample are fully evaluated. Compared with existing SEI methods based on deep learning models, results show the effectiveness of the CFNO structure, as well as the proposed paired sample scheme. The significance of this paper is that the proposed framework fully utilizes the domain knowledge in the communication field, resulting in excellent performance improvement.