Given the increasing complexity in the modern semiconductor integrated circuit manufacturing process, a variety of defects may occur in each process step, which would eventually lead to loss in wafer yield. The complication and heterogeneity of defect morphologies add great challenges for in root cause analysis. In addition, because the defect data of scanning electron microscope(SEM) images is not easy to obtain, there is no public large data set, which brings difficulties to the training of the algorithm. In this study, we introduce a novel UNet architecture that integrates deep residual networks and an attention mechanism for the segmentation of wafer SEM images. The proposed methodology adopts an encoder-decoder structure, and adds an intermediate attention module (IAM) to enhance features using residual attention mask blocks (RAMBs). To validate the efficacy of the proposed RA-UNet model, a real dataset of defect SEM images in a foundry was manually collected and labeled. The results demonstrate that the proposed model achieves an Intersection over Union (IoU) of 71.11%, providing empirical evidence for the effectiveness of the segmentation approach in the analysis of wafer defect SEM images.