Automatic building segmentation from high resolution remote sensing images is currently a research hotspot. In recent years, deep learning, particularly CNNs (convolutional neural networks), has shown promising results. However, the state-of-the-art models currently available still struggle with boundaries, sharp edges, and complex geometry shapes. In order to address these challenges, we propose Boundary and Edge Aware U-Net (BEAU-Net), a boundary and edge attention network that is based on UNet. BAM focuses on the features with rich boundary information. It enhances the models' capability to differentiate between buildings and their surroundings, specially useful in urban regions where buildings are concentrated in proximity. EAM emphasizes refined features such as sharper edges and complex geometric shapes. It enables the model to capture complex geometrical building architectures and detect abrupt changes in edge details. We have conducted extensive experiments on the Massachusetts Building, WHU Building, and INRIA Aerial Image Labeling Dataset to validate the effectiveness of our framework. The results demonstrate significant improvement in the IoU and F1 scores, especially where narrow boundaries and complex shapes are present.