State-of-the-art deep neural networks (DNNs) for inverse scattering are typically trained for fixed arrangements of transmitters and receivers, restricting their applicability to setups with fixed viewing angles. Consequently, any change in these angles necessitates retraining the network. One way to avoid retraining is to train the DNN on all possible combinations of viewing angles for varying numbers of views, but this can be tedious. Instead, we employ a more efficient approach that uses two U-Nets. First, the scattered field from each view is independently processed through a shared U-Net to produce view-wise contrast reconstructions. These view-wise reconstructions are aggregated and then refined using the second U-Net to obtain the final reconstruction. Using two different sets of contrast profiles-one with circular cylinders and the other from the MNIST dataset-we demonstrate that this approach achieves good contrast reconstruction for arbitrary viewing angles and an arbitrary number of views. However, performance decreases on rotated MNIST images, as this dataset is not rotation-augmented. While data augmentation through image rotations is one solution, we propose training the U-Nets on data transformed into polar coordinates. We show that this method accommodates arbitrary views and is robust to rotated versions of the contrast profiles without relying on data augmentation.