The most reliable way to develop poverty maps is through household surveys. However, in many developing countries, economic data from household surveys are infrequent. As a result, a data gap has emerged, making it impossible to identify and understand the locations of poor people. New approaches that integrate household survey data with non-traditional data sources (such as satellite imagery, call records, Wikipedia, google street view) using machine learning allows for improved resolution and scale in poverty mapping. Nevertheless, these studies developed poverty maps for study areas located in the same geographical region and did not map the location of poor people at the finest level (building level). Integrating income level from household survey data and building footprints from OpenStreetMap data with high-resolution satellite imagery, we extract the building rooftop images and classify them into two classes as poor(< = $5.50/day) and non-poor(> $5.50/day) based on the international poverty lines given by the world bank. We use these rooftop images as training data, develop a deep learning classification model, and estimate whether a building is poor or non-poor. We use physical factors like building area, elevation with rooftop images for transfer learning to contribute to the model’s accuracy. We attempt to build a versatile model that maps the locations of poor and non-poor people at the building level by developing, calibrating, and validating the model for Myanmar, Thailand, and Nicaragua study areas. Our findings show that for the model to travel well intra-regionally and inter-regionally, the study areas should be from the same geographical regions with similar roof types and percentages of people living in poverty. Combining the rooftop with income level is a fitting parameter to measure poverty, and roof color plays a crucial role along with the roof texture, shape to increase the model’s accuracy. The proposed methodology helps develop poverty maps for different income levels ($1.90, $3.20, $21.70, or any desired income level) from limited household survey data. This research study identifies the location and the total number of poor households living in a specific region which helps plan for effective poverty reduction, urban planning, and disaster prevention tailored to local conditions.