Mapping housing stock characteristics from drone images for climate resilience in the Caribbean

Abstract

Comprehensive housing stock information is crucial for informing the development of climate resilience strategies aiming to reduce the adverse impacts of extreme climate hazards in high-risk regions like the Caribbean. In this study, we propose an end-to-end workflow for rapidly generating critical baseline exposure data using very high-resolution drone imagery and deep learning techniques. Specifically, our work leverages the segment anything model (SAM) and convolutional neural networks (CNNs) to automate the generation of building footprints and roof classification maps. We evaluate the cross-country generalizability of the CNN models to determine how well models trained in one geographical context can be adapted to another. Finally, we discuss our initiatives for training and upskilling government staff, community mappers, and disaster responders in the use of geospatial technologies. Our work emphasizes the importance of local capacity building in the adoption of AI and Earth Observation for climate resilience in the Caribbean.

Impact Statement

AI and drone technologies have immense potential to enable data-driven decision-making in support of resilient housing and infrastructure initiatives across climate-vulnerable regions like the Caribbean. In this work, we have developed an end-to-end pipeline for automatically generating building footprints and roof classification maps using computer vision and drone images in support of government-led climate resilience programs. By leveraging innovative technologies and building local capacity, government agencies are better equipped to not only respond to disasters but also anticipate risks and mitigate impacts, thereby increasing climate resilience in the region.

1. Introduction

The Caribbean is among the world’s most climate-vulnerable regions due to the prevalence and intensity of extreme climatic hazards such as tropical cyclones, floods, and landslides. In recent years, Category 5 hurricanes like Dorian, Irma, and Maria have devastated many small island developing states (SIDSs), leaving widespread trails of loss and destruction across the region. SIDS often bear the brunt of the economic costs from climate-extreme events, with the highest degree of losses typically sustained in the housing sector (Government of the Commonwealth of Dominica, 2023). Hurricane Maria, for example, damaged over 28,000 homes—roughly 90% of Dominica’s housing stock—and accumulated costs over 200% of the nation’s GDP (Government of the Commonwealth of Dominica, 2023). Moreover, extensive damage to structural assets from natural hazards can bring about numerous challenges, such as precarious housing conditions, building collapse, and loss of life. Postdisaster housing recovery, which involves retrofitting damaged buildings, reconstructing public infrastructure, and relocating displaced populations, can result in severe budget cuts and increased debt, further straining already fragile economies (Government of the Commonwealth of Dominica, 2023). As global temperatures continue to rise, extreme weather events will only grow in severity, putting many more vulnerable populations at risk.

The loss and devastation brought about by super typhoons have spurred ambitious climate resilience programs by national government agencies aiming to reduce the adverse effects of extreme weather events in the housing sector (Government of the Commonwealth of Dominica, 2023; World Bank, 2022). For instance, the Disaster Vulnerability Reduction Project (DVRP) by the Government of Saint Lucia (GoSL) seeks to measure and reduce vulnerability to climate change impacts through institutional strengthening and retrofitting of structural assets (Government of Saint Lucia, 2023). Another example is the Resilient Housing Scheme by the Government of the Commonwealth of Dominica (GoCD), which strives to make 90% of Dominica’s housing stock resilient by 2030 through the construction of 5000 new resilient homes for vulnerable citizens (Government of the Commonwealth of Dominica, 2023). For these climate adaptation initiatives to be successful, local governments require accurate, complete, and up-to-date housing stock information to inform pre- and postdisaster retrofitting, reconstruction, and relocation strategies. However, the traditional house-to-house approach to identifying high-risk structures is often cost-prohibitive for many developing countries, prompting the need for more timely and cost-efficient alternatives.

In response to these challenges, researchers have turned to artificial intelligence (AI) and Earth observation (EO) to quickly and efficiently distil meaningful information from large unstructured geospatial data (World Bank, 2022; Triveno et al., 2019). Previous works demonstrate how deep learning (DL) techniques can be used to extract building footprints and their corresponding characteristics from high-resolution aerial images (Partovi et al., 2017; Castagno & Atkins, 2018; Solovyev, 2020; Buyukdemircioglu et al., 2021; Kim et al., 2021; Huang et al., 2022; Tingzon et al., 2023). More recent years have seen a growing interest in applying DL to drone images and leveraging their very high spatial resolutions to generate more accurate and granular building stock information (Triveno et al., 2019; Akbari et al., 2021; Calantropio et al., 2021; Takhtkeshha et al., 2022; Fujita & Hatayama, 2023). Yet despite the evident advantages of using AI and drone technologies to support climate resilience strategies, the widespread adoption of these solutions is hindered by gaps in the local capacity to develop and maintain systems for generating critical baseline exposure datasets.

This work aims to bridge these gaps by providing government agencies with an end-to-end workflow for rapidly generating critical baseline housing information using very high resolution (VHR) drone images. Specifically, we leverage computer vision (CV) models such as segment anything model (SAM) and CNNs for building footprint delineation and rooftop classification, respectively. We also evaluate the cross-country generalizability of roof classification models across SIDS to determine the extent to which models trained in one country can be adapted to another. Finally, we underscore the importance of strengthening local capacity through ongoing efforts to promote the adoption of AI and EO-based solutions among national and regional government agencies. This work is done under the Digital Earth Project for Resilient Housing and Infrastructure, a World Bank project funded by the Global Facility for Disaster Reduction and Recovery (GFDRR), in partnership with GoCD and GoSL, in support of government-led initiatives to enhance the climate resilience of the housing and infrastructure sector in Caribbean SIDS.

2. Data

In this section, we detail our process for generating our ground truth housing stock datasets using VHR aerial images in the form of aircraft- and drone-derived optical imagery and building footprints for Dominica and Saint Lucia, our primary regions of interest.

2.1. VHR aerial images

We acquired the following VHR aerial images from various data sources, including partner government agencies GoCD and GoSL, the World Bank Global Program for Resilient Housing (GPRH) (World Bank Global Program for Resilient Housing (GPRH), 2023), and the open data platform OpenAerialMap (OAM) (OpenAerialMap, 2023):

• • Aircraft-derived nationwide orthophotos of Dominica (2018–2019) and Saint Lucia (2022)

• • Pre- and postdisaster drone images of 10 villages in Dominica, namely Colihaut, Coulibistrie, Delices, Dublanc, Kalinago, Laplaine, Marigot, Pichelin, Roseau, and Salisbury (2017–2018)

• • Drone images of 3 districts in Saint Lucia, namely Dennery, Castries, and Gros Islet (2019)

The spatial resolutions of aircraft-derived images range from 10 to 20 cm/px, whereas the resolutions of drone-derived images range from 2 to 5 cm/px. Table 1 further details the spatial resolution, coverage, year of acquisition, and number of annotated buildings per aerial image used in this study.

Table 1. Aircraft- and drone-derived aerial images used in this study (World Bank Global Program for Resilient Housing (GPRH), 2023; OpenAerialMap, 2023; Government of Saint Lucia, 2023; Government of the Commonwealth of Dominica, 2023)

2.2. Building footprints data

For the aircraft-derived orthophotos, we obtained nationwide building footprints in Dominica and Saint Lucia in the form of vector polygons delineated from the corresponding aerial image, as provided by the World Bank. For the drone images, we initially looked to alternative data sources such as OpenStreetMap (OSM) (OpenStreetMap Buildings, 2023), Microsoft Building Footprints (Microsoft Building Footprints, 2023), and Google Open Buildings (Google Open Buildings, 2023); however, we observed significant misalignment between the publicly available building footprint polygons and the underlying drone images, as illustrated in Figure 1. In response to this issue, we turned to SAM to delineate building instances directly from drone images (Kirillov et al., 2023). Additional information on the SAM configuration is detailed in Section 3.

Figure 1. Building footprint polygons from (a) Microsoft Building Footprints, (b) Google Open Buildings, (c) OpenStreetMap Buildings, and (d) SAM superimposed on an OAM drone image taken in Salisbury, Dominica (OpenAerialMap, 2023).

2.3. Rooftop image tiles

We generated our dataset for roof classification by selecting 80 randomly sampled 500 m x 500 m tiles across Dominica and Saint Lucia and removing tiles with no building footprints using the datasets described in the previous section. We further augmented these tiles with the villages, cities, and districts where drone images were available (Tingzon et al., 2023). For each building footprint within the selected tiles and regions, we extracted the minimal bounding rectangle of the building polygon from the corresponding aerial image, scaled by a factor of 1.5 and zero-padded to a square. We then proceeded to annotate 9150 buildings in Dominica and 5175 buildings in Saint Lucia via visual interpretation of the aerial images for a total of 14,325 labeled building footprints across the two countries. We classified the buildings based on two attributes: (1) roof type, which includes hipped, gabled, flat, and no roof, and (2) roof material, which includes healthy metal, irregular metal, blue tarpaulin, concrete or cement, and incomplete rooftops. The class distributions for each roof attribute are presented in Table 2. Examples of drone image tiles for each roof type and roof material category are illustrated in Figure 2.

Table 2. Class distributions for roof type and roof material labels across Dominica and Saint Lucia

Figure 2. Examples of VHR drone-derived roof image tiles for each of the roof material categories (top row) and roof type categories (bottom row).

3. Methods

This section outlines our workflow for generating critical housing stock information in the Caribbean using drone images, as summarized in Figure 3.

Figure 3. Proposed workflow for the automatic generation of housing stock information from drone images using DL models.

3.1. Building footprint delineation

To extract building footprints from drone images, we leveraged the Segment-Geospatial Python package for segmenting raster images based on SAM (Wu & Osco, 2023; Osco, 2023; Osco, 2023; Kirillov et al., 2023). We also used Language SAM (LangSAM) (Medeiros, 2023) Python package to combine instance segmentation with text prompts to generate masks of specific objects in the drone images. In our study, we set our text prompt to “house,” the box threshold, that is the threshold value used for object detection in the image, to 0.30, and the text threshold, that is the threshold value used to associate the detected objects with the provided text prompt, to 0.30. In practice, these parameters can be tweaked as necessary to achieve the best results. As a postprocessing step, we applied the Douglas–Peucker algorithm to simplify the generated building polygons, as implemented in GeoPandas (Douglas & Peucker, 1973).

3.2. Roof classification

Given the VHR aerial images and corresponding building footprint polygons, we proceeded with developing our roof classification models. We began by fine-tuning CNN models pretrained on the ImageNet dataset (Deng et al., 2009) with architectures ResNet50 (He et al., 2016), VGG16 (Simonyan & Zisserman, 2015), Inceptionv3 (Szegedy et al., 2016), and EfficientNet-B0 (Tan & Le, 2019) using cross-entropy loss for both roof type and roof material classification tasks. The input rooftop image tiles are zero-padded to a square and resized to 224 x 224 px for ResNet50, EfficientNet-B0, and VGG16 and 299 x 299 px for InceptionV3. We set the batch size to 32 and the maximum number of epochs to 60, and we use an Adam optimizer with an initial learning rate of $ 1{e}^{-5} $ , which decays by a factor of 0.1 after every 7 epochs with no improvement. For data augmentation, we implemented horizontal and vertical image flips with a probability of 0.50 and random rotations ranging from $ -90{}^{\circ} $ to $ 90{}^{\circ} $ . Given that our data is imbalanced (see Table 2), we also implemented random oversampling for the minority classes. To prevent overconfident predictions, we applied label smoothing as a regularization technique, with smoothing set to 0.1 (Müller et al., 2019).

3.3. Model evaluation

We split each country-level dataset into training (80%) and test (20%) sets using stratified random sampling to preserve the percentage of samples per class, as shown in Table 2. Due to the high costs associated with acquiring nationwide aircraft-derived orthophotos, we recommend government partners leverage drone technologies to collect VHR aerial images; thus, for our experiments, the test sets for Dominica and Saint Lucia comprise entirely of drone images. Additionally, to test whether geographically diverse training data improves the prediction, we combine the training sets across Dominica and Saint Lucia (henceforth referred to as the “combined” dataset).

We report the F1 score, precision, recall, and accuracy using standard definitions as follows:

(3.1) $$ \mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}} $$

(3.2) $$ \mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} $$

(3.3) $$ \mathrm{F}1\;\mathrm{score}=\frac{2\cdot \mathrm{Precision}\cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}} $$

(3.4) $$ \mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}} $$

where TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives, and FN is the number of false negatives. The precision, recall, and F1 score metrics are computed as the unweighted mean of the metrics (that is macro-averaged) calculated per class.

4. Results and Discussion

For roof type classification, we achieve the highest F1-score of 87.1% for Dominica using an EfficientNet-B0 model and 89.5% for Saint Lucia using a ResNet50 model. Likewise, for roof material classification, we achieve our best score of 89.5% for Dominica using a ResNet50 model and 91.7% for Saint Lucia using an EfficientNet-B0 model. Among the models trained on the combined datasets from Dominica and Saint Lucia, our highest scores of 90.0% and 90.4% were achieved using an EfficientNet-B0 model for roof type classification and an Inceptionv3 model for roof material classification, respectively. Note that the latter models were evaluated using the combined test sets of Dominica and Saint Lucia. Table 3 details the complete F1-scores, precision, recall, and accuracy scores, and Figure 4 illustrates sample model outputs in the form of roof material classification maps using drone images taken before and after Hurricane Maria in Coulibistrie, Dominica.

Table 3. Test set model performance scores (%) in terms of F1-score (F1), precision (P), recall (R), and accuracy (Acc) of CNN architectures for roof type and roof material classification trained using (a) only Dominica data, (b) only Saint Lucia data, and (c) using the combined datasets of Dominica and Saint Lucia

Figure 4. Drone images (left) and roof material classification maps (right) of Coulibistrie, Dominica taken before (top) and after (bottom) Hurricane Maria in 2017. Roof categories include healthy metal (green), irregular metal (red), concrete/cement (yellow), blue tarpaulin (blue), and incomplete (purple).

Lastly, we investigate the cross-country generalizability of the best models by evaluating their performance on the designated test sets of each country. As shown in Table 4, our results indicate that for roof type classification, the best model trained on the combined data from Dominica and Saint Lucia (that is the “combined model”) performs marginally better than models trained using only local data (that is data from the same country); however, for roof material classification, we find that locally trained models consistently outperform the combined model. These results also indicate that local models trained on the Dominica subset are more robust to geographical shifts than local models trained on the Saint Lucia subset, as local models for Saint Lucia appear to generalize poorly in the Dominica context. We posit that this is likely due to differences in class distributions between Dominica and Saint Lucia. Specifically, the training data for Dominica is derived primarily from postdisaster aerial images taken in the aftermath of Category 5 Hurricane Maria, meaning approximately 15% of the training data are comprised of blue tarpaulins typically used to cover severely damaged rooftops. Meanwhile, the aerial images for Saint Lucia were taken in nondisaster contexts, and thus, the Saint Lucia dataset is completely devoid of blue tarpaulins (see Table 2). More generally, the high levels of variability in local model performance for out-of-distribution countries indicate the importance of collecting highly localized and contextualized training data. Further research on domain adaptation is needed to reduce performance degradation in the face of geographic distribution shifts.

Table 4. Cross-country generalizability in terms of F1-score (F1), precision (P), recall (R), and accuracy (Acc) of the best roof type and roof material classification models as shown in Table 3

4.1. Building local capacity in Caribbean SIDS

The overarching goal of the Digital Earth for a Resilient Caribbean Project is to strengthen local capacity in Caribbean SIDS to harness AI and EO technologies in support of resilient housing operations. The project has three main components:

1. 1. Capacity building. This project supports the training and upskilling of government staff, academics, and other key stakeholders in the use of EO data, tools, and services.

2. 2. Operational support. Our team aims to support the generation, integration, and use of critical baseline exposure datasets using AI and EO-based technologies.

3. 3. Knowledge sharing. Through knowledge exchange and dissemination, we strive to promote our methods and tools to encourage wider adoption across the Caribbean region.

To this end, our team is assisting government agencies in establishing geographic information systems (GISs) units capable of generating, managing, and maintaining large-scale disaster risk datasets. In partnership with technical experts and local stakeholders, we are designing training programs on geospatial data analytics, community mapping, drone operations, pilot coordination, image processing, machine learning, and large-scale geospatial data management (Humanitarian OpenStreetMap Team (HOT), 2023). Additionally, we have developed educational resources to enable the rapid generation of baseline exposure datasets aimed at local government staff, community mappers, and disaster responders. This includes executable Google Colaboratory notebook tutorials ^{1 , 2} demonstrating how to run the SAM and CNN models to quickly extract building footprints and create roof classification maps from locally collected drone images.

5. Conclusion

This work proposes an end-to-end workflow for filling critical baseline exposure data gaps using VHR aerial images and DL techniques in support of government-led climate resilience initiatives in the Caribbean. Specifically, we demonstrate how CV and drone images can be used to rapidly generate housing stock information, including building footprints and roof type and roof material classification maps, for disaster risk reduction and recovery. Based on our evaluation of the cross-country generalizability of DL models, we urge caution in applying locally trained models off the shelf to new geographic regions and emphasize the significance of collecting local, highly contextualized training data. We also highlight the importance of local capacity building, skills development, and cocreation of geospatial datasets in deploying sustainable AI-for-climate solutions, especially in Global South contexts. Through this work, we hope to empower government agencies to strengthen the local capabilities needed to sustainably generate AI and EO-derived housing stock data to better inform climate resilience programs in the Caribbean.