Overview

Dataset Introduction

In this article we present a new dataset (AI4Shipwrecks) and benchmarking results for segmentation of shipwrecks from side scan sonar imagery collected on an AUV. This dataset was collected over the span of 5 weeks in Thunder Bay National Marine Sanctuary (TBNMS) in Lake Huron, Michigan, USA. Spanning 4,300-square-miles, TBNMS contains almost 100 known shipwreck sites and over 100 undiscovered sites. We leverage the unique abundance of known shipwreck sites to curate a rich dataset for investigating the application of machine learning for this task. In this paper, we present details of dataset collection and preparation for easy indexing of our dataset by machine learning pipelines. To encourage further advances in segmentation for side scan sonar, we report extensive results on our dataset from modern, state-of-the-art segmentation models. Lastly, we include discussion on lessons learned from our field expeditions and experiments to provide insight on future challenges and opportunities for machine learning for processing sonar data. The resulting dataset and benchmarking tools will be made publicly available as a benchmark dataset for segmentation from sonar imagery to enable future research in machine learning for ocean exploration.

Methodology Overview

Site Selection

Surveys were conducted in Thunder Bay National Marine Sanctuary (TBNMS) in Lake Huron, MI. The figure below shows shipwreck sites that were imaged in our surveys during four weeks over the course of two years. The abundance of known shipwreck targets in TBNMS was a crucial factor in our selection of this field site, as this enabled us to maximize the number of targets observed within a relatively constrained area and short timespan. These proposed survey regions were selected in coordination with scientists at TBNMS to cover a wide range of ship types, site relief, wreck characteristics, and water depth, prioritizing sites within a reasonable distance from the port of Alpena. This ensured that we could survey a maximum number of sites while still capturing variation across samples, providing a unique and valuable dataset for training machine learning methods. Furthermore, this large and diverse dataset allows us to validate and thoroughly evaluate the accuracy and generalizability of developed methods.

Side-scan Sonar

A side-scan sonar (SSS) emits fan-shaped low and high-frequency acoustic pulses from two transducers aimed at the seabed floor. The signal travels through the water column and reflects off the terrain or other objects in the swath area before being received by the sensor. After a sonar chirp has been emitted, the sensor receives and bins the intensity of returns according to time-of-flight. On the horizontal axis, an SSS image is a histogram of return intensity at equally spaced intervals in time. Each histogram is accumulated in the vertical axis as the transducer moves to produce an image with two dimensions. The higher the echo signal intensity, the higher the pixel value in the resulting sonar image. Raw SSS imagery is single channel grayscale but can be viewed with various color palettes for improved visibility.

Although side scan sonar produces high resolution images of the environment, view-dependent shadowing effects, self-occlusion, material-dependent acoustic noise, and distortion make object detection a difficult task for both humans and automated algorithms. The figure below illustrates common distortions and noise found in side scan sonar imagery.

Strided Cropping

As survey lengths can vary for each mission, the resulting sonar images are of different lengths. the dataset provides full-sized images, but we square-cropped these images to be input to deep neural networks. Please check our paper for more details.

Labeling Policy

Pixel-wise labeling of the post-processed sonar images was conducted by a team of three researchers with shared labeling guidelines. The labels were then reviewed in-depth by a marine archaeologist from the State of Michigan who is an expert on the shipwrecks at TBNMS. There are two labels: shipwreck and other. shipwreck consists of the primary wrecks as well as any debris. If part of the ship is obscured in an acoustic shadow, the label was extrapolated into the shadowed region to follow the expected shape of the wreck based off of expert knowledge. The labels are exported to a standard binary mask format where 0 represents the other class and 1 represents the shipwreck class.

Evaluation

The segmentation metrics presented in the paper are calculated according to the PASCAL VOC paper. IoU, F1 score, true positive rate, and true negative rate are calculated from total true positive, true negative, false positive, and false negative counts for each test image, and then averaged across all images.

Dataset Organization

The dataset is organized by the Side-scan Sonar (SSS) images and their corresponding labels. We provide the labels as segmentation masks of objects belonging to a shipwreck in the SSS image, including the shipwreck itself and any debris associated with the shipwreck. This includes parts of the shipwreck that may have come off from the shipwreck and parts of the ship that has been dislodged. The images and labels are separated into test and train sets using a 50/50 split. We select a 50/50 split to ensure that the test set is large enough to capture the variance of the shipwrecks sites, ensuring that the robustness of the dataset can be evaluated through our dataset. In addition, the images are grouped on a per-site basis. This is done to ensure that SSS images that see the same terrain, shipwreck, or debris are not shared between the test and train sets of our dataset.

Dataset File Structure

The dataset file structure is shown in the figure below.

Please note that we have re-named files show in the paper containing “NellieGardner” to “Exploratory2”.

Dataset Helper Scripts

We provide helper scripts to load the dataset and visualize the dataset here.