THOR

The Earth’s oceans cover over 70% of its surface, though much of this remains unexplored. Current and future data characterizing seafloor habitats is inherently large – accurately discriminating between marine species requires that sensors collect images at very high resolution, very close to the seafloor to yield small imaging footprints with the necessary high detail. Existing photographic seafloor surveys, for instance, have already collected petabytes of field data, and new survey equipment, like that in development by the PIs, substantially increases image resolution and the corresponding data set size.

The new methods produced in this project will enable scientists to create detailed biological and environmental maps of seafloor communities at unprecedented scales and do so repeatedly through time as these habitats change under the influence of human activity. Such maps provide a necessary long-term contextual framework for subsequent sustainability analysis and decision making in critical areas including monitoring the health of marine ecosystems, creating detailed censuses of marine species and resources, tracking species migration and invasions associated with climate change, and monitoring and optimizing marine reserve design.

THOR Technology

THOR is a seafloor acoustic and optical imaging system equipped with sensors was developed by IMT lab scientists and collaborators to capture optimal seafloor data.

Experiment

  • Ocean Opacity: Because the ocean remains opaque to satellite and airborne sensors, collecting and processing equivalent baseline data for seafloor habitats on a landscape or regional scale remains a major challenge. These data are essential for managing marine resources and understanding the long-term impact on marine ecosystems from human activity, such as from commercial fishing, mineral exploration, onshore and offshore constructions, and climate change. Despite the success in automated process of satellite and aerial hyper-spectral images of shallow sub-tidal habitats, the minimum resolving scale of this imagery is 1-10 m in length and 10 m in depth in the clearest water.
  • Manual Taxonomic Classification: Advances are needed in high-performance image classification and search in order to process the data from large-scale seafloor biodiversity and habitat surveys. The analysis of these images still typically involves manual processing by an expert in taxonomic identification, either by tracing identified organisms or by identifying a statistical subset of organism beneath an overlay of random points.

Automatic Content-Based Image Retrieval (CBIR) methods divide the image classification task into two phases:

  • Indexing: Images are characterized by a set of feature vectors that use some form of Scale-Invariant Feature Transform (SIFT) to summarize principal
    image content attributes such as shape, texture, color, and spatial layout.
  • Search Phase: The feature vector of a sample image with unknown content is compared against previously classified images in a classification library. A distance measure between feature vectors is used to find the closest library matches and classify the sample image. Distance measures include classic Euclidean distance, Mahalanobis distance , or robust estimates, residuals, and outlier detection with multi-response data.

Prior to image classification, survey images must be pre-processed through several stepsRead and decompress the image

  1. Enhance image contrast and correct color
  2. Compute red, green, and blue histograms
  3. Compute gray scale image and discrete cosine transform (DCT) coefficients

When repeated over many thousands of images in a survey, the time required for these tasks becomes significant. The project has therefore focused on revising baseline code to perform these tasks faster. Processing tasks have been written in more efficient code (C++) with multi-threaded processing (OpenMP). Performance comparisons below look at the relative performance of the original Matlab code and the revised code running on a single 6-core processor.

Water scatters, refracts, and absorbs light. At greater depths, the reds, oranges, and yellows of sunlight are filtered out, leaving the seafloor illuminated only be a dim blue light. Artificial lighting is essential for underwater photography, but this light still mixes with the bluish light of filtered sunlight, giving images a blue cast. Survey images must be processed to brighten them, improve their contrast, and balance their colors to remove this bluish cast. The images below illustrate this processing with original unprocessed images on the top row, and corrected images below.

Bonaire 2008, Sample 1 Uncorrected
Bonaire 2008 , Sample 1 (uncorrected)
Bonaire 2008, Sample 1 Corrected
Bonaire 2008 , Sample 1 (corrected)
Bonaire 2008, Sample 2 Uncorrected
Bonaire 2008 , Sample 2 (uncorrected)
Bonaire 2008, Sample 2 Corrected
Bonaire 2008 , Sample 2 (corrected)
loading images data

Survey images are stored in large, minimally compressed full-color JPEG image files on disk-based storage. To process the images, they must be loaded from disk and decompressed into pixel arrays. The revised implementation uses the widely-used open source libjpeg. Unfortunately, this remains single threaded and I/O bound. Run times have been reduced 67% from 0.43 sec/image to 0.14 sec/image.

Histogram Image Data

Color is a useful factor in differentiating one species from another. Our species identification method characterizes organism colors by computing a color histogram that lists the colors present and how often they occur. Since a single image could depict a large number of different organisms, each image is first divided into several thousand small overlapping tiles so that each one usually spans only a single organism. The colors in each tile are then tabulated. The revised implementation uses custom tiling and histogram generation for a coarse histogram of red, green, and blue image channels. The new code is multi-threaded compared to the original single-threaded code. Run times have been reduced 99% from 26 sec/image to 0.03 sec/image.

enhance images data

The revised implementation uses custom dynamic range scaling for the red, green, and blue image channels to enhance and correct images. The new code is multi-threaded compared to the original single-threaded code. Run times have been reduced 93% from 0.19 sec/image to 0.01 sec/image.

DCT Processing Image Data

Surface texture is a strong factor when differentiating one species from another. To characterize texture, our method uses a Discrete Cosine Transform (DCT) that breaks an image tile down into a series of overlapping cosine waves that, when summed together, recreate the bright and dark features of an organism’s texture. The revised implementation is based on the open source FFTW image processing library. The new code is multi-threaded compared to the original single-threaded code. Calculations are done using single-precision floating-point math. Run times have been reduced 99% from 38 sec/image to 0.09 sec/image.

In the original Matlab code, histogram generation and DCT coefficient calculations were by far the dominant tasks. These required 26 and 38 sec/image, respectively, compared to just 0.4 and 1.9 sec/image times to load and enhance images.

In the revised implementation, the image load time dominates at 0.14 sec/image compared to 0.01, 0.03, and 0.09 sec/image for enhancement, histograms, and DCT coefficient calculations. The image load remains single threaded and is limited by the cost of I/O to read the source image from disk-based storage.

Overall, the revised implementation’s run times have been reduced 99% from 64.8 sec/image to just 0.27 sec/image. This is about 200x faster.

THOR Parts

Hover or tap to view each part of THOR.

Main Body Frame

The main body frame is located in the center of THOR. The main body controls the wing motors, distribute power, and ensures communication.

Wing

THOR’s wings are located on the left and right side of its body. The wings move up and down, controlling the flow of the water as THOR dives.

Central Brain

THOR’s brain is located in the center, near its body. Parts includes in the central brain are: acoustics, cameras, control systems, and lasers.

Tail Fin

The tail fin provides stability for THOR.

Nose Cone

The nose cone is located at the front of THOR. There is a forward sonar to avoid obstacles and a telemetry modem for communication with IMT Lab scientists.

Seabird Precision

This part is located near THOR’s body to control the temperature and pressure.

Precision Acoustic Altimeter

This part is located and connected to the center of THOR, in its main body.

Light Panel

The light panels are located in the front and back, underneath THOR’s body, providing florescence and broadband.

THOR

The Earth’s oceans cover over 70% of its surface, though much of this remains unexplored. Current and future data characterizing seafloor habitats is inherently large – accurately discriminating between marine species requires that sensors collect images at very high resolution, very close to the seafloor to yield small imaging footprints with the necessary high detail. Existing photographic seafloor surveys, for instance, have already collected petabytes of field data, and new survey equipment, like that in development by the PIs, substantially increases image resolution and the corresponding data set size.

The new methods produced in this project will enable scientists to create detailed biological and environmental maps of seafloor communities at unprecedented scales and do so repeatedly through time as these habitats change under the influence of human activity. Such maps provide a necessary long-term contextual framework for subsequent sustainability analysis and decision making in critical areas including monitoring the health of marine ecosystems, creating detailed censuses of marine species and resources, tracking species migration and invasions associated with climate change, and monitoring and optimizing marine reserve design.

THOR Technology

THOR is a seafloor acoustic and optical imaging system equipped with sensors was developed by IMT lab scientists and collaborators to capture optimal seafloor data.

Experiment

  • Ocean Opacity: Because the ocean remains opaque to satellite and airborne sensors, collecting and processing equivalent baseline data for seafloor habitats on a landscape or regional scale remains a major challenge. These data are essential for managing marine resources and understanding the long-term impact on marine ecosystems from human activity, such as from commercial fishing, mineral exploration, onshore and offshore constructions, and climate change. Despite the success in automated process of satellite and aerial hyper-spectral images of shallow sub-tidal habitats, the minimum resolving scale of this imagery is 1-10 m in length and 10 m in depth in the clearest water.
  • Manual Taxonomic Classification: Advances are needed in high-performance image classification and search in order to process the data from large-scale seafloor biodiversity and habitat surveys. The analysis of these images still typically involves manual processing by an expert in taxonomic identification, either by tracing identified organisms or by identifying a statistical subset of organism beneath an overlay of random points.

Automatic Content-Based Image Retrieval (CBIR) methods divide the image classification task into two phases:

  • Indexing: Images are characterized by a set of feature vectors that use some form of Scale-Invariant Feature Transform (SIFT) to summarize principal
    image content attributes such as shape, texture, color, and spatial layout.
  • Search Phase: The feature vector of a sample image with unknown content is compared against previously classified images in a classification library. A distance measure between feature vectors is used to find the closest library matches and classify the sample image. Distance measures include classic Euclidean distance, Mahalanobis distance , or robust estimates, residuals, and outlier detection with multi-response data.

Prior to image classification, survey images must be pre-processed through several stepsRead and decompress the image

  1. Enhance image contrast and correct color
  2. Compute red, green, and blue histograms
  3. Compute gray scale image and discrete cosine transform (DCT) coefficients

When repeated over many thousands of images in a survey, the time required for these tasks becomes significant. The project has therefore focused on revising baseline code to perform these tasks faster. Processing tasks have been written in more efficient code (C++) with multi-threaded processing (OpenMP). Performance comparisons below look at the relative performance of the original Matlab code and the revised code running on a single 6-core processor.

Water scatters, refracts, and absorbs light. At greater depths, the reds, oranges, and yellows of sunlight are filtered out, leaving the seafloor illuminated only be a dim blue light. Artificial lighting is essential for underwater photography, but this light still mixes with the bluish light of filtered sunlight, giving images a blue cast. Survey images must be processed to brighten them, improve their contrast, and balance their colors to remove this bluish cast. The images below illustrate this processing with original unprocessed images on the top row, and corrected images below.

Bonaire 2008, Sample 1 Uncorrected
Bonaire 2008 , Sample 1 (uncorrected)
Bonaire 2008, Sample 1 Corrected
Bonaire 2008 , Sample 1 (corrected)
Bonaire 2008, Sample 2 Uncorrected
Bonaire 2008 , Sample 2 (uncorrected)
Bonaire 2008, Sample 2 Corrected
Bonaire 2008 , Sample 2 (corrected)
loading images data

Survey images are stored in large, minimally compressed full-color JPEG image files on disk-based storage. To process the images, they must be loaded from disk and decompressed into pixel arrays. The revised implementation uses the widely-used open source libjpeg. Unfortunately, this remains single threaded and I/O bound. Run times have been reduced 67% from 0.43 sec/image to 0.14 sec/image.

Histogram Image Data

Color is a useful factor in differentiating one species from another. Our species identification method characterizes organism colors by computing a color histogram that lists the colors present and how often they occur. Since a single image could depict a large number of different organisms, each image is first divided into several thousand small overlapping tiles so that each one usually spans only a single organism. The colors in each tile are then tabulated. The revised implementation uses custom tiling and histogram generation for a coarse histogram of red, green, and blue image channels. The new code is multi-threaded compared to the original single-threaded code. Run times have been reduced 99% from 26 sec/image to 0.03 sec/image.

enhance images data

The revised implementation uses custom dynamic range scaling for the red, green, and blue image channels to enhance and correct images. The new code is multi-threaded compared to the original single-threaded code. Run times have been reduced 93% from 0.19 sec/image to 0.01 sec/image.

DCT Processing Image Data

Surface texture is a strong factor when differentiating one species from another. To characterize texture, our method uses a Discrete Cosine Transform (DCT) that breaks an image tile down into a series of overlapping cosine waves that, when summed together, recreate the bright and dark features of an organism’s texture. The revised implementation is based on the open source FFTW image processing library. The new code is multi-threaded compared to the original single-threaded code. Calculations are done using single-precision floating-point math. Run times have been reduced 99% from 38 sec/image to 0.09 sec/image.

In the original Matlab code, histogram generation and DCT coefficient calculations were by far the dominant tasks. These required 26 and 38 sec/image, respectively, compared to just 0.4 and 1.9 sec/image times to load and enhance images.

In the revised implementation, the image load time dominates at 0.14 sec/image compared to 0.01, 0.03, and 0.09 sec/image for enhancement, histograms, and DCT coefficient calculations. The image load remains single threaded and is limited by the cost of I/O to read the source image from disk-based storage.

Overall, the revised implementation’s run times have been reduced 99% from 64.8 sec/image to just 0.27 sec/image. This is about 200x faster.

THOR Parts

Hover or tap to view each part of THOR.

Main Body Frame

The main body frame is located in the center of THOR. The main body controls the wing motors, distribute power, and ensures communication.

Wing

THOR’s wings are located on the left and right side of its body. The wings move up and down, controlling the flow of the water as THOR dives.

Central Brain

THOR’s brain is located in the center, near its body. Parts includes in the central brain are: acoustics, cameras, control systems, and lasers.

Tail Fin

The tail fin provides stability for THOR.

Nose Cone

The nose cone is located at the front of THOR. There is a forward sonar to avoid obstacles and a telemetry modem for communication with IMT Lab scientists.

Seabird Precision

This part is located near THOR’s body to control the temperature and pressure.

Precision Acoustic Altimeter

This part is located and connected to the center of THOR, in its main body.

Light Panel

The light panels are located in the front and back, underneath THOR’s body, providing florescence and broadband.

We would like to give thanks to: