Marine Imaging Workshop Abstracts

Expanding AI annotation assistance in BIIGLE

Daniel Langenkämper , Biodata Mining Group, Faculty of Technology, Bielefeld University, Germany

Martin Zurowietz (Genome informatics Group, Bielefeld University, Germany), Tim W Nattkemper (Biodata Mining Group, Faculty of Technology, Bielefeld University, Germany)

As the quality, resolution, and volume in underwater visual data (i.e. photos and videos) keeps increasing with the number of applied platforms as well as with progress in digital camera and storage technology, the bottleneck in the analysis and interpretation of marine visual data becomes more serious. For example, the recent trend to develop so called ocean digital twins, to monitor and model marine habitats and anthropogenic impacts, requires quantitative and/or qualitative data extracted from such visual data, so this rich source of information can be integrated in this development. In this contribution we will present new features of the open source online visual data annotation tool BIIGLE that address this bottleneck problem by implementing different kinds of AI (artificial intelligence) support. First, the nowadays most obvious approach is to apply machine learning for automatic detection and classification of objects (such as fishes, corals, sponges etc). While BIIGLE’s deep learning-based object detection method MAIA has been implemented a few years ago and is still improving, we will present other new AI-related functions of BIIGLE. One example is the Segment Anything Model-based “Magic SAM” instrument in the image annotation tool that automates the segmentation of objects during manual annotation. Another example is an improved version of the LARGO (LAbel Review Grid Overview) tool, which is used by many users for the posterior assessment and processing of manual or computational detection and classification results. To make this process much more efficient, we implemented new sorting functions and an outlier detection, based on deep learning–derived morphological similarities. A new import interface supports the import and export of annotation data in the COCO (Common Objects in COntext) format, so the interplay between manual annotation and review in BIIGLE and external machine learning applications is facilitated.

FathomNet Database: An open-source image database and machine learning model repository for underwater artificial intelligence

Kevin Barnard , MBARI

Brian Schlining, MBARI (USA), brian@mbari.org Lonny Lundsten, MBARI (USA), lonny@mbari.org Giovanna Sainz, MBARI (USA), gsainz@mbari.org Eric Orenstein, National Oceanography Centre (UK), Eric.Orenstein@noc.ac.uk Erin Butler, CVision AI (USA), erin.butler@cvisionai.com Benjamin Woodward, CVision AI (USA), benjamin.woodward@cvisionai.com Kakani Katija, MBARI (USA), kakani@mbari.org

Since its launch in 2021, FathomNet Database has served as a public, open-source database for the aggregation of expertly-labeled ocean imagery. Over the past two years, FathomNet Database has grown significantly in both size and functionality. Today, FathomNet is home to over 100,000 images and nearly 300,000 localizations, made possible by its community of over 600 individuals, with contributing institutions including MBARI, NOAA Ocean Exploration, National Geographic Society (NGS), Ocean Networks Canada (ONC), University of Hawaii, and University of Plymouth. The FathomNet website now offers new community features to support taxonomic ID discussions, annotation history tracking, ORCID integration, and mechanisms to follow contributions related to topics (animals, community members, marine regions) of interest. Additionally, the FathomNet Model Zoo provides a platform to host, organize, and serve machine learning (ML) models trained using FathomNet data. FathomNet’s open-source ecosystem of tools and services are designed to easily integrate into existing workflows at any point of the data lifecycle. We will present an overview of FathomNet Database, and its related services, with a focus on improvements released in Version 1 (July 2023). We will also discuss several general findings regarding ML usability and interoperability for ocean science with visual data based on our user-centered design process. FathomNet is an evolving public resource, and we hope to encourage further engagement from oceanographic community members as we build toward a dataset representing all ocean life.

Aquascope: automated imaging and machine learning to understand and predict plankton community dynamics in lakes

Francesco Pomati , Eawag: Swiss Federal Institute of Water Science and Technology, Aquatic Ecology

Ewa Merz (Eawag & UCSD), Pinelopi Ntetsika (Eawag), Marta Reyes (Eawag), Stefanie Merkli (Eawag), Luis Gilarranz (Eawag), Stuart Dennis (Eawag), Marco Baity-Jesi (Eawag)

Plankton communities are highly diverse and dynamic and influence important ecosystem properties at the local and global scale. They are sensitive indicators of ecosystem health and global change's effects on ecosystem services. Algal blooms, for example, are an emergent property of a complex interaction network of producers and consumers, often triggered by climate change and eutrophication. Traditional plankton monitoring based on sampling and microscopy fails to capture high-frequency dynamics and complexity in plankton communities. Here we present automated in-situ tracking and modelling of plankton interaction networks in lakes based on automated underwater microscopy, machine learning and time-series analysis. The imaging approach is embedded into a floating platform, comprising sensors for profiling water column physics and chemistry, and meteorological information. Sensor data are integrated with lake hydrodynamic models, and the whole platform is suitable for stable long-term deployments for research and water quality monitoring. A dual magnification dark-field microscope and associated image processing and classification provide real-time, high-frequency plankton data from ~10 μm to ~ 1 cm, covering virtually all the components of the planktonic food web and their morphological diversity. We find that vision transformers, with targeted augmentation, constitute the most robust models for high-accuracy plankton classification and low sensitivity to temporal dataset shifts. Comparing imaging data from the field to traditional sampling and microscopy revealed a general overall agreement in plankton diversity and abundances, despite some limitations in detection (e.g. small phytoplankton) and classification (e.g. rare and morphologically similar taxa). Using high-frequency time-series of plankton abundances, or their daily growth rates, environmental conditions and state-space reconstruction methods for modelling chaotic dynamics, our data allow us to: map interactions between taxa, reverse engineer mechanisms that drive algal blooms or biodiversity change, and forecast ecosystem properties.

FathomVerse: Gaming for ocean exploration

Kakani Katija , Bioinspiration Lab, Research and Development, MBARI

Lilli Carlsen (MBARI), Emily Clark (MBARI), Giovanna Sainz (MBARI), Joost Daniels (MBARI), Kevin Barnard (MBARI), Ellemieke Berings (&ranj Serious Games), Meggy Pepelanova (&ranj Serious Games), GAF van Baalen (&ranj Serious Games)

In order to fully explore our ocean and effectively steward the life that lives there, we need to increase our capacity for biological observations; massive disparities in effort between visual data collection and annotation make it prohibitively challenging to process this information. State-of-the-art approaches in automation and machine learning cannot solve this problem alone, and we must aggressively build an integrated community of educators, taxonomists, scientists, and enthusiasts to enable effective collaboration between humans and AI. FathomVerse, a mobile game designed to inspire a new wave of ocean explorers, teaches casual gamers about ocean life while improving machine learning models and expanding annotated datasets (FathomNet). Of the three billion gamers worldwide, up to 70% say they care about the environment, and FathomVerse taps into this engaged community with innovative gameplay and rich graphics that draw players into the captivating world of underwater imagery and cutting-edge ocean science. Here we will share our process of designing FathomVerse, discuss early successes from the v1 launch on May 1, 2024, and highlight some areas of future focus. In less than one month after v1 launch, 8k players from 100 different countries generated >3M annotations that are currently being used to generate consensus labels and retrain machine learning models. Through FathomVerse, we hope to activate global audiences in high school and up, with the goal of increasing public awareness and inspiring empathy for ocean life.

Harnessing 3D Geometrical Features for Automated Coral Habitat Classification

Larissa M. C de Oliveira , 1. Earth & Ocean Lab, Department of Geography, University College Cork, Ireland 2. Environmental Research Institute (ERI), University College Cork, Ireland

Patricia Schontag 3, Judith Fischer 3, Timm Schoening 3 3. Data Science Unit (DSU), GEOMAR Helmholtz Centre for Ocean Research Kiel, Germany

Cold-water coral reefs are among the most structurally complex marine ecosystems, supporting vast biodiversity. Recent advancements in photogrammetry, such as Structure-from-Motion (SfM) and gaussian splatting, have facilitated the creation of high-resolution 3D models of these environments. However, integrating 3D structural complexity into artificial intelligence (AI)-based classification methods remains an emerging area of research. While established 3D classification methodologies have proven effective in terrestrial applications, their potential for underwater contexts has yet to be fully explored. Measures of 3D complexity provide ecological indicators for coral reef assessments. But their use beyond the ecological landscape remains uncertain. This study investigates the application of geometrical features to enhance the automated classification of 3D reconstructions of coral reefs. By computing geometrical features at multiple scales, seabed details are captured at varying spatial resolutions, allowing for a data-driven optimisation of 3D scene classification parameters. We developed four feature sets incorporating geometric features derived from point cloud covariance matrices and primary features such as 3D coordinates and colour. These feature sets were passed through a gradient-boosted tree model to assess performance and feature importances. Our results indicate that primary features alone are insufficient for accurate classification, whereas incorporating geometrical features significantly enhances performance. Furthermore, the combination of geometrical features with colour information provided only marginal improvements. These findings underscore the robustness of geometrical features for precise 3D classification and habitat mapping in complex 3D environments like coral reefs.

High-resolution time series of plankton and carbon from underwater imaging

Klas Ove Möller , Institute of Carbon Cycles, Helmholtz-Zentrum Hereon

Ankita Vaswani, Katharina Kordubel, Daniel Blandfort, Götz Flöser (all Institute of Carbon Cycles, Helmholtz-Zentrum Hereon) and Saskia Rühl (Plymouth Marine Laboratory)

High-resolution snapshots of plankton biodiversity, behavior and carbon flux from an underwater imaging time series: Plankton, organic particles and aggregates ("marine snow") form the basis of marine ecosystems and play a fundamental role in the food webs and the biological carbon cycle of the ocean. For these reasons, it is necessary to understand the processes that influence the spatial and temporal distribution as well as the origin, characteristics and biodiversity of these organisms and particles. However, these processes are still insufficiently resolved and quantified, since marine ecosystems are characterized by immense variability and traditional measurement methods are limited in their resolution. To overcome this limitation we deployed an underwater high-resolution imaging observatory in the North Sea to investigate the dynamics and characterization of plankton and particles and to quantify the balance between biological productivity and carbon export at an unprecedented resolution (from seconds to months). To unlock the full potential of our observations, we developed new AI based image analysis pipelines that go beyond classifying organisms by taxonomic groups, but also quantify functional traits and biological phenomena from images. We here present highlights and snaphots of our unique high-resolution time series, including a one year time series of carbon flux in a coastal environment and the underlying drivers, the biodiversity, seasonal variation and vertical migration patterns of plankton organisms, and the increase and impact of Noctiluca on the food web dynamics. These high-resolution observations of the interactions between physics, biology and geochemistry at unprecedented spatial and temporal scales contribute significantly to our understanding of the productivity, resilience and carbon storage capacity of marine ecosystems as a function of climatic changes (rise in temperature, acidification, oxygen minimum zones) and anthropogenic influences and their future developments (nutrients, economic use, loss of biodiversity).

Imaging and taxonomic identification of meiofauna: progress on the BLUEREVOLUTION project

Catherine Borremans , IFREMER (Biology and Ecology of dEEP marine ecosystems Unit (BEEP)/Deep Sea Lab)

Daniela Zeppilli (IFREMER), Abdesslam Benzinou (ENIB), Kamal Nasreddine (ENIB), Valentin Foulon (ENIB), Anthonin Martinel (ENIB), Edwin Dache (IFREMER)

The Earth’s Ocean is the largest three-dimensional living space on our Planet. It is crucial to life as we know it, yet we know less about the entire seafloor than we do of the surface of the moon. Infaunal benthic communities (meiofauna) comprise some of the most diverse groups of organisms on Earth, and only a very small amount of this diversity has been described by science. Despite our partial exploration of this vast domain, all marine habitats, including the deepest trenches, experience direct or indirect human impacts. Thus, the scientific community faces the need for fast and accurate monitoring studies and building baseline datasets to measure future changes and biodiversity losses. In this context the “Biodiversity underestimation in our bLUe planEt: artificial intelligence (AI) REVOLUTION in benthic taxonomy” (BLUEREVOLUTION) project aims at developing in- and ex-situ methods using automated or semi-automated imaging method on meiofauna linked with taxonomic classification tools (AI) for high throughput analysis of benthic diversity. Different imaging techniques -from low to high resolution- were assessed (e.g. flow imaging, focus stacking brightfield microscopy, 3D-fluorescence imaging) and developed (‘MeioCam’) to identify the best configuration(s) according to resolution (capacity to taxonomic identification), throughput, quantitative power, data volume (Foulon et al., 2024). In addition, computer vision-based methods were investigated to support the need of faster recognition and morphometric measurement of meiofauna (e.g. nematods), as well as to compensate for the lack of available imaging dataset to train artificial intelligence models. Approaches based on deep learning and GAN are proposed in this work. The presentation will give an updated overview of the developed imaging pipeline that will ultimately allow for quantitative and functional data of benthic communities being generated at speeds unseen before. It will produce a standardized method for building open-access reference databases together with fast and reliable tools for impact assessments and biodiversity surveys. Ref : Foulon, V., et al. 2024. Taxonomic identification of meiofauna through imaging – a game of compromise. Submitted in Limnology and Oceanography: Methods

Interactive machine learning: reflections on how a human-in-the-loop approach can increase model training efficiency.

Dr Laurence H. De Clippele , School of Biodiversity, One Health & Veterinary Medicine, University of Glasgow

De Clippele L.H., Dominguez Carrió C., Vad J., Vlaar T.,  Balsa J., Beckmann L.M., Boolukos C., Burgues I., Clark H.P., Cook E., de la Torriente Diez A., Easton. B., Ferreira T. C., Guzzi A., Harris L., Kaufmann M., Mendoza J.C., Riley T.G., Villalobos V.I.

Due to technical advances in imaging technologies and cost-effective data storage solutions, there is a global increase in the amount of underwater image data. Since it is too time-consuming to analyse this data manually, marine scientists are increasingly interested in using convolutional neural networks to automate their annotations. However, deep learning is considered to be a “black box” because it is difficult for the user to understand how deep neural networks make their decisions. In addition, if the user lacks an understanding of the machine learning fundamentals, it is unlikely that they will be able to develop models that are effective and produced in a time-efficient manner. Interactive, “human-in-the-loop”, machine learning is here suggested as a novel approach to increase the user’s understanding of fundamental machine learning concepts, increase model training efficiency and accuracy drastically, and make deep learning less of a black box. In June 2024, a Marine Animal Forest COST action training school was organised at the University of Glasgow. This training school brought together early career researchers from across the UK and European institutions. Through training models on a variety of Remotely Operated Vehicle and timelapse image datasets, reflections were gathered on how interactive machine learning can contribute to increasing the efficiency of model training. The main conclusions were regarding the importance of user expertise of the target object (i.e. the species), the complexity of the habitat, the quality of the data, the abundance of the species, and the psychological effect of observing a change in model performance. Other interesting observations were made regarding a lack of model improvement when trying to “recover” from mistakes during the training process and dealing with the uncertainty when creating a training dataset. A practical workflow is proposed to encourage more efficient model training decisions by users new to machine learning.

Marine snow particle imaging resolves the mechanism and magnitude of coastal carbon export

Colleen Durkin , MBARI

Danelle Cline (MBARI), Duane Edgington (MBARI), Sachithma Edirisinghe (U. Maine), Margaret Estapa (U. Maine), Nils Haentjens (U. Maine), Christine Huffard (MBARI), Paul Roberts (MBARI), Henry Ruhl (MBARI), Fernanda Lecaros Saavedra (MBARI), Melissa Omand (URI), Sebastian Sudek (MBARI)

Ocean waters are filled with marine snow: particles made of fecal pellets, aggregates, and other detritus generated by marine life. These particles are the primary food source for many deep-sea ecosystems, and are a key mechanism for ocean carbon uptake and sequestration. Constraining the spatial, temporal, and compositional variability of sinking marine snow is a fundamental challenge for both ocean biogeochemistry and deep-sea biology. We image individually resolved particles collected in samplers and also by in situ instruments, to estimate the quantity of carbon export in the ocean and its ecological drivers. We apply methods in computer vision and machine learning to automate the classification of large imaging datasets into ecologically meaningful categories of marine snow. These particle classes can then be used to estimate carbon biomass and export. Coordinated observations made in Monterey Bay, California demonstrate how a suite of imaging systems can expand the scale of particle flux measurements while also resolving mechanisms driving its variability. We deployed traditional sediment traps to collect physical samples of sinking particles, and used chemical measurements of carbon flux to ground-truth estimates generated by particle imaging. These ground truth measurements were related to in situ imagers of particles, including camera systems attached to long-range autonomous underwater vehicles, remotely operated vehicles, neutrally buoyant floats, and moored to the seafloor. Together, these temporally sustained and/or spatially distributed observations will enable us to quantify carbon export in dynamic coastal environments where the ecological source and fate of sinking carbon is difficult to trace.

Object Detection for quantitative analysis of deep-sea benthic megafauna distribution: making the best of video transects

Nils Piechaud , Institute of Marine Research, Bergen, Norway

Heidi Meyer¹, Pål Buhl-Mortensen¹, Genoveva Gonzalez-Mirelis¹, Gjertrud Jensen¹, Yngve Klungseth Johansen¹, Anne Kari Sveistrup¹, Rebecca Ross¹ 1: Institute of Marine Research, Bergen, Norway

The MAREANO program maps habitats on the Norwegian seabed by analysing data from a large dataset of underwater imagery collected over many years. Counting individual objects and animals in videos is a common technique to sample the seabed and its automation (partial or complete) with Computer Vision (CV) and Machine Learning (ML) would greatly help quantitative benthic ecology. Yet, many practical problems must be solved before it can reliably generate data on species distribution and abundance. An important drawback of automated annotations is the difficulty for Object Detectors (OD) models to account for contextual information in the video where the behaviour of the object over multiple frames can inform its identification. We investigated how recent innovations and open source libraries could help address this challenge and enable an OD to quickly generate the data required for mapping. Seabed images were analysed manually in Biigle to get bounding boxes annotations which were used as a training set for various OD models (V8m, V8X, Fathomnet and Fathomnet’s Megalodon). Predictions from these models were used to count the number of individuals of selected taxa in videos which were compared to the manual counts. We used additional techniques (tracking, deflector classes, zonal counting) to refine and filter the predictions and maximize performances. F1 scores in training can reach 0.95 for target taxa. Automated Individual counts per videos can also be within 95% of the manual counts. Performances can, however, drop dramatically in challenging visibility conditions prompting the need to manually review and filter the predictions to ensure consistency and reliability. These results show that, in the right conditions, CV can perform well and be useful to enumerate target taxa. However, the need to manually review the results remains a bottleneck and we discuss the possible ways around it.

Online Learning of Appearance Models to Enable Real-Time Visual Tracking of Arbitrary Marine Animals

Levi Cai , MIT and WHOI

Nathan McGuire (WHOI), Roger Hanlon (MBL), T. Aran Mooney (WHOI), Yogesh Girdhar (WHOI)

Collection of video and co-located telemetry information about various marine animals is crucial for understanding their behaviors and even establishing conservation practices. Divers or tags are the two most commonly used approaches to gathering this type of data, but is mostly limited to surface dwelling species and difficult to scale. Autonomous underwater vehicles (AUVs) equipped with cameras and edge computers directly integrated into their control systems are showing increased promise for enabling collection of this type of data without the use of tags or divers. However, there is a dearth of well-labeled visual data accessible in the marine domain for many species. In this work, we propose the use of a class of visual algorithms known as semi-supervised learning and tracking approaches to estimate online appearance models of never-before-seen visual objects. These approaches require a scientist to initially provide a bounding box or visual template of an animal, and then tracking can be done autonomously. We demonstrate how to incorporate these models into AUV platforms to enable tracking of marine animals, in-situ and in real-time, for which labeled data may not exist. Additionally, this allows us to simultaneously gather video and co-located telemetry information about them. Furthermore, we discuss strategies for deploying these types of systems on a larger-scale and argue the importance of continued innovation and development of AUV platforms that are equipped with modern edge devices and stereo-vision that are deeply integrated into their control systems.

Rate-A-Skate – Deep learning for individual recognition of Flapper Skates

John Halpin , The Scottish Association for Marine Science

Joe Marlow1, Steven Benjamins1, Jane Dodd2, Thomas Wilding1 1 - Scottish Association for Marine Science, Oban, Argyll, PA37 1QA 2 - NatureScot

Flapper skates (Dipturus intermedius), once common in the Northeast Atlantic, are currently classified as Critically Endangered by the International Union for the Conservation of Nature (IUCN). To assess the efficacy of management measures, such as dedicated marine protected areas on skate populations, novel monitoring approaches are needed. Flapper skates can be accurately identified by their distinctive dorsal spot patterns. This has led to the development of the SkateSpotter database, managed by SAMS and NatureScot. The database collates images submitted by anglers for identification/tracking of individual skates, enabling study of skate migration and home-ranges, important considerations in their management. SkateSpotter presently relies on manual identification of skates which, while accurate, is time consuming. Here we present a deep learning method for skate re-identification, using a custom 2-stage deep learning algorithm. The algorithm can re-identify skate with a high degree of certainty (for a test set, unseen during training, 80% of images had correct individual as 1st match) and deal with the variation in the angler submitted images, for example, glare and occlusion. The algorithm is being incorporated into SkateSpotter, enabling a step-change in the speed at which new skate images can be checked for a match. This will enable the cost-effective extension of the database (in time and geography) enhancing scientific value. The combination of Skate Spotter and Rate-A-Skate represent best practice for future animal re-identification challenges, with the success of the deep learning algorithm being proportional to the quality of the images in the database.

Sweating the small stuff: Annotation methods to improve small and low cover organism estimates

Emma J. Curtis , School of Ocean and Earth Science, National Oceanography Centre, University of Southampton Waterfront Campus, Southampton, SO14 3ZH, UK; Centre for In Situ and Remote Intelligent Sensing, University of Southampton, Southampton, SO16 7QF, UK

Jennifer M. Durden, Ocean BioGeosciences, National Oceanography Centre, European Way, Southampton, SO14 3ZH, UK; Blair Thornton, Centre for In Situ and Remote Intelligent Sensing, University of Southampton, Southampton, SO16 7QF, UK, Institute of Industrial Science, The University of Tokyo, 4-6-1 Komaba Meguro-ku, Tokyo 153-8505, Japan; Brian J. Bett, Ocean BioGeosciences, National Oceanography Centre, European Way, Southampton, SO14 3ZH, UK

Small and low cover organisms can make up a large proportion of community structure and provide important insight into the resilience of ecosystems. Manually extracting measurements used to monitor these organisms with image annotation can be a high effort process and subject to human error and bias. Automating image annotation of marine images can reduce manual annotation effort and some human error, but supervised techniques and evaluating predictions rely on robust manual annotations. We present considerations for annotating small and low cover organisms from seafloor images, using images annotated for sparsely distributed cold-water coral by 11 researchers as an example case. Using at least two annotators with some overlap in annotation allows for an evaluation of annotation success and uncertainty in derived measurements. Segmenting distinct coral colonies reduced the standard deviation of cover estimates threefold compared to a grid-based cover estimation method, with no significant difference in annotation time. Size bias in manual annotation can lead to missing or incomplete annotations of small organisms and can be propagated by supervised machine learning techniques when incomplete annotations are used as training data, as demonstrated in our example case. By modelling missing sizes of colonies, we reduced the error in our coral cover estimates by up to 23%. When these modelled sizes were used to generate segments for incomplete annotations, trained machine learning predictions significantly improved. We discuss some additional methods which are either more computationally expensive or require more human effort to reduce size bias in image annotations, giving a summary of the impacts and limitations of the outlined methods to provide researchers with guidance on how to more efficiently and reliably monitor small and low cover organisms using seafloor images.

Validating an AI-based holographic imaging platform (Aqusens) for monitoring harmful algal bloom species

Holly Bowers , Moss Landing Marine Laboratories

Maxim Batalin, Lucendi, Inc. Alexis Pasulka, California Polytechnic State University

Harmful algal bloom (HAB) species can have detrimental effects on ecosystems, public health, fisheries, drinking water, subsistence harvesting, aquatic recreation and tourism. The HAB field is working to expand detection capabilities within a nationally networked system for long-term monitoring and early warning (National HAB Observing Network; NHABON). To meet this need, instruments demonstrating portability, robustness, ease-of-use, and the ability to produce real-time data are highly desired. Our project compares and validates a new cost-effective automated cell imaging platform (Aqusens) with the Imaging Flow CytoBot (IFCB). The Aqusens is based on holographic imaging principles coupled with machine learning for optimizing computation and object characterization. The Aqusens does not contain expensive and complex to maintain optical or mechanical components. Most of the hardware is off-the-shelf and mass-produced, allowing for scalable production and cost reduction. Accompanying software is used to control Aqusens operations and analyze collected data. It is user friendly and does not require substantial training or maintenance. Validation of the Aqusens is being carried out through three objectives: 1) laboratory experiments with a variety of cultured HAB and non-HAB species to provide foundational platform comparisons; 2) deployments at the Santa Cruz Wharf monitoring site to test performance during varied conditions (e.g. algal blooms, upwelled sediment); and 3) in San Francisco Bay cruise transects (slated for 2025) targeting seasonal succession of phytoplankton populations. These cruises, along with the Santa Cruz Wharf monitoring station, provide avenues for new platform integration into well-established programs that offer publicly available long-term data sets. Furthermore, platform portability is allowing us to support a broad variety of stakeholders (e.g. aquaculture, educational programs). These efforts are assessing functionality of the Aqusens in various settings and evaluating integration of data into existing public portals (California Ocean Observing Systems Data Portal [CalOOS] / Harmful Algal Bloom Data Assembly Center [HABDAC]).

AI-Derived Krill Detection and Quantification from ROV Video

Savana Stallsmith , Oregon State University

Chris Sullivan Oregon State University: Astrid Leitner Oregon State University and MBARI

Artificial Intelligence is becoming a popular solution for solving a variety of complex problems in a way that allows us to work smarter and not harder. One important and evolving application of AI is in marine imaging, specifically helping to widen the bottleneck that the time consuming annotation of marine video creates for oceanographic research. While rapid advances are being made in AI for benthic marine images, the pelagic system is much more challenging because animals are highly mobile and low contrast with flexible, delicate morphologies. Krill are a key component of the California upwelling system food web and are highly mobile and often highly abundant from the surface to 200m depths. Quantifying krill from video is extremely difficult and time consuming. To solve this issue, we trained an A.I. model to automatically detect and quantify krill in pelagic ROV video transects from the Monterey Bay. We compare the AI generated quantifications with pre-existing hand quantified videos. We show that it is possible to train an A.I. model to detect krill and to export those detections to generate counts, though comparison to manual counts remains a challenge and requires tracking across frames. Several solutions to tracking individuals were tested, and with continuing development this approach could make video a viable method for quantifying these important prey organisms as well as other challenging midwater species.

AI-based Identification of Ocean Organisms in Imagery and Enhanced Human-Machine Collaboration for improved Video Analysis

Patrick Cooper , University Corporation for Atmospheric Research / NOAA Ocean Exploration Affiliate

Mashkoor Malik_2, Ashley N. Marranzino_1, Philip Hoffman_2 (1-University Corporation for Atmospheric Research / NOAA Ocean Exploration Affiliate, 2- NOAA Ocean Exploration )

Marine imagery analysis is essential for understanding and exploring underwater environments. However, manually annotating and characterizing captured footage and images is a time-consuming task for human experts. NOAA Ocean Exploration currently leverages SeaTube - A tool built by Ocean Networks Canada for human-based video annotations. To address these limitations, NOAA Ocean Exploration is scoping and developing several tools including “Seabot” and “Critter Detector” to increase annotation efficiency and evaluate the potential for automated annotations. The unique data distribution of marine imagery and sparse sampling compared to terrestrial domains requires human-in-the-loop systems and our efforts are focused on improving the efficiency of human-based annotations. Seabot uses Vision Transformers (ViTs) to categorize and localize organisms within individual images, relying on the Ocean Vision AI / FathomNet training dataset. Critter Detector operates on video footage and identifies distinct video segments of interest, entities, organisms, and debris within arbitrary video clips. Critter Detector utilizes deep learning, specifically convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, to detect significant events and visually interesting content within underwater video footage leveraging SeaTube footage and annotations to train the Critter Detector. While Seabot emphasizes organism identification on individual images, Critter Detector is designed to utilize the recurrence relation of an RNN architecture to allow for identification of significant temporal regions within video. Both of these proposed models, trained on a diverse dataset of annotated images and videos, achieve useful accuracy, reducing the time and effort required for manual analysis. As the technology advances, incorporating features such as species recognition and behavior analysis, realtime deployment, it may provide new opportunities for marine research, targeted sampling operations and conservation efforts.

AQUA-VIS: Autonomous QUantitative Analysis and Visual Identification System

Judith Fischer , GEOMAR

Autonomous Underwater Vehicle (AUV) missions have revolutionized the exploration of the underwater world and enabled high-resolution seafloor imaging. This imagery is instrumental for monitoring benthic ecosystems, assessing biodiversity, or detecting objects such as unexploded ordnance. However, post-processing the acquired data from AUV surveys is often time-consuming, hindering immediate decision-making for subsequent mapping locations and effective environmental analysis. This work addresses this issue by developing an advanced image processing workflow tailored for AUVs that leverages open-access image processing tools. The proposed workflow aims to process incoming images in near real-time, extract features for 3D reconstruction, and detect objects from seafloor images. The workflow consists of two main steps: i) feature extraction and ii) object detection. Feature extraction is carried out using the open-source software COLMAP, which is a "general-purpose Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline". In the second part of the workflow, pre-trained tracking models such as YOLO are utilized to create an object detection framework that can detect objects from incoming images. Our current focus is on training YOLO using seafloor image datasets. We aim to train models for different locations, such as the Atlantic, Pacific, and Baltic Sea, thereby creating trained models that the wider community can use. These workflows are integrated into the AUV system to ensure they can operate in near real-time conditions. Processing time is optimized to handle the large number of incoming images while maintaining high accuracy. We anticipate that the results of this framework will support the accessible use of open-source workflows to the research community. Scientists can use this tool to gain valuable insights into underwater ecosystems and make informed and fast decisions about further dive sites based on immediate reports generated by the AUV at the end of each dive.

Addressing challenges in using object detection for small datasets of imagery with a data-centric approach

Caroline Chin , National Institute of Water and Atmospheric Research (NIWA), Wellington, New Zealand; Te Herenga Waka - Victoria University of Wellington, Wellington, New Zealand

Ashley Rowden (National Institute of Water and Atmospheric Research (NIWA), Wellington, New Zealand; Te Herenga Waka - Victoria University of Wellington, Wellington, New Zealand), Alan Tan (National Institute of Water and Atmospheric Research (NIWA), Auckland, New Zealand); Daniel Leduc (National Institute of Water and Atmospheric Research (NIWA), Wellington, New Zealand

In marine imagery research, real-world applications using object detection models face challenges such as class imbalance due to heterogeneity within benthic communities, class/labelling ambiguity, and image quality (e.g., illumination). With these challenges, it is difficult to determine the optimal quantity and quality of research-derived training data – that are typically small datasets – thus impacting model performance. Furthermore, traditional object detection performance metrics such as mean Average Precision (mAP), F1-score, precision, and recall may not comprehensively evaluate a model’s efficiency nor inform the researcher on where errors or gaps may exist within the training data that can be targeted for improvement. In this study, our aim was to understand benthic community patterns within the Kermadec Trench by utilising object detection models for video imagery. We created training datasets by annotating benthic megafauna from random subsets of submersible dive imagery from abyssal and hadal depths using the BIIGLE 2.0 annotation tool. Using these datasets, we evaluated the performance of YOLOv8 object detection models trained on both single- and multi-class datasets. Our methods included analysis of performance metrics and data visualisations provided within the YOLOv8 framework. Our results demonstrated the importance of analysis within the training dataset to identify issues such as bounding box size, orientation, and localisation within an image that can be resolved with methods such as data augmentation. Furthermore, we observed that random sampling of annotated imagery resulted in a skewed dataset biased towards abyssal megafaunal classes, subsequently increasing class imbalance. To mitigate this issue, we undertook stratified random sampling of training imagery by depth that balanced and improved dataset quality, particularly as faunal classes within abyssal and hadal zones exhibited heterogeneity. Overall, our study demonstrates that by using a data-centric approach towards deep-learning, researchers can relatively easily infer how small-scale training data can be improved for object detection performance.

Applications of the YOLOv8 Computer Vision Model for Long-term Underwater Ecological Monitoring.

Talen Rimmer , Department of Biology, University of Victoria

Bates, C.R. - University of British Columbia McIntosh, D., Zhang, T., A., Branzan, Albu A., & Juanes, F. - University of Victoria

The world's oceans are undergoing rapid changes due to climate change and other anthropogenic impacts, affecting marine species' distribution, abundance, and behavior. Traditional ecological monitoring methods struggle to keep pace with these transformations, especially in underwater habitats. Recently, computer vision techniques have emerged to enhance the efficiency of video and image-based underwater monitoring. These methods facilitate the detection and classification of objects in visual data, potentially streamlining the process of counting and classifying marine organisms. However, the adoption of computer vision in marine ecology has been slow, partly due to its inaccessibility to ecologists and the lack of easily adaptable tools for ecological monitoring. This study investigates the application and validation of computer vision techniques for monitoring underwater pelagic macrofaunal diversity, using a case study from coastal British Columbia. Over 9000 hours of underwater video were collected from four sites over 18 months, using mounted cameras programmed to record five minutes of video every hour. A supervised computer vision model (YOLOv8) was trained with approximately 240,000 annotations to assess the presence and abundance of marine species at these sites. This approach provided high-resolution temporal data on the diversity and abundance of pelagic fish and gelatinous zooplankton at these sites. We detail the process of employing computer vision techniques for long-term underwater ecological monitoring, emphasizing accessibility for ecologists. A stepwise method for adapting computer vision techniques to achieve biodiversity monitoring objectives is presented, highlighting the strengths and limitations of our approach. Finally, we propose future directions for integrating these technologies into new and existing monitoring programs, and suggest priority areas for future research to advance the use of computer vision in underwater ecological monitoring.

Crowd-powered precision: Enhancing image segmentation with collaborative annotations

Emma J. Curtis , School of Ocean and Earth Science, National Oceanography Centre, University of Southampton Waterfront Campus, Southampton, SO14 3ZH, UK; Centre for In Situ and Remote Intelligent Sensing, University of Southampton, Southampton, SO16 7QF, UK

Jennifer M. Durden, Ocean BioGeosciences, National Oceanography Centre, European Way, Southampton, SO14 3ZH, UK; Blair Thornton, Centre for In Situ and Remote Intelligent Sensing, University of Southampton, Southampton, SO16 7QF, UK, Institute of Industrial Science, The University of Tokyo, 4-6-1 Komaba Meguro-ku, Tokyo 153-8505, Japan; Brian J. Bett, Ocean BioGeosciences, National Oceanography Centre, European Way, Southampton, SO14 3ZH, UK

Instance segmentation of organisms from marine images can be used for biodiversity estimation and can be used as training data for supervised machine learning techniques. However, manual segmentation can be subject to human error found in other visual data extraction techniques; including misidentification, imperfect detection and inconsistent boundary delineation; which can be propagated through trained machine learning predictions. User variability can lead to low segmentation agreement (intersection over union scores; IoU; of 29%) as seen in our case example of 11 researchers segmenting cold-water coral colonies. We outline two methods for combining users’ segments together to produce less variable annotations. The first method establishes a user consensus on the detection and identification of the annotation and then combines the distance scenes of user drawn segments together to generate an average consensus segment, which can be used to estimate organism properties or as training data for machine learning. The second method overlays the user drawn segments over each other to create a heatmap mask with an associated user certainty for the detection, identification and delineation of the organism. These non-binary masks can be used for organism estimates, or, by adapting the loss function of Mask R-CNN as an example, can be used to train machine learning. Both methods reduce the variability of drawn segments in our example, with a 12% and 25% increase in average IoU scores by combining two and three users’ segments, respectively. The value of combining user annotations at the expense of increased annotation effort and the ease of implementing each combination technique are summarised to advise researchers on how to create more reliable segment annotations and higher quality training data for supervised machine learning techniques.

Developing automated multi-modal monitoring strategies of vulnerable marine ecosystems (VMEs)

Chloe Game , University of Bergen, Norway

Associate Professor Pekka Parviainen, University of Bergen, Norway Dr. Pål Buhl-Mortensen, Institute of Marine Research, Norway Associate Professor Ketil Malde, University of Bergen, Norway and Institute of Marine Research, Norway Dr. Pedro A. Rebeiro, University of Bergen, Norway Dr. Rebecca Ross, Institute of Marine Research, Norway;

Anthropogenic impacts on the marine environment are increasing as growing resource demands must be met. Recently (2024), the Norwegian government voted to open an area of the Norwegian Continental Shelf for seabed mining, putting the deep-sea bed, which harbours a rich diversity of ecologically and economically valuable, yet vulnerable, ecosystems under serious threat. It is critical that extensive accurate maps of the seafloor are created to establish baselines and support monitoring of impacts and recovery. Given the region’s vastness and isolation, such monitoring is logistically challenging and too slow to meet the requirements. It must therefore be conducted with photography and machine learning (ML) used to automatically identify and quantify seafloor organisms; avoiding labour-intensive manual analysis. This project aims to design and develop a multimodal DL model for automated identification of VMEs and VME indicator-species in Norwegian waters. This model will help to optimize mapping efforts, by promoting efficiency, consistency and quality of ecological data extraction. Crucially, this will better enable sustainable management of VMEs. Current monitoring efforts do not provide the scale and resolution of data required to protect and conserve these valuable seabed assets. Specifically, we are investigating how effective can off-the-shelf Vision Transformers (ViT) automate classification of VMEs from seabed imagery compared to current approaches (CNNs). We are also exploring whether the addition of environmental data (from other sensors such as local topography and water temperature), in a new multimodal architecture can increase accuracy of automated analysis and refine the level of biological details used to make predictions, which is currently not possible for ML with images alone. Importantly, we will consider explainability of model decisions, opening up the ‘black-box’, and seek to generalize the approach across target seabed communities and datasets.

Enabling Large Scale Coral Reef 3D Models with Accurate Color Representation using DeepSeeColor

Yogesh Girdhar , Woods Hole Oceanographic Institution

Daniel Yang, John Walsh, Stewart Jamieson

Successful applications of complex vision-based behaviours underwater have lagged behind progress in terrestrial and aerial domains. This is largely due to the degraded image quality resulting from the physical phenomena involved in underwater image formation. Spectrally-selective light attenuation drains some colors from underwater images while backscattering adds others, making it challenging to perform vision-based tasks underwater. These distortions make image observation depend on distance, beyond simply resolution and scale change. We present DeepSeeColor algorithm, which is a state-of-the-art underwater image formation model with the computational efficiency enabled by modern deep learning. The proposed approach enables efficient color correction for underwater imagery enabling novel applications such as accurate in-situ imaging using AUVs and and scaling up building of underwater 3d models with accurate color representation. We show the application of the approach to coral reef survey imagery collected by an underwater robot.

Exploring Oceans of Visual Data: Integrating Deep Learning Computer Vision and Information Visualization for Large-Scale Ecological Assessment of Cold-Water Coral Reefs

Daniel Langenkämper , 1. Biodata Mining Group, Faculty of Technology, Bielefeld University, Germany

Ingunn Nilssen Aksel A. Mogstad (Environmental Technology, Technology, Digital & Innovation, Equinor ASA, Norway), Tim W. Nattkemper (Biodata Mining Group, Faculty of Technology, Bielefeld University, Germany)

Growing amounts of marine video and image data are collected using different kinds of camera setups and technologies (e.g., ROV, AUV, stationary platforms) to explore and monitor marine habitats. Concurrently, ambitious ocean digital twin (ODT) projects aim for the integrative collection and management of multi-modal sensor data in large data bases. The aim on ODT development is to make use of this data for analysis, visualization and modeling, together with establishing feedback loops into the marine system by providing documentation for decision makers. However, there are still few clearly defined workflows from raw marine image (or video) data to real data products required for such ODTs. Progresses in artificial intelligence and computer vision (like convolutional neural networks) have increased the potential of computational approaches to (semi-) automatic marine image analysis significantly and the number of papers describing their applications for marine object classification or segmentation is growing year by year. Usually, the works end up with an accuracy assessment and a comparison with competing algorithms, but a gap to a real data product remains. In this contribution, we address the problem of large-scale assessments of cold-water coral reefs based on ROV-collected video data. These assessments are still performed by human operators evaluating hundreds of video files, sometimes with sub-optimal visibility. We present a workflow to estimate the coverage of Desmophyllum pertusum, Paragorgia arborea, other gorgonians and sponges on reefs based on semi-automatic annotation of less than 0.0003% of the available data, training a segformer b3 model and applying it to automatically segment 855 ROV videos from the Haltenbanken area (Norwegian Sea). The results are displayed in a web-based customized visual interface that supports an integrative analysis of the computed coverage data together with data and manual evaluations derived from operator reports.

Making use of unlabeled data: Comparing strategies for marine animal detection in long-tailed datasets using self-supervised and semi-supervised pre-training

Tarun Sharma , Caltech

Danelle E. Cline (MBARI) and Duane Edgington (MBARI)

This paper discusses strategies for object detection in marine images from a practitioner’s perspective working with real-world long-tail distributed datasets with a large amount of additional unlabeled data on hand. The paper discusses the benefits of separating the localization and classification stages, making the case for robustness in localization through the amalgamation of additional datasets inspired by a widely used approach by practitioners in the camera-trap literature. For the classification stage, the paper compares strategies to use additional unlabeled data, comparing supervised, supervised iteratively, self-supervised, and semi-supervised pre-training approaches. Our findings reveal that semi-supervised pre-training, followed by supervised fine-tuning, yields a significantly improved balanced performance across the long-tail distribution, albeit occasionally with a trade-off in overall accuracy. These insights are validated through experiments on two real-world long-tailed underwater datasets collected by MBARI.

Marine Image Analysis with RootPainter

H. Poppy Clark , University of Aberdeen

Abraham George Smith (Department of Computer Science, University of Copenhagen, Universitetsparken 1, 2100 Copenhagen, Denmark) Daniel Mckay Fletcher (Rural Economy, Environment and Society, Scotland’s Rural College, West Mains Road, Edinburgh, EH9 3JG, Scotland, United Kingdom) Ann I. Larsson (Tjärnö Marine Laboratory, Department of Marine Sciences, University of Gothenburg, Strömstad, Sweden) Marcel Jaspars (Marine Biodiscovery Centre, Department of Chemistry, University of Aberdeen, Old Aberdeen, AB24 3DT, Scotland, United Kingdom) Laurence H. De Clippele (School of Biodiversity, One Health & Veterinary Medicine, University of Glasgow, Bearsden Road, G61 1QH, Scotland, United Kingdom)

RootPainter is an open-source and user-friendly interactive machine learning tool that is capable of analysing large marine image datasets quickly and accurately. This study tested the ability of RootPainter to extract the presence and surface area of the cold-water coral reef associate sponge species, Mycale lingua, in two datasets: 18,346 time-lapse images and 1,420 remotely operated vehicle video frames. New corrective annotation metrics integrated with RootPainter allow objective assessment of when to stop model training and reduce the need for manual model validation. Three highly accurate Mycale lingua models were created using RootPainter, with an average dice score of 0.94 ± 0.06. Transfer learning aided production of two of the models, increasing analysis efficiency from 6 to 16 times faster than manual annotation for time-lapse images. Surface area measurements were extracted from both datasets allowing future investigation of sponge behaviours and distributions. The ability of RootPainter to accurately segment the cold-water coral Lophelia pertusa was also tested in 3,681 ROV video frames, with model dice scores surpassing 0.80. Moving forward, interactive machine learning tools and model sharing could dramatically increase image analysis speeds, collaborative research, and our understanding of spatiotemporal patterns in biodiversity.

Object Detection of Benthic Morphospecies using Synthetic Data and Self-Training Domain Adaptation

Heather Doig , University of Sydney

Oscar Pizarro (Norwegian University of Science and Technology, University of Sydney); Stefan Williams (University of Sydney)

Scientists can observe benthic morphospecies using images taken from underwater robotic vehicles. The presence and quantity of morphospecies like endangered handfish or invasive sea urchins can be collected using neural network object detectors on high-resolution images. The detector is trained with labelled images, but its performance can suffer from two issues. First, the detector can degrade when used on images captured from different cameras, water conditions, locations or time compared to the source training images, known as domain shift. Secondly, very few labelled examples of the morphospecies of interest needed to train a supervised model may exist. We propose a framework to train a benthic morphospecies detector using synthetic underwater images followed by unsupervised domain adaptation to bridge the domain gap from synthetic to real. We generated synthetic images with a customised version of Infinigen (infinigen.org) from Princeton Vision & Learning Lab that creates natural scenes with annotation data for training. We have customised Infinigen to mimic images from underwater robots, modelling the attenuation of light and including species of interest, such as invasive sea urchins. We train an object detector using synthetic source images followed by an adaptation step with unlabeled target images. We tested our approach on sea urchins and other species of interest and successfully detected these species in visually complex seafloor images. The framework offers a novel approach for detecting rare and endangered morphospecies using increasingly popular 3D modelling software.

PAMS, a low-cost, simple-to-assemble seafloor imagery solution for monitoring corals

Thierry Baussant , NORCE Norwegian Research Centre

Christian Andreas Hansen, Alan Le Tressoler, Junyong You, Gro Fonnes, Xue-Cheng Tai, Morten Kompen*, Christina Rørvik - NORCE; * NORCE and Compelling AS

NORCE has developed PAMS (Polyp Activity Monitoring System) as an innovative camera-based sensing platform and environmental effect methodology to monitor coral welfare in their natural environment and impacts resulting from use of the ocean space. The system comprises a hardware and software delivery, with the main value attributed to expert and machine learning (ML)-based interpretation of data. Based on off-the-shelf components, PAMS can be regarded as an effective, affordable, simple-to-assemble yet modular and customizable tool for users who want to monitor corals (shallow and deep) and assess their status. Whilst there exist solutions to measure coral endpoints in shallow waters, there is still a technology gap for deepwater corals and in general - it is difficult to find an appropriate parameter for monitoring corals in a non-intrusive manner which reflect any influences from their surrounding environment. High resolution still photos can support the non-intrusive and specific identification of polyp behavioral change on corals exposed to particle plumes or change of conditions in the water. The PAMS technology is based on years of lab research and observations dedicated to specific analysis of coral endpoints and more specifically coral behavioral endpoints with polyp activity. It is currently developed using specific ML modules for automatic coral recognition and classification of the coral status according to their polyp activity using a high-throughput process independent of user expertise. Currently at Technology Readiness Level (TRL) 5-6, the vision for PAMS encompasses both full development and qualification for industrial applications as well as the advancement of research on marine seafloor environments where coral is found. In this workshop, we will present a detailed description of the current hardware and the status of the software, including an open-source data visualization platform for displaying and analyzing coral images. The PAMS project is funded by the Research Council of Norway.

Publicly available and freely accessible Imagery AI Training Data set repository

Errol Ronje, NOAA National Centers for Environmental Information

Megan Cromwell (NOAA National Ocean Service), Vidhya Gondle, Yee Lau, Kirsten Larsen, Hassan Moustahfid, Rachel Medley (NOAA Ocean Exploration), Mashkoor Malik (NOAA Ocean Exploration), Patrick Cooper (University Corporation for Atmospheric Research UCAR / NOAA Ocean Exploration Affiliate), Kakani Katija, Brian Schlining (Monterey Bay Aquarium Research Institute)

Efforts to advance artificial intelligence (AI) applications, particularly in underwater image data analysis, are hindered by the absence of a curated, long-term repository for AI training data. The lack of such resources constrains collaboration and knowledge sharing across various entities, including governmental organizations, research institutions, and industries. In order to address this community-wide need for a robust standardized and openly accessible training data set, NOAA National Centers for Environmental Information (NCEI), NOAA Ocean Exploration, and the Monterey Bay Aquarium Research Institute have agreed to co-design and develop a public and freely available underwater imagery training data repository at NCEI using curated data from MBARI’s FathomNet project. This initiative will allow data and appropriate metadata submitted to FathomNet to be transferred to NCEI for web-accessible hosting and continuous community access. Dedicated pipeline development has begun with the image and metadata extraction automation from FathomNet image banks for hosting on a partner managed web-accessible server. The FathomNet data and metadata will be combined into a version-controlled bi-annual archival package within NCEI and assigned a digital object identifier (DOI). This facilitates citability and replication of the database and image collection over-time. This process is a pioneering effort to create, collect, and maintain AI-ready data for the oceanographic imaging community. The goal is to foster collaboration and innovation, empowering NOAA and its partners to leverage AI technologies effectively for advancing environmental research and stewardship.

Returning to the Deep: Maximizing Insights from Deep-Sea Coral and Sponge Imagery from the U.S. Pacific Islands

Savannah Goode , Deep Sea Coral Research & Technology Program, NOAA

Kiki Kamakura (Barnard College, NOAA Pacific Islands Fisheries Science Center), Beth Lumsden (NOAA Pacific Islands Fisheries Science Center), Heather Coleman (Deep Sea Coral Research & Technology Program, NOAA)

One major disadvantage to seafloor image analysis is the bottleneck created by the time required to manually annotate imagery for scientific study. It is thus uncommon for imagery to be re-analysed after the original study is completed. Consequently, it is difficult to combine imagery datasets to conduct broader-scale investigations, which can be particularly important when studying habitats that are difficult to survey, such as the deep-sea benthos. A second issue that has arisen from limited re-annotation of seafloor imagery is that many existing image datasets are composed primarily of images with non-localized annotations (i.e., records of different taxa and/or individuals not differentiated spatially within a single image). Such images cannot be readily incorporated as training data for automated annotation pipelines, nor can they be easily used by non-experts for identification purposes. We sought to address these issues of non-standardized, non-localized annotations using photos obtained from [NOAA’s National Database of Deep-Sea Corals and Sponges](https://www.ncei.noaa.gov/maps/deep-sea-corals/mapSites.htm), focusing on the U.S. Pacific Islands region due to the abundance of high-resolution images available. Using BIIGLE 2.0 (an online image annotation platform), 14,226 images were initially assessed to determine whether they comprised a single taxon or a wider ‘community’. Images labeled ‘community’ (3,230 total) were selected for re-analysis. Each image was annotated to characterize the visible substrate (using the Coastal and Marine Ecological Classification Standard ([CMECS](https://iocm.noaa.gov/standards/cmecs-home.html)) classification) and to localize the taxon of interest (using rectangular bounding boxes). This dataset will be used to: (1) investigate taxon-substrate relationships in this region, (2) provide representative _in situ_ taxon images for non-experts, (3) train automated classification tools, and (4) inform a national guide to identify important and vulnerable deep-sea habitats, currently in development by the NOAA Deep Sea Coral Research & Technology Program and the Bureau of Ocean Energy Management.

Tackling exponential increases in deep-sea video data: VARS + AI

Lonny Lundsten , Monterey Bay Aquarium Research Institute

Nancy Jacobsen Stout (MBARI), Kyra Schlining (MBARI), Kristine Walz (MBARI), Larissa Lemon (MBARI), Megan Bassett (MBARI), Brian Schlining (MBARI), Kevin Barnard (MBARI), Danelle Cline (MBARI), Duane Edgington (MBARI)

The Monterey Bay Aquarium Research Institute (MBARI) has made significant advancements towards incorporating machine learning (ML) into the annotation and analysis of our 37-year archive of deep-sea video. Central to these efforts is the integration of the Video Annotation and Reference System (VARS) with advanced ML tools and the development of a new machine-assisted annotation workflow. VARS is a software system developed by MBARI to annotate and manage video data collected from remotely operated vehicles (ROVs), autonomous underwater vehicles (AUVs), and other camera platforms. Using VARS, researchers create detailed annotations of marine organisms, geological features, and habitats from deep-sea video footage using a structured format within a searchable relational database. Visual observations are merged with additional data collected during vehicle deployments, such as salinity, temperature, position, and depth. Historically, MBARI’s Video Lab staff have manually annotated all deep-sea videos recorded by MBARI's ROVs in a consistent manner. However, the recent, exponential increase in camera platforms used for conducting surveys has necessitated the development of innovative workflows to handle this massive influx of visual data, which is rapidly outpacing the feasibility of manual processing. To address this, we have developed new tools and workflows for 1) localizing ML model training data, 2) generating ML proposals rapidly and inexpensively, and 3) validating and editing ML generated proposals. To date, we have produced 500,000 deep-sea localizations for ML training and used the subsequent ML models to generate detections on 200 hours of ROV video and 800+ hours of AUV video (PiscivoreCam, i2MAP). These efforts underscore MBARI's commitment to utilizing cutting-edge technology to advance marine research, ultimately providing critical insights into ocean health and ecosystems while fostering collaboration among scientists.

Towards Species-Level Detection of Caribbean Reef Fish

Levi Cai , WHOI

Austin Greene (WHOI), Daniel Yang (MIT and WHOI), Nadege Aoki (MIT and WHOI), Sierra Jarriel (WHOI), T. Aran Mooney (WHOI), Yogesh Girdhar (WHOI)

Visual detectors and classifiers have shown remarkable utility for animal conservation and monitoring tasks by providing rapid estimates of species abundance and biodiversity or can also reduce labeling efforts during data preprocessing tasks. These approaches have been proposed for animals in underwater environments, such as the deep ocean with FathomNet [1]. However, their deployment in coral reef environments still faces many obstacles. Like other marine environments, there is a lack of previously annotated data and the ecological nature of the data faces a long-tail distribution problem where many less common species are difficult to obtain observations about. Distinct from the more controlled environment of imaging animals in the deep ocean, coral reefs present a visually rich and complex environment. Their location in shallow waters requires visual detectors to cope with significant variations in lighting while corals themselves provide visually confounding characteristics to ecologically interesting species. We have built a new dataset for training and evaluating visual detectors and classifiers at the species-level in coral reef environments. It consists of fish-species labels from diver transect videos across 5 different reef sites spanning over 8 years in the U.S. Virgin Islands. In total there are 162 videos, of which we label 14K images. We also provide tracking information for every fish across each video, enabling novel insights into tracking challenges. Furthermore, we examine the entire pipeline, from data pre-processing, labeling, and training, to deployment. We investigate how best to utilize pre-existing fish detection datasets when applied to our specific domain, either as supplemental data or in transfer learning. We further show results of detectors trained on our dataset that are subsequently deployed on stationary camera or AUV data from the same or nearby sites, to demonstrate the utility and challenges that still remain in deploying these systems even in similar locations. [1] K. Katija et al. “FathomNet: A global image database for enabling artificial intelligence in the ocean” Nature Scientific Reports, 2022.

Towards an AIoT-based Marine Plankton Imaging Network

Jianping Li, Claude , 2. Shenzhen Institute of Advanced Technology, CAS, Shenzhen, China; 3. University of CAS, Beijing, China;4. Xcube Technology Co, Ltd., Shenzhen, China;

Zekai Zheng 1,2, Zhenping Li 2,3, Chi Liu 2, Junqiang Jiang 2,4, Peng Liu 2,3, Liangpei Chen 2,3, Shunming Chen 2,4, Zhisheng Zhou 2,3, Ming Zhu 2,3, Haifeng Gu 5, Jiande Sun 1, Jixin Chen 6. 1. Shandong Normal University, Jinan, China; 2. Shenzhen Institute of Advanced Technology, CAS, Shenzhen, China; 3. University of CAS, Beijing, China; 4. Xcube Technology Co, Ltd., Shenzhen, China; 5. Third Institute of Oceanography, Ministry of Natural Resource, Xiamen, China; 6. Xiamen University, Xiamen, China.

In situ observation of plankton based on underwater imaging should theoretically have real-time and continuous advantages. However, current methods rely almost entirely on post-processing to convert image data into observational information. This results in significant delays in information acquisition and exerts enormous pressure on transmission network bandwidth and real-time processing capabilities required for the massive amounts of data collected from remote instruments. To address these issues, we have developed an Artificial Intelligence of Things (AIoT) marine plankton imaging system as a proof-of-concept to combine edge-computing and cloud-computing to network multiple underwater dark-field imagers (imaging plankton probe, IPP) distributed across various locations, aiming to leverage the high temporal resolution advantage of in situ observations while expanding the spatial coverage of ocean observations. At the edge, a client imager first captures in situ images and then processes them through operations such as object detection, DOF extension, feature extraction, etc. On the cloud side, the server supports complex plankton data management and AI retrieval functions, with an intelligent application layer providing user management, device control and monitoring, and real-time data analysis and visualization functions. Compared to traditional methods, this AIoT-based multi-location underwater imaging approach has significant advantages. The system optimizes computing and storage resources, enhances real-time data processing and security, improves transmission reliability, and reduces deployment and maintenance costs. By deploying multiple IPPs in various locations in China and Australia for periods ranging from one to eight months, we have preliminarily demonstrated the capability of this innovative AIoT-based paradigm to monitor complex marine planktonic dynamics, such as the bloom and decay of Noctiluca scintillans. This offers a more cost-effective solution for expanding global marine plankton observation capabilities in the future.

Using Computer Vision to Objectively Quantify Features in Geological Drill Core

Lewis Grant , University of Southampton

Rosalind M. Coggon (University of Southampton), Blair Thornton (University of Southampton), and Damon A. H. Teagle (University of Southampton)

Ocean crust paves two thirds of Earth’s surface and its physiochemical exchanges with seawater circulating through it exert a primary control on oceanic and atmospheric composition through time. Quantifying the rate and nature of this exchange of heat, elements and nutrients is imperative if we are to understand long term climate, changing ocean conditions, and possible sites for the origin of life. Drilling into in situ ocean crust, as well as ancient crustal slabs obducted onto land (ophiolites) provides valuable records of these interactions. Geologists painstakingly document the distributions of petrographic features in recovered cores, based on visual assessments. Core description is therefore limited to the time and resources available during core recovery and prone to human bias. As cores are often digitally imaged, we have explored the potential benefits computer vision provides to the expert geologist tasked with describing the recovered material. We have demonstrated that self-supervised machine learning methods are capable of accurately identifying and quantifying features within core images, and that the addition of spatial metadata during network training improves accuracy by as much as 50 %. Using the classification output of a spatially-guided contrastive learning model (GeoCLR), important geological information has been quantified through 400 m of core at a resolution impossible for a human expert to generate in the same time. This work highlights the promising potential of using machine learning to extract valuable information from large unutilised image datasets, although improvements in core imaging best-practices are needed to improve the machine readability of future core imagery. Furthermore, automated core description has wide-reaching potential to improve the efficiency of numerous industries such as geoengineering, remote sensing and mineral exploration.

Using machine learning to investigate temporal dynamics of methane seep fauna at the Ocean Observatories Initiative Regional Cabled Array

Katharine T. Bigham , University of Washington

Michael F. Vardaro, University of Washington; Deborah S. Kelley, University of Washington

Methane hydrate seeps are unique chemosynthetic marine environments where organisms use subsurface-derived chemicals and volatiles rather than sunlight to create energy. Seeps are found on continental margins worldwide; >800 gas emission sites have been imaged along the Cascadia Margin. The seeps provide important services such as energy storage in the form of methane hydrates, habitats for commercially important fish stocks, and contribute to regional biodiversity and productivity. One of these methane seeps, Southern Hydrate Ridge (SHR), has hosted a diverse suite of instrumentation on NSF’s Ocean Observatories Initiative (OOI) Regional Cabled Array (RCA) that has been streaming real-time data to shore for a decade. Among the instruments is a SubC digital still camera, which takes images every half an hour resulting in >50,000 images in a year. Additionally, ROV imagery (still, HD, 4k) are collected at the site annually as part of the RCA operations and maintenance cruises. In concert, this has amounted to a collection of >37 TB of visual data that has the potential to provide important insights into the evolution of these seeps over time, as well as into the dynamics of the benthic fauna community that thrives there. Due to the time-consuming nature of manually annotating the large amount of SHR imagery that the RCA has collected and continues to collect, machine learning pipelines are being developed to annotate imagery as it is captured. This effort will serve as an important contribution, by making the OOI-RCA imagery more accessible to researchers and the public, and product integration with a wealth of available stored and real-time environmental data. We will present preliminary results from those pipelines, discuss methods being used to integrate human verification in the pipeline, give an example of the type of data produced by the pipeline, and discuss the ecologic questions that we aim to answer.