The Scripps Plankton Camera system: A framework and platform for in situ microscopy

The large data sets provided by in situ optical microscopes are allowing us to answer longstanding questions about the dynamics of planktonic ecosystems. To deal with the influx of information, while facilitating ecological insights, the design of these instruments increasingly must consider the data: storage standards, human annotation, and automated classification. In that context, we detail the design of the Scripps Plankton Camera (SPC) system, an in situ microscopic imaging system. Broadly speaking, the SPC consists of three units: (1) an underwater, free‐space, dark‐field imaging microscope; (2) a server‐based management system for data storage and analysis; and (3) a web‐based user interface for real‐time data browsing and annotation. Combined, these components facilitate observations and insights into the diverse planktonic ecosystem. Here, we detail the basic design of the SPC and briefly present several preliminary, machine‐learning‐enabled studies illustrating its utility and efficacy.

Studying and understanding the fluctuations of planktonic populations is of critical importance to assessing the health and functioning of the ocean. Plankton include the primary producers in the ocean, which form the base of the food web. Through carbon fixation, nutrient uptake, and oxygen production, these organisms influence global scale biogeochemical cycles (Arrigo 2005;Hays et al. 2005). Sampling these populations is extremely challenging due to the wide range of spatial and temporal scales relevant to their population dynamics (Haury et al. 1978). Comprehensive studies of plankton therefore generally require extensive field campaigns requiring hundreds of hours of human labor.
Established methods for studying plankton are often limited by high financial costs, low temporal resolution, limited spatial coverage, or low taxonomic specificity. Net tows, for example, are used to sample water at a specific time and place in the ocean by filtering water in situ to concentrate biological material. The abundance of target species is then manually enumerated using a microscope in a lab (Wiebe and Benfield 2003). In addition to the logistical difficulties of deployment, and the human costs associated with organism enumeration and identification, water and net sampling has been shown to significantly under-sample environmental conditions and fragile and gelatinous organisms (Remsen et al. 2004;Benfield et al. 2007;Jochens et al. 2010). At the other end of the sampling spectrum, fluorometers estimate chlorophyllcontaining organismal abundance; they sample continuously, generating a numeric indicator that is correlated with primary producer abundance (Cowles et al. 1993;Kolber and Falkowski 1993). While allowing for dense temporal sampling, fluorometers generally integrate over a small volume and yield bulk measurements of fluorescent organisms.
Since the early 1990s, oceanographers have been developing and deploying digital in situ imaging systems to address the need for taxon-specific data captured at high temporal or spatial resolution (Benfield et al. 2007). The abilities of such optical instruments to sample organisms in the ocean are well established. Indeed, the detailed population time series or maps of spatial distributions they yield have allowed new insights into difficult-to-study aspects of plankton dynamics: environmental pressures on plankton populations, the role of gelatinous organisms in ecosystems, parasitic activity, and inter-and intra-species interactions to name a few (Bi et al. 2013;Peacock et al. 2014;Biard et al. 2016).
When designing optical instruments, a sacrifice is necessarily made between camera resolution and the sample volume. An instrument with a given resolution is limited in its ability to quantitatively sample both abundant small objects and rare *Correspondence: e1orenst@ucsd.edu This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. large ones. At the smallest resolution, the system captures many tiny, indistinguishable samples. Conversely, large objects are relatively easy to identify, but are captured so rarely as to be statistically insignificant.
In their study of colloidal particles in Monterey Bay, Jackson et al. (1997) recognized the importance of imaging system roll off-the inability of a particular instrument to effectively sample objects at the largest and smallest ends of its resolvable size range. To make their observations, they used complementary sampling methods ranging from shipboard Coulter counters for micron-scale particles to an in situ tow sled with a planar laser imaging system. Jackson et al. combined all these data streams by computing the minimum detectable particle size and the size spectra from each instrument. These values were then stitched together by computing the volumetric abundance of organisms in each system and combining the results.
We built on Jackson et al.'s insights and determined that effective quantification of organisms across the whole size spectrum of a plankton population requires a suite of instruments with overlapping spatial resolution, deployed in a variety of configurations. In principle, any imaging system or combination of systems could be used in this manner. Critical to this framework is the development and consistent maintenance of a serverbased data management system to collate data from all inputs.
Optical-based instruments, such as the Imaging FlowCytobot, the In Situ Ichthyoplankton Imaging System, the Underwater Vision Profiler, the Zooglider, the Continuous Particle Imaging and Classification System, and the PlanktonScope have been developed to monitor plankton populations Cowen and Guigand 2008;Picheral et al. 2010;Ohman et al. 2018;Gallager 2019;Song et al. 2020). These instruments, among many others, are being deployed in a variety of environments, in a variety of configurations, and are producing highquality data (Lombard et al. 2019). The Scripps Plankton Camera (SPC) was designed to be a flexible, easily reconfigured imaging system: it can be outfitted to observe objects from tens of microns to several centimeters; it can be deployed on a variety of platforms; it can record data autonomously or to a remote disk; and it operates with a highly extensible database management system.
The SPC was originally developed to augment the Scripps Pier plankton time series currently maintained by the Southern California Coastal Ocean Observing System (SCCOOS) through the Harmful Algal Bloom Monitoring and Alert Network (HABMAP) (Kim et al. 2009;Kenitz et al. 2020). The SCCOOS time series, dating back to 2005, was built from weekly hand-collected net tows and discrete water samples. While this time series is extremely valuable, it misses shorter time scale population fluctuations and likely undersamples fragile and gelatinous organisms. With these limitations in mind, the SPC was designed to sample at high temporal frequency, with minimal influence on the fluid being imaged, and to maximize the ratio of the amount of water sampled to the amount of image data stored.
One way to minimize instrument interference with an ambient population is to use a free-space imaging setup-one in which a light source and a camera are focused on the same distant plane in an open volume. There are several types of free-space imaging systems defined by the illumination location relative to the camera: backscatter systems, where the illumination is directed at the sample from the same angle as the camera; side scatter systems orient a camera to observe a plane illuminated by an oblique source; and transmission systems, in which a light is facing the image plane. Backscatter and side scatter illumination have advantages for imaging opaque objects, but suffer from poor scattering efficiency in dynamic, particulate-rich media like the waters off the Scripps Pier. Transmission imaging performs better in such environments and is better able to resolve objects that have an index of refraction close to that of sea water.
There are a number of relevant transmission imaging modalities, all with various trade-offs. Holography, for example, has several attractive properties for plankton microscopy, such as large imaged volume and the ability to resolve object positions in three dimensions (Sheng et al. 2006). Shadowgraph imaging likewise expands the depth of field by using a collimated light source to allow recording of silhouettes of objects in the beam path (Settles 2012). Darkfield microscopy sacrifices sample volume for enhanced edge contrast and color images for translucent objects (Gage 1920). Moreover, because darkfield imaging uses forward scattered light, it is effective for imaging small objects down to tens of wavelengths of the illumination and provides color information.
The SPC was thus constructed as a free-space, darkfield imaging microscope to maximize data collection in turbid costal conditions, and to observe fragile gelatinous organisms. A single SPC has two housings, one for the illumination hardware and the other for the camera and electronics. The system uses no nets, filters, or pumps; it images only particles that enter the sample volume via ambient flow. The embedded computer segments the raw frames in real time, saving only subimages of foreground objects. The resulting regions of interest (ROIs) are then stored either on-board or downloaded externally via an Ethernet connection. An accompanying database management system, user interface, and annotation tools were built in concert to work with the incoming data. This framework images 10s to 1000s of liters of sea water per year-depending on the volume of interrogation as defined by the specific optical setup-with minimal computer memory requirements. For example, the instrument package deployed on the Scripps Pier uses a pair of microscopes with 0.5× and 5× objective lenses that collectively produce approximately 5 terabytes of foreground ROIs out of 2000 terabytes of raw image data over a year of continuous operation.
The SPC's adaptable configuration lends itself to straightforward integration and expansion; the same physical footprint can be used to observe many different size ranges and the data can be saved without manipulation using the same data management system. This basic framework could enable many new sampling designs, long-term monitoring efforts, and experimental setups. Several versions of the SPCS are already in service in ocean environments from the Gulf of Alaska to the Cayman Islands and freshwater systems from the Sacramento River to the Greifensee Lake in Switzerland (SI 1). The original system, deployed as a permanent installation on the Scripps Pier in 2014, has collected more than 1 billion ROIs and has facilitated observations of fragile gelatinous organisms, ongoing studies of episodic blooms of extremely long diatom chains, and characterization of forms of parasitism never recorded before in the Pacific.

Materials and procedures
Model A fundamental issue in particle imaging is the trade-off between microscope resolution and effective sample rate for a given sized object assuming a constant size-distribution of organisms. To better quantify this trade-off, we developed a physics-based model of a free-space imaging system to guide the instrument design. The model computes the amount of time needed to collect a desired number of images of a range of object sizes given an underlying particle size-distribution, a microscope magnification, and a sensor size. Our hypothetical system assumes a free-space light microscope with a welldefined relationship between optical resolution and the resulting imaged volume. The model, however, could be adapted to evaluate the same trade-offs with different imaging modalities such as holographic or plenoptic imaging.
The resolution and sample volume of a diffraction-limited microscope are defined by the numerical aperture (NA) of its objective. NA is a dimensionless number that describes the range of angles over which a lens can collect light: where n is the index of refraction of the medium (e.g., air or water) and θ max is the angle describing the maximum cone of light picked up by the lens. The size of the finest detail that can be resolved by the microscope d is inversely proportional to the NA.
with λ representing the wavelength. The constant is defined by the Rayleigh criterion for resolving two point sources of light on an image plane (Hecht 2016). We rearrange Eq. 2 to express the NA as a function of the desired minimum resolution and an assumed wavelength of collected light.
In this illustrative model, the smallest resolvable object d is a constant defined by x obj , the size of a pixel in the image plane: x obj is in turn defined by the pixel size on the sensor, x pix , and the magnification, M.
The maximum angle over which the microscope can collect is likewise defined by the NA of the system.
The depth of field over which an object can be resolved is then: where β is a constant describing the amount of acceptable blur and x obj is the pixel size in object space (Eq. 5). This model assumes that blur is purely a Gaussian function of distance from the image plane described by the shape of the aperture (Joshi et al. 2008). The volume of water observed in an individual frame, v f , is then computed as the product of the full sensor size and the DoF, taking care to convert to the appropriate units.
where i and j are the dimensions of the sensor in pixels and vol is a volume conversion factor. The rate of objects imaged per unit time is a function of particle concentration per unit volume κ and the system frame rate f where v f is the computed sample volume from Eq. 8. In all model runs, κ was set assuming an allometric size-scaling of metabolic rate constraining the size abundance distribution of the observed plankton (Huete-Ortega et al. 2011). Defining a "sample" as a collection of imaged objects, dividing Eq. 9 by f stat -the number of objects needed to create a statistically relevant sample-yields the rate of statistically significant samples per unit time: The model then outputs the number of samples collected per hour as a function of plankton concentration, the system magnification, and frame rate.
When designing the SPCs, we used the model to visualize the sampling rates r samp for six different microscope objectives with magnifications between 5× and 0.05×. The model assumed that each system used a 12 megapixel sensor, an object must occupy at least 20 pixels to be counted, and the sample ratio f stat was set to 100 objects per sample (Supporting Information Table S1). Each line in Fig. 1 represents the sample rate of each microscope objective when operating at eight frames per second in an environment composed purely of plankton.
The left-most end of each line is the sample rate r samp of small objects imaged with enough pixels to be counted. This is the lower marginal performance boundary of an objectivethe case in which the camera images many low-resolution objects. The right-most terminus of each line is at a sample rate r samp of 0.01 samples per hour, corresponding to a single object per hour when the desired sample ratio f stat = 100 objects/sample. Moving from higher to lower power objectives yields an increase in sample rate r samp obtained by lowering magnification while maintaining at least 20 pixels per object diameter.
Consider, for example, a system akin to the MICRO-SPC fitted with a 0.5× microscope objective (Table S2). With a 20 pixels per body length lower bound, the smallest object the instrument will image has a length of 138 μm. At eight frames per second, the system will capture approximately 55 samples comprised of 100 ROIs every hour (r samp = 55). This 0.5× microscope will image larger targets rarely-a 5 mm object will only show up in the data approximately once an hour (r samp = 0.01-necessitating long binning intervals (hundreds of hours) to acquire a sample with a sufficiently high signalto-noise (SNR) ratio.
This model informed our design criterion for all deployments. All systems were developed in close coordination with the teams using the instruments to ensure that the target plankton of interest would be identifiable and collected with sufficient SNR. When designing the SPC deployment on the Scripps Pier, we desired a lower bound SNR of 10 dB-the system must image 100 objects of a given size to constitute a statistically significant sample-for objects ranging in size from 10 μm to 1 cm in size. No single microscope could sample that broad a size spectrum with reasonable statistics; our solution, therefore, was to use two microscope objectives.
The model results are intuitive: all systems interrogate a volume of water defined by the system design, collecting many images of potentially difficult-to-identify objects at the low end of its resolution. Large, rare objects approaching the size of the imaged volume require a long integration time to capture statistically significant estimates of abundance. These limitations must be considered during the design phase of a system to achieve a desired SNR for a given sized object.
It is important to note that this model only considers biological objects. A free-space imaging system such as the SPC will capture images of any object passing through the sample volume. Objects such as sand, particulate matter, or detritus might not conform to an allometric size-abundance scaling. System performance should not, however, be affected with regard to biological sampling. Once manual or automated processing has been applied to remove noise (i.e., unwanted images), the same principles apply-the time to collect statistically relevant samples in situ will be unaffected by noise.

General system design
We used the model to determine the maximum required time to sample a minimum sized object of interest with reasonable statistics (1/f samp ). To support the weekly Scripps Pier sampling program, the system was designed to resolve objects spanning 2-3 orders of magnitude of body size. From an engineering standpoint, the minimum resolvable size class-as defined by the statistically significant sample rate in the Model section-of a system is determined by several factors: the available camera hardware, imaging sensor format, bus bandwidth, and data rate into the real-time processor. Taken together, these limitations imply that no individual camera system can realistically achieve the desired scientific output. As an example, consider a single camera that samples the size range of 10 μm to 10 cm with only 5 pixels per minimum object diameter. Based on the model, this would require an optical resolution of 2 μm or better and a field of view of 10 cm × 10 cm. It would require 50,000 × 50,000 pixels, or 2.5 gigapixels; an impractical sensor even by futuristic standards. Each line represents a different microscope objective in a telecentric optical set-up imaging onto a 12 megapixel sensor. The lower bound of objects to be counted is set at 20 pixels per characteristic object length and the sample size is assumed to be 100 objects. The MINI-SPC is represented by the blue line, the MICRO-SPC is shown in purple, and the MACRO-SPC is in maroon.
Instead, a suite of cameras with overlapping detectable size ranges is more practical for imaging such a large range of sizes.

Scripps plankton camera
The SPC is a full, end-to-end plankton observation platform. Broadly speaking, the system consists of three distinct nodes: the in situ free-space imaging system, a server array to manage data and facilitate analysis, and an interface for remote clients to observe and annotate images (Fig. 2). The imaging unit is a set of darkfield telecentric microscopes that can resolve objects from tens of microns to several centimeters (Table S2). The server organizes data from any number of underwater cameras and hosts a web interface. Remote users can view data in real-time, add labels, and sort data according to several basic filters. The in situ imaging system can be configured in a number of ways to observe a broad size range of organisms and particles in the planktonic ecosystem. To date, we have developed four distinct in situ systems denoted by their target size range: MACRO-, MINI-, MICRO-, and DUAL-SPC (Table S2). Each one is fitted with different microscope objectives, which change the resolution, sample volume, and data rate. The DUAL-SPC combines the MINI-and MICROsystems into a single housing. All SPC instruments can be deployed autonomously-saving data to onboard storage before offloading to the server, or cabled-saving data directly to the remote server via Ethernet.

Mechanical
The underwater unit of the SPC remains largely the same, regardless of the particular deployment type, and consists of two housings: one containing the embedded computer, camera, and microscope optics; the other has the illumination hardware (Fig. 2). Both housings have a clear port, made of acrylic or sapphire, to transmit or collect light. The two housings are mounted facing each other and are attached by standoffs or a rigid plate. The physical dimensions of each housing vary by deployment type (Table 1). The illumination is powered and triggered by the camera via a 5-pin Subconn cable (MCIL5).

Optical
Microscope A machine vision camera is fitted with a microscope objective and a tube, or telecentric, lens. Together, these optical components both magnify objects and move the image plane away from the port. The SPC has been fitted with a variety of objectives to target different sizes of organisms (Table 1). Further adjustments to the optical design can be made to optimize to a particular sampling protocol. The physical size of the housing containing the system is governed by the diameter and length of the telecentric lens, which in turn is dictated by the desired resolution.

Illumination
Darkfield transmission illumination was chosen to enhance the edges of translucent organisms with an index of refraction Fig. 2. Free-space in situ imaging system concept. (a) Schematic of the entire framework. A subsurface underwater microscope (bottom) illuminates an object (plankton) in free-space using a high-power strobed LED and darkfield optics. The scattered light is imaged by a microscope objective and tube lens on to a camera sensor. An embedded CPU and GPU process images from the camera in real time, saves ROIs to local storage, and sends them to a remote database (when available). On-site data servers host images and provide a client application for remote browsing and annotation of images. (b) An image of the MINI-and MICRO-SPC on the Scripps Pier before deployment. The camera housing is on top and the illumination housing below.
close to that of the surrounding water. This illumination technique improves the system's ability to resolve fine details of objects such as extremities or interior structures that are close to the focal plane. The optical set up also yields color images of transparent objects. Moreover, targets in the sample volume appear bright on a dark background because only scattered light is collected; this facilitates ROI extraction. As with all in situ imaging modalities, the image quality degrades as a function of water turbidity. For systems like the SPC high concentrations of small particles will generate forward-scattered light, eliminating the darkfield effect, and causing any segmentation routine to fail.
The key parameter in designing darkfield illumination is the NA of the illumination. It must be greater than the NA of the imaging lens such that only scattered light is collected by the system (Gage 1920). If even a small amount of unscattered light leaks onto the collection sensor, the contrast will be significantly reduced. In most designs, this NA constraint is achieved by placing a darkfield stop in the illumination path. The size of the stop, and focal lengths of the lenses determine the minimum NA of the illumination. The illumination can be designed easily using nonsequential optics simulation software such as Zemax. However, in practice, the illumination must be adjusted manually after assembly to balance color, intensity and imaging quality. It is essential to ensure the opto-mechanical design allows for these small adjustments.
The MINI-and MICRO-SPC use similar illumination paths: expanding light from an LED point source passes through a collector lens and the central beam is blocked by an opaque stop. The remaining annulus of light is focused at the imaged volume by a condenser lens. An object in the volume scatters light which is then collected by the camera (Fig. 3a).
The MACRO-SPC uses a projection lens design to create a darkfield image over a larger sample volume. The darkfield stop blocks light from the LED at the source, yielding an expanding annulus of light. A series of plano-convex lenses collimate the source light from an LED, retaining enough angle to allow the beam to pass obliquely through the sample volume (Fig. 3b). This design retains the desired contrast enhancing effect.

Biofouling
During the deployment of the SPC on the Scripps Pier, organism growth, sedimentation, and the presence of larger organisms in the sample volume have all interfered with imaging. Several approaches to mitigating biofouling have been tested including water jets blown across the ports, copper components and strobed UV LEDs to limit growth, and mechanical wipers to remove settled material. In practice, we found that the most effective solution has been running Hydro-Wipers (Zebra-Tech; Nelson, New Zealand) once an hour to remove sediment and growth from the ports. This reduced the need for diver servicing of the system to a frequency of once a month. Copper mesh cages of various dimensions have also been used to prevent lobster and fish from inhabiting the sample volume. Other deployments of the SPC have had fewer issues with biofouling due to factors such as geographic location, physical movement of the system, and parking the housings below the euphotic zone (Supporting Information S1).

On-board processing
After an image is initially acquired, an on-board computer segments bright objects from the dark background in real time with an image processing routine written in C++. The software uses OpenCV routines and relies heavily on the multithreading library from POCO with a wrapper class from Open Frameworks (Bradski 2000). Raw, full frame images are downsampled by a factor of 2 or 4 by block-averaging pixels. This step is critical to allow megapixel-scale images to fit in the processor cache and be processed efficiently. A Canny edge detector is then applied to the downsampled images to find objects in the frame and a region filling algorithm is used to close contours. For each ROI in the frame, a bounding box is drawn from the centroid of the region. The dimensions of the box are doubled to ensure that the object is completely segmented and that extremities are not cropped out. Pixels from the raw (not block-averaged) images in each box are extracted and saved locally before being exported to network storage (Fig. 4). Each ROI is tagged with a datetime stamp, the pixel coordinates of the upper left corner of the bounding box, and the ROI area. User-defined thresholds dictate the minimum and maximum areas of the bounding boxes to save and must be experimentally adjusted based on the water quality of the study site.
The first iteration of the SPC, built in 2013 and deployed in 2014, runs a lightweight distribution of Linux on a 1.8 GHz Fig. 4. Schematic of the region of interest (ROI) selection procedure. The raw image is downsampled by averaging pixels into 4-or 16-pixel blocks. A Canny edge detector is then used to detect edges in the image and a region filling algorithm is used to fill closed contours. Bounding boxes with area within a specified range are then mapped back to the original raw image and pixels from the raw image are saved out as ROIs.
Quad Core Odroid XU3 board. With just 2 GB of RAM, this computer ran the camera, segmented ROIs, and saved images at 8 frames per second even when object densities were high. New versions of the system run on an Auvidea J120 carrier board with an NVIDIA Jetson TX1 embedded GPU. The new build allows real-time operation at 20 frames per second, allows simultaneous video and image capture, supports onboard classification with deep neural networks, and can run multiple cameras simultaneously (Supporting Information S1). We note that these are minimum hardware specifications for the operation of the system presented in the current paper. Embedded computing technology is rapidly improving and will undoubtedly yield enhanced performance for future in situ imagers.

Server and database
The server provides redundant data storage for ROIs and metadata, the database-an organization system that sorts the images by morphological features, time, taxonomic labels, semantic tags-and web-based tools for browsing, searching, annotating, and categorizing ROIs (Fig. 2). The server loads a new ROI, converts the raw pixels to color, autoscales the pixel values to 8-bit, extracts morphological features such as major and minor axis length and aspect ratio, saves JPEG compressed and uncompressed PNG files to the ROI storage, and creates a new entry. The database entry holds the unique image ID, file path, major and minor axis length, aspect ratio, timestamp in UTC, and image height and width. This design can support 100 s of millions of ROIs in the database in 64 GB RAM. Assuming a file size of 100 kB/ROI, 50 million images requires 5 TB of storage-the equivalent number of full-frame images would require on the order of 1000s of TB of storage space.

Web application
All ROIs can be browsed and downloaded via a JavaScript web application (spc.ucsd.edu). The ROI files are served by a nginx server that proxies requests from a Django-based application through a Gunicorn WSGI HTTP server. The Django application supports a REST API for searching and annotating images using the JavaScript application. The web application itself is highly extensible, allowing for easy modification and feature addition. The web app currently allows users to browse ROIs by date and time, major and minor axis length, aspect ratio, and human labels. Other functionality is easily built into the framework depending on the desired display interface. A static version of the interface can be used to browse images in the field without access to a remote server.

Automated classification
Over time, the SPC can collect millions of individual ROIs, regardless of the deployment type. As with many other plankton imaging systems, the high data rate far outstrips a researcher's ability to manually sort it (Benfield et al. 2007). Automated classification is therefore critical to the ultimate success of the SPC framework. Many plankton researchers have begun looking toward machine learning to alleviate the human cost of classifying the data, and to expedite scientific results (Blaschko et al. 2005;Sosik and Olson 2007;Ellen et al. 2019). Experiments with automated classification, particularly modern deep learning methods, have been conducted throughout the development process of the SPC.
We have selected two studies being done with SPC data and machine learning to illustrate the sorts of information that can be extracted from the system. Each experiment uses a different lightweight neural network architecture, but the same supervised training procedure: fine tuning. Fine tuning takes networks that were previously trained on images of diverse macroscopic objects and refines them to examine plankton. The method improves classification scores when there is limited training data and is demonstrably effective for oceanographic applications (Yosinski et al. 2014;Orenstein and Beijbom 2017). Moreover, the technique allows practitioners to use the best available system from the machine learning community to initialize their system. All training and testing were done on a server-based NVIDIA GTX-1080.

Study 1: Observation of a host-parasite interaction
During human inspection of ROIs collected at Scripps Pier, it was noted that the cosmopolitan copepod Oithona sp. was often infected by the parasite presumed to be Paradinium sp. (Fig. 5). We believe this to be the first observation of this parasite on its copepod host in the Pacific Ocean. Paradinium sp. is a parasitic rhizarian that grows in the copepod's hemocoel, migrates to the digestive systems, is expelled through the anus, and attaches as a cell mass called a gonosphere to the anal somite in the host's urosome (Shields 1994;Skovgaard and Daugbjerg 2008). The life cycle of Paradinium is poorly understood due to difficulties studying it in the lab and the field. The obvious external gonosphere remains attached to the host for less than an hour before bursting and releasing spores into the environment. Moreover, the gonosphere is fragile and can break off, making it difficult to find in net samples. The gonosphere is thus the only portion of the Paradinium life cycle visible in the SPC images.
We took advantage of the SPC's dense temporal sampling to generate a time series indicating the prevalence of the parasite. This entailed training and applying a version of the Ale-xNet convolutional neural network written in Caffe (Krizhevsky et al. 2012;Jia 2013). The process required creating a human-curated training image set, using it to train the computer classifier, and finally applying it to all the images not observed by the human.
Images of parasitized, apparently unparasitized, and ovigerous Oithona sp. were labeled from 58 randomly selected, nonconsecutive 4-h chunks of time during the summer of 2015. All ROIs not belonging to the three groups were considered noise. ROIs from the MINI-SPC were prefiltered by major axis length to between 0.5 and 2.5 mmthe general size range of Oithona with ± 0.5 mm to account for foreshortening-before human observation. Approximately 650,000 ROIs were sorted into the four classes.
AlexNet was fine-tuned from a version originally trained on ImageNet data (Russakovsky et al. 2014). The final three fully connected layers were removed and retrained with the human labeled Oithona images. The network was then trained for 40,000 iterations with a base learning rate of 0.0002. The final classifier achieved 89% accuracy on an independent test set. The trained network then classified all ROIs captured by the MINI-SPC from March 2015 to April of 2016.
Study 2: Tracking bloom forming species A classifier was developed to find potential harmful algal bloom (HAB) formers in the MICRO-SPC time series to assist SCCOOS researchers. Thirteen common taxa, including seven potential harmful algal bloom formers, and eight noise categories were identified for this preliminary study. A human domain expert sorted objects into these categories using the SPC online interface. The annotated data set was then used to fine tune a deep residual network: the ResNet34 implementation, originally tuned for ImageNet data (He et al. 2016).
Each class was represented by 1000 labeled examples: 800 for training and 200 for validation. Classes with more than 1000 examples were subsampled and those with fewer were augmented with randomly affine-transformed images as needed (Orenstein and Beijbom 2017). The best resulting classifier was applied to all MICRO-SPC data from 2018 (Fig. 6).

Performance at the Scripps Pier
The SPC was originally deployed on the Scripps Pier in La Jolla, California. The system was designed to supplement the existing Scripps Pier plankton time series that monitors harmful algal bloom species at weekly intervals. Water samples and net tows are performed at the end of the pier and the samples examined under a microscope to enumerate species. The SPC was envisioned as a tool to track population fluctuations during the periods between the net samples.
Researchers also wanted to study local zooplankton populations in addition to the phytoplankton and microzooplankton of interest to the HAB monitoring program. These plankton range in size from tens of microns to several centimeters. A single free-space imaging system could not effectively sample this entire size spectrum.
Combining the MINI-and MICRO-SPC allowed us to effectively image organisms over the entire desired size range. The system uses 0.5× and 5× microscope objectives contained in separate housings. The two units are attached to a single frame mounted to a pier piling and connected to a surface unit via a single Subconn Ethernet cable.
Together, the two cameras collect thousands to millions of ROIs per day depending on the ambient particle density. All ROIs are stored on a remote server. To date, the pair of instruments has collected over 1 billion individual ROIs. The system has sampled during blooms of long chain-forming diatoms, captured short-term fluctuations of ecologically important species, observed high volumes of gelatinous organisms, and imaged a form of parasitism never observed at the Scripps Pier.

Machine classification
Preliminary classification work has been done using data from both the MINI-and MICRO-SPC on the Scripps Pier. We stress that these are preliminary results and do not claim to draw firm ecological conclusions. They are presented here to illustrate the procedure and types of questions that can be addressed.

Oithona
A version of AlexNet was fine-tuned and run over all data in the appropriate size bin for a calendar year to search for Oithona sp. and a possible parasite. The classifier achieved an accuracy of 89% on an independent test set. To estimate the performance of the classifier, a human expert observed all data sorted by the machine annotator into the three classes of interest between March and August of 2015 (Table 2). The classifier had the most trouble identifying egg-bearing Oithona, falsely identifying 32% of the ROIs.
We note that the false negative rate is extremely low. A subset of ROIs classified as noise was also observed to estimate the classifiers false negative rate. A total of 29,459 ROIs were examined; only 1 parasitized, 1 ovigerous, and 3 healthy Oithona sp. were missed by the CNN-a false negative rate of approximately 0.0001%.
The classifier output was used to estimate fraction of parasitized and ovigerous individuals relative to the total local Oithona population between March 2015 and April 2016 (Fig. 7). Counts were binned on a daily basis and smoothed with a 24-h Gaussian filter. While these outputs are uncorrected for false detections, the low false negative rate  derived from the validated data indicates that the gross patterns are indicative of the true signal.

Harmful algal bloom
A ResNet34 was fine-tuned using a small set of images from the MICRO-SPC to observe fluctuations in phytoplankton and microzooplankton. The machine annotator was highly accurate on an independent test set, scoring 98% over the 38 classes. The trained classifier was used to sort a random sample of 10,000 ROIs from each day of 2018.
Previous studies have noted that classifier performance varies as a function of changes in the underlying population being observed (Moreno-Torres et al. 2012). Examples of socalled "dataset shift" have been noted in time series studies of plankton (González et al. 2017). To examine the issue in the context of the SPC, we estimate the classifier's performance on four classes in two contexts: elevated and normal prevalence of an organism of interest.
Spikes in the abundances of Akashiwo sanguinea, Cochlodinium spp., Lingulodinium polyedra, and Polykrikos spp. were highlighted by computing the fractional anomaly on a class-by-class basis at daily intervals. The fractional anomaly is defined as the total number of ROIs of a class on a given day relative to its annual mean. Thus, if the classifier identifies a greater number of ROIs than the mean, the fractional anomaly is greater than one. The larger the fractional anomaly, the more unusual the observation (Fig. 6). We define "normal" as periods of time when the fractional anomaly was within two standard deviations of the mean.
To estimate the false detection rate under normal relative abundances, 10 ROIs of the organism were randomly selected from the classifier output on 20 random days. When the classifier selected fewer than 10 ROIs on a given day, all ROIs were retained. The process was repeated until 200 ROIs had been selected. A human expert examined a mosaic of these ROIs and counted the number of false positives (example for A. sanguinea; Fig. S1). The average false detection rate during these low abundance periods was quite high for all organisms.
A single day of elevated conditions was evaluated for each of the 4 example organisms. Two hundred random ROIs from the class of interest were drawn and examined to estimate the false detection rate (example for A. sanguinea; Fig. S2). Likewise, 200 random samples were drawn from all other classes to produce a false omission rate (example for A. sanguinea; Fig. S3). Note that this procedure only provides a single estimate of the classifier performance during periods of elevated abundance. The false detection rate on elevated days is substantially lower than during baseline periods. Likewise, the false negative rate appears negligible for all four of the evaluated organisms evaluated on their peak abundance days (Table 3).

Discussion
When the SPC was under development, we envisioned it as a digital plankton net: a system that would capture and analyze undisturbed organisms in a given size range, eliminating the need for time-consuming and costly enumeration of physical specimens. The SPC has indeed been a successful filter, acquiring images of objects large and small in their natural habitat. Moreover, it has enabled novel sampling designs by densely sampling in time, producing data in real time, and being capable of untethered deployments.
There is, however, no single instrument that can observe the entire size spectrum of the myriad microscopic denizens of our oceans. There are fundamental physical limitations, defined both by instrument design and the environment itself, to what can be observed by any single instrument. Taking both the instrument and the environment into account is critical to developing a system and designing an experiment or observational schematic to appropriately address a particular question.
The trade-offs between system resolution and expected observational ability must be carefully considered when designing experiments to target a population of interest. Our model of free-space microscopy performance as a function of planktonic size spectra is an effective guideline for selecting the appropriate hardware. It also serves to highlight when multiple systems are appropriate and how to maximize their efficacy for a given sampling protocol.
The SPC lends itself to designing studies in this way. The system's flexibility allows it to be deployed in many configurations and on a variety of platforms. When designing any experiment with imaging microscopes, the entire framework-the in situ imager, real-time image processing, and consistent database practices for storage and annotation-must be treated as a cohesive whole. Acting in concert, these elements enable arbitrary expansion of a plankton sampling infrastructure. One can imagine using such a system to develop a distributed network of complementary in situ observatories working together to better understand the planktonic ecosystem (Lombard et al. 2019).

Hardware comparison
The utility of the SPC contributes to an increasingly broad universe of plankton imaging systems. Lombard et al. (2019) Table 4 follows the layout of their comparison to contextualize the SPC (see table 1 in Lombard et al., 2019). The choice of a dual resolution deployment of the SPC on Scripps Pier is akin to the use of two magnifications on the VPR (Davis et al. 2004). In both cases, the research teams determined that sampling the whole desired size range required parallel imaging systems. The VPR, however, is strictly a towed array and is not well-suited for moored deployments. It is also not appropriate for fully autonomous deployment as it does not have on-board processing capabilities. Other instruments have been designed that can be built with several magnifications, but none to our knowledge have been deployed in parallel, in situ.
Many of the systems discussed in Lombard et al. (2019) were purpose-built for a particular deployment type-towed or moored, in situ or deck-based-and targeting of certain organisms. The diversity of sampling methodologies makes direct comparison very challenging. Lombard et al. present a schematic visual aide describing the range of sampled organisms, but do not compare the sample rate (time to collect a statistically relevant number of ROIs of a given size) of each system.
The SPC is not a replacement for any these instruments. It is instead best viewed as a complementary system. For example, when the SPC is moored it sacrifices spatial resolution for temporal resolution; a mobile or profiling system might add an additional dimension to the protocol. The range of body sizes of plankton, the temporal variability in their populations, and the spatial heterogeneity in their distributions necessitate a holistic sampling approach-one that integrates several systems with overlapping resolutions and different deployment strategies.

On-board processing
Part of what makes the SPCS adaptable and reconfigurable is the embedded computer. Having processing power and storage on-board makes it relatively easy to reprogram the instrument for different missions. We have experimented with some alternative deployments that have used the SPCS for tow-yos and vertical profiles with minimal hardware modifications aside from the addition of a battery pack (Supporting Information S1). Many of these configurations have required the system to collect data on-board until the instrument is recovered. The ability to process images into ROIs in real-time allows the instrument to operate for much longer without filling the local storage.
One can imagine reconfiguring and reprogramming the instrument to sample in many environmental conditions with varying degrees of human intervention. The most extreme case might be a remotely deployed system transmitting relative species abundance estimates via satellite. Doing so would require substantial effort in quantifying ROI extraction performance and understanding the limitations of the trained automated classifier (Bi et al. 2015;Orenstein et al. 2020).

Automated analysis comparison
Many groups have begun integrating machine learning techniques into their workflow. Indeed, it is a necessity as increasing amounts of data are collected by digital imaging systems. These processing techniques are all very similar, implementing some form of supervised learning-training an algorithm on a curated set of labeled images, evaluating using an independent subset, and then applying the algorithm incoming data. In the past 5 yr, virtually all plankton image classification schemes have adopted a flavor of deep learning (Luo et al. 2018;Ellen et al. 2019;Briseño-Avena et al. 2020). These methods are quite accurate when evaluated using a random subset of the training data, typically achieving accuracies around 90%. The two studies with SPC data outlined above are no different; both did well on independent test sets.
It is not surprising that tests with independent subsets of training data are uniformly high. Despite high accuracy in testing, system performance often degrades when applied to new incoming data; an issue known as dataset shift in the machine learning community (Moreno-Torres et al. 2012). Our results also suffered from dataset shift, with accuracies dropping when considering new images (Tables 2 and 3). We believe that our approach of performing further human evaluation on new classifier output will help mitigate errors in population estimates . The server backend is built for compiling the output of classifiers and quickly displaying the results via the front-end web interface. This type of pipeline is crucial to maximize the work hours of human experts.

Automated analysis considerations
System performance will be affected by changes in the relative distributions of particles in the water. This could manifest as a bloom saturating the signal, a storm suspending sand or other particles, or larger organisms taking up residence in the sample volume. These noise sources vary dramatically among locations and the population of study. Specific environments and deployments will require careful consideration when selecting instruments, selecting onboard filtering criteria, and making storage provisions.
We have dealt with noise in two primary ways: (1) sorting images offline to remove the noise when there is a consistent underlying noise source; or (2) subsampling when a noise source is abundant enough to overwhelm the real time processing. Removing noise in post-processing is simply a classification procedure: a system is designed to remove the irrelevant data points. When the noise is high enough to inhibit the onboard image processing, the data rate can be reduced to accommodate the influx of objects. This can be done either by saving a random assortment of ROIs or, in extreme cases, saving full-frame images at longer intervals.

Automated and manual annotation
With the rapid development of digital imaging systems comes a concomitant data problem. For systems like the SPC to be truly effective, careful consideration must be given to classification procedures. While there are promising early returns for unsupervised classification, the majority of imaging systems currently make use of expert trained algorithms (Schröder et al. 2020). In either learning paradigm, special attention must be paid to how the highly trained human annotators sort the dataset. Well-designed software tools and experimental procedures can facilitate rapid development of effective automated classifiers (Gomes-Pereira et al. 2016).
The two case studies presented in this work demonstrate both the feasibility and limitations of such procedures. In both cases, we designed deliberately biased classifiers by forcing the training set distribution to be even rather than mirroring the relative distributions of the underlying populations. This was done due to the limited available training data and a desire to make them sensitive to large population changes (González et al. 2017). Indeed, both classifiers acted as effective detectors, if not perfect classifiers.
These preliminary studies with machine learning demonstrate that such techniques could eventually be quite effective. In future work, we will validate the output by directly comparing estimated relative abundance from the cameras on the Scripps Pier to values observed by the SCCOOS HABMAP program (https://sccoos.org/harmful-algal-bloom/). Consideration will also be given to understanding how the shifting nature of the population being observed affects the output of a classifier (Moreno-Torres et al. 2012). Developments in this area will facilitate "smart" sampling by using trained classifiers onboard the camera's embedded computer to further filter data. In the ideal case, such systems would be able to output numbers of organisms rather than images, enabling autonomous deployment of imaging systems on long duration platforms.
Free-space imaging limitations When using traditional microscopy with free-space imaging, the imaged volume is inherently size-dependent: small objects will blur into the background over a shorter distance along the optical axis than large ones. This has been characterized in our model and to a large extent does not significantly impact studies of changes in relative abundance. However rigorous calibration is necessary when moving from relative to absolute abundance. We envision accomplishing this through a series of dilution experiments for species of interest. A known concentration of the species is prepared, diluted, and subsequently imaged by a bench top version of the system. This is then repeated for several different size classes and an effective imaged volume is estimated for each size. This is relatively straightforward for phytoplankton but likely quite challenging for larger zooplankton. In that case, constraining the distance between viewports to be within the depth of field provides a better solution. This method is used for the MACRO-SPC to provide a known imaged volume.