ADCP Data Processing

A modular framework for processing vessel-mounted Acoustic Doppler Current Profiler (ADCP) data. This code is designed to centralize core processing functions while retaining enough flexibility to process data from a variety of sources and handle different organizational structures with minimal configuration.

Basic processing instructions are managed with input files, while a system of “hook” functions allows the user to implement arbitrarily complex additional processing at various stages of execution.

Configuration

Configuration is abstracted into three levels: cruises, vessels, and deployments.

• A cruise is the highest level of abstraction. This is usually named after the date and location of an experiment involving one or more vessel-mounted ADCPs. The cruise name is used as a directory for vessel- and deployment-specific code, and is the only required input to the outermost processing function.
• A vessel is any ship or other mobile platform on which an ADCP is mounted during a cruise. Each vessel has an entry in the main processing program’s configuration structure, containing basic information about the vessel (such as its name). Separating configuration by vessels is useful for configuring deployments which may share many properties.
• A deployment is the lowest level of abstraction. Think of deployments as discrete units that pass through the processing code. For each deployment, raw data is loaded or parsed and a processed data file is created.

Configuring a new dataset

Cruise configuration

The first step to configuring a new dataset is to assign a cruise name and create a folder within the Code/cruises/ directory. A MATLAB function named after the cruise and prefixed with config_ must be present in this directory. As an example, note that the leconte_2017_sep cruise has a Code/cruises/leconte_2017_sep/ directory containing config_leconte_2017_sep.m. A file of this form is necessary for each cruise.

The function defined in this file has no inputs and returns a Vessels structure as its only output. This structure contains file input/output information and processing instructions.

Using the leconte_2017_sep cruise an an example:

>> Vessels = config_leconte_2017_sep()

Vessels = 

  1x4 struct array with fields:

    name
    deployment
    dirs

Vessel configuration

The 4 entries in the above Vessels example correspond to the 4 vessels used during this cruise: 3 ROSS kayaks (Casey, Rosie, and Swankie) and the MV Steller. The third entry belongs to the ROSS kayak Swankie:

>> Vessels(3)

ans = 

  struct with fields:

           name: 'Swankie'
    deployment: [1x19 struct]
           dirs: [1x1 struct]

The deployment structure will be described in the next section.

The dirs structure has 3 required fields (all are strings):

Field	Description
raw	The base directory from which raw data is read
proc	The base directory to which processed data will be written
figs	The base directory to which figures will be writted

On this particular cruise, a network drive which I mounted as /Volumes/data/ contained raw instrument data. The values of the fields in dirs reflect the organizational structure of the files stored on this drive:

>> Vessels(3).dirs

ans = 

  struct with fields:

     raw: '/Volumes/data/20170912_Alaska/data/raw/ROSS/Swankie/'
    proc: '/Volumes/data/20170912_Alaska/data/processed/ADCP_ROSS/Swankie/'
    figs: '/Volumes/data/20170912_Alaska/figures/ROSS/Swankie/'

Compare this to the dirs structure in the MV Steller’s Vessels entry for an example of handling different organizational structures on a per-vessel basis:

>> Vessels(4).dirs

ans = 

  struct with fields:

     raw: '/Volumes/data/20170912_Alaska/data/raw/ADCP_steller/'
    proc: '/Volumes/data/20170912_Alaska/data/processed/ADCP_steller/'
    figs: '/Volumes/data/20170912_Alaska/figures/ADCP_steller/'

Deployment configuration

The majority of configuration information is stored within each vessel’s deployment structure. Entries in this structure contain processing instructions and information for locating deployment-specific files - they are similar to the vessel Vessels structure in which they are contained:

Field	Description	Type
name	Deployment name	string
tlim	Deployment start and end time	vector
dirs	Deployment-specific directories	struct
files	Deployment-specific files	strucg
proc	Processing options	struct
plot	Plotting options	struct

>> Vessels(3).deployment

ans = 

  1x19 struct array with fields:

    name
    dirs
    tlim
    proc
    plot
    files

The name and tlim fields are straightforward. The other fields are described below. Any field without a default value must be specified by the user. Otherwise, unspecified options will be filled with their default values.

dirs: Deployment-specific directories

Field	Description	Example
raw	deployment-specific raw data directory	'deployment_20170913_132345'

Note: If deployment data is not separated by directories (i.e. all data are within the vessel’s dirs.raw folder), this can be set to an empty string. This is often the case with data logged by ADCP data acquisition programs, e.g. VMDAS, which dump many .ENR (raw ADCP) and .N1R (navigation) data into the same folder.

files: Deployment-specific files

Field	Description	Type	Example
gps	raw GPS filenames	String or Cell	'GPS/*.log'
adcp	raw ADCP filenames	String or Cell	'ADCP/*timestamped*.bin'

If these fields are strings, they are used as completion patterns with MATLAB’s dir function (a full filename will match an individual file). If they are cell arrays of strings, each string will be used as a completion pattern.

In the example shown, file completion patterns with wildcards are used to match all files of a certain type within a directory. However, GPS and ADCP data need not be in distinct directories, and completion patterns need not contain wildcards.

proc: Processing options

Field	Description	Type	Default
heading_offset	ADCP vessel mounting offset	double	none
trim_methods	Method(s) to use for trimming ADCP data	struct	none
adcp_load_func	Name of function to use for parsing raw ADCP data	string	'adcp_rdradcp_multi'
adcp_load_args	Additional arguments to pass to adcp_load_func	cell	{}
adcp_rotation_func	Name of function to use for transforming ADCP data coordinates	string	'adcp_beam2earth'
adcp_rotation_args	Additional arguments to pass to adcp_rotation_func	cell
nmea	NMEA prefixes contained in GPS data files	cell	{'GPRMC','HEHDT'}
skip	A flag to skip processing deployments	bool	false
adcp_raw2mat	A flag to force re-parsing of raw ADCP data	bool	false
gps_raw2mat	A flag to force re-parsing of raw GPS data	bool	false

plot.make_figure: Figure flags

For each entry in this structure, a corresponding figure function will be called if its value is true. For example, if make_figure.summary == true, then the function figure_make_summary will be called using the deployment’s configuration as input. There are several figure functions included in this repository in the Code/figures/ folder.

plot: Plotting options

Fields accessed by figure functions.

Field	Description	Type	Default
lonlim	Longitudinal axis limits for spatial plots	double	none
latlim	Latitudinal axis limits for spatial plots	double	none
make_figure	Structure containing flags for making specific figures	struct	all false
ylim	Vertical axis limits for 2d plots	double	[0 200]

Organizing cruise configuration

It is not necessary to define all fields described in the previous section directly within the cruise’s config function. It might make sense to separate configuration into vessel-specific files which are in turn called from the main config function, e.g.:

Vessels(1) = leconte_2017_sep_casey();
Vessels(2) = leconte_2017_sep_rosie();
Vessels(3) = leconte_2017_sep_swankie();
Vessels(4) = leconte_2017_sep_steller();

Using fill_defaults

It can be tedious and cumbersome to repeatedly specify the same options for deployments that share properties. The ~fill_defaults~ function can be used to avoid this by recursively replacing blank or non-existent fields with those from a template, or “default”, structure. For example, we can fill a structure named incomplete with default values from a structured named defaults:

>> incomplete

incomplete = 

  struct with fields:

         someField: 'I will not be overwritten'
    someOtherField: []

>> defaults

defaults = 

  struct with fields:

            someField: 'I want to overwrite someField!'
       someOtherField: 'I am a default value!'
    anotherOtherField: 'Where did this field come from?'

>> fill_defaults(incomplete,defaults)

ans = 

  struct with fields:

            someField: 'I will not be overwritten'
       someOtherField: 'I am a default value!'
    anotherOtherField: 'Where did this field come from?'

Complete example

See the leconte_2017_sep folder for a complete example of a cruise configuration that has been broken into several files and streamlined using fill_defaults:

• config_leconte_2017_sep
  • leconte_2017_sep_casey
  • leconte_2017_sep_rosie
  • leconte_2017_sep_swankie
  • leconte_2017_sep_steller

Complex configuration with hook functions

There are several points during execution at which the program looks for cruise-specific functions and calls them, if they exist. These functions are called hooks, and are described below with some example applications. It can be useful to set up switch statements within these functions that execute a certain set of instructions if a vessel or deployment name is matched. These functions allow for the addition of arbitrarily complex user-defined processing without modification of the core processing code.

Note that a cruise’s config file is a hook - it is special in that it is required and not named with a _hook suffix, but it is analogous to other hooks described below.

Be careful when using hooks. Compatibility of data structures with the core processing program must be maintained.

Due to MATLAB structure limitations, new fields cannot be added to structures modified by hooks, i.e. deployment must have the same fields as some_hook(deployment) if making an assignment using:

deployment = some_hook(deployment)

To avoid this issue, all custom fields added to structures by hooks should be added to existing sub-structures.

post_setup_hook

[vessel] = post_setup_hook(vessel)

Modify vessel entries in the main Vessels structure before any processing has been done. This can be used to set up user-defined file and directory structures programatically for access in later hooks.

See this example, where an additional entry in each deployment’s files structure has been added for a particular vessel.

post_load_hook

[deployment, adcp, gps] = post_load_hook(deployment, adcp, gps)

Modify a deployment’s adcp, and/or gps data structures after loading raw data, before processing. This can be used, for example, to integrate external data sources into the adcp and gps structures before they are passed through the main processing routine.

During the leconte_2017_sep cruise, the ADCP file naming convention was changed mid-cruise, causing files to be concatenated out of chronological order. Re-sorting data by timestamps in leconte_2017_sep_post_load_hook fixed this issue.

pre_rotation_hook

[deployment, adcp] = pre_rotation_hook(deployment, adcp)

In case one wants to do additional processing that requires beam velocities before the rotation to earth coordinates. For example, remove data from a beam that was contaminated during a certain deployment.

post_rotation_hook

[deployment, adcp] = post_rotation_hook(deployment, adcp)

Similar, but after velocities have been rotated to earth coordinates. Maybe the vertical motion of a CTD was detected during sections of a particular deployment and can be removed with some qick deployment-specific code.