Skip to content

Transformation spec proposal for NGFF

Jump to bottom
bogovicj edited this page Nov 16, 2021 · 14 revisions

Coordinate space

Right-handed coordinate system (x→, y↓, z↗), positive rotation angles rotate clock-wise (↻). Index according to axis index (x:0, y:1, z:2, ...).

Transformations

Transformations are specified as forward transformations, i.e. a transformation transforms a vector from source into target space, a sequence of transformations is applied from first to last element. Rendering transformed images or volumes typically goes through the inverse of a transformation and some transformations cannot be inverted easily. If the sole purpose of a transformation is to be inverted (as in for rendering an image or volume), then its inverse can be explicitly defined as inverse_of an otherwise undefined transformation.

JSON elements

Each transformation has a String:type property from the below list and defines how other fields are interpreted. More transformations and their fields will be added to the list as they become available.

Transformation types

type fields description
identity identity transformation, is the default transformation and is typically not explicitly defined
sequence Transformation[]:transformations sequence of transformations, an empty sequence is the identity
translation one of:
number[]:translation
String:path 		translation vector, stored either as a number[] (translation) or as binary data at a location in this container (path). If both are present, path is preferred. length of vector defines number of dimensions
scale one of:
number[]:scale
String:path 		scale vector, stored either as a number[] (scale) or as binary data at a location in this container (path). If both are present, path is preferred. length of vector defines number of dimensions
affine one of:
number[]:affine
String:path 		affine transformation matrix defined as list consisting of n sets of n + 1 scalar numbers, stored either as a number[] (affine) or as binary data at a location in this container (path). If both are present, path is preferred. n is number of dimensions
d_field String:url
String:path(null)
boolean:interleaved(true) 		deformation / displacement field storing one offset for each grid coordinate, url points to the data container (e.g. "/field.hdf5" or "/field.tif" and path points to the dataset if the container can hold multiple datasets or is empty for single dataset containers
p_field String:url
String:path(null)
boolean:interleaved(true) 		position field storing one absolute position for each grid coordinate, url points to the data container (e.g. "/field.hdf5" or "/field.tif" and path points to the dataset if the container can hold multiple datasets or is empty for single dataset containers
inverse_of Transformation:transformation inverse of a transformation

Examples

translation 
{
  "type" : "translation",
  "translation" : [-1, 2.2, 0]
}
inverse_of / scale 
{
  "type" : "inverse_of",
  "transformation" : {
    "type" : "scale",
    "scale" : [1.1, 2.2, 3.3]
  }
}
affine (2d) 
{
  "type" : "affine",
  "affine" : [2.0, 0.1, 10, -0.1, 2.0, -10]
}
affine (2d) path 
{
  "type" : "affine",
  "path" : "/a/volume/affine_parameters"
}

where path in this container must contain a 1d dataset of length 6.

affine (3d) 
{
  "type" : "affine",
  "affine" : [2.0, 0.1, -0.01, 10.0, -0.1, 2.0, 0.01, -10.0, 0.0, 0.0, 1.0, 5.0]
}
sequence 
{ 
  "type" : "sequence",
  "transforms" : [
    { 
      "type" : "translation",
      "translation" : [ -1, 2.2, 0 ]
    },
    { 
      "type" : "scale",
      "scale" : [ 1.2, 1, 2.2 ]
    }
  ]
}
displacement / deformation field hdf5

see this specification

{
   "type" : "d_field",
   "url" : "/path/to/my/JRC2018F_FCWB.h5",
   "path" : "/0/dfield"
}
displacement / deformation field nrrd 
{
   "type" : "d_field",
   "url" : "file:///path/to/my/JRC2018F_FCWB.nrrd,
}

Axis types

This section is preliminary.

Axes may also be tagged with a type that describes the kind of domain (time, space, wavelength, amino-acid residue).

This should be stored in an axes field in a datasets attributes. The value of this field must be an array of length to the dataset's dimensionality. Each element of the array should be an axis object as described below.

See here for a longer list of specifications used for brainstorming.

Proposal A

"axes": {
    "labels" : ["x", "y", "c", "z", "t" ],
    "types" : ["space", "space", "channel", "space", "time" ],
    "units" : [ "µm", "µm", "pixel", "µm", "ms" ]
}

See https://github.com/ome/ngff/issues/35

Proposal B

Example:

"axes": [
{
  "label": "x",
  "type": "space",
  "unit": "µm"
},
{
  "label": "y",
  "type": "space",
  "unit": "µm"
},
{
  "label": "c",
  "type": "channels",
  "unit": null
},
{
  "label": "z",
  "type": "space",
  "unit": "µm"
},
{
  "label": "t",
  "type": "time",
  "unit": "ms"
}

Each axis object has the following fields:

  • label: string
    • an arbitrary string label / name for the axis
  • type: string
    • the type of domain of the axis. e.g. ( space, time,channel )
  • unit: string
    • the unit of the domain, or one of (pixel, px, or null) if arbitrary or not applicable. e.g. ( mm, microns, seconds)

Proposal C

This proposal combines the descriptions of axes and transformations. A benefit of this is that it more clearly describes which dimensions can be "mixed" by transformations (see the alternative below). In proposals A and B, the assumption is that dimensions of the same type can be "mixed".

To be precise, "mixed" means it is possible to interpolate between those dimensions. It also suggests that a flag should indicate whether interpolation is allowed within an axis, i.e., whether its domain is continuous or discrete (see the "Axis types and Interpolation" section below).

The example below shows an example that motivates the inputIndexes and outputIndexes fields. ImageJ's ImagePlus stores dimensions in the XYCZ order, but if these dimensions are stored as XYZC order in the storage container, then a permutation is necessary: a feature that these fields enable.

{ "axes": [
    {
        "type":"spatial",
        "labels":["X","Y","Z"],
        "inputIndexes":[0,1,2],
        "outputIndexes":[0,1,3],
        "transform": {
            "type":"affine",
            "affine":[2.0, 0.0, 0.0, 10.0,
                      0.0, 0.5, 0.0 -5.0,
                      0.0, 0.0, 1.1 0.1 ]
        },
        "unit":"nm"
    },
    {
        "type":"channel",
        "discrete" : true,
        "labels":["C"],
        "inputIndexes":[3],
        "outputIndexes":[2],
        "transform": { "type":"identity" }
    }
  ]
}

The type, and label fields are as above, though here the label field is a String[], and gives a name to the dimension in the output "world" coordinates.

  • transform (Optional) : an object describing a transformation object
    • if not provided, assume the identity transformation.
  • inputIndexes (Optional) :
    • if not provided, assume [ M+1, M+2, ..., M+N-1], where M is the largest integer in the inputIndex fields for all axes in the list prior to this one, and N is the number of dimensions for this axes (length of labels). See the "Inferring indexes" section below for examples.
  • outputIndexes (Optional) :
    • if not provided, assume is identical to inputIndexes

Inferring indexes

Example 1 
[
    {
        "type":"spatial",
        "labels":["X","Y"],
        "unit":"nm"
    },
    {
        "type":"channel",
        "labels":["C"],
    }
]

implies

[
    {
        "type":"spatial",
        "labels":["X","Y"],
        "unit":"nm"
        "inputIndexes":[0,1],
        "outputIndexes":[0,1],
    },
    {
        "type":"channel",
        "labels":["C"],
        "inputIndexes":[2],
        "outputIndexes":[2],
    }
]
Example 2 
[
    {
        "type":"channel",
        "labels":["C"],
    },
    {
        "type":"space",
        "labels":["X","Y"],
        "unit":"nm"
    }
]

implies

[
    {
        "type":"channel",
        "labels":["C"],
        "inputIndexes":[0],
        "outputIndexes":[0],
    },
    {
        "type":"space",
        "labels":["X","Y"],
        "unit":"nm"
        "inputIndexes":[1,2],
        "outputIndexes":[1,2],
    }
]

Axis types and Interpolation

Specifying the type (continuous / discrete) of the domain indicates which axes may be interpolated, which may be jointly interpolated, and how that interpolation should occur.

  • An axis may be interpolated if it's domain is not discrete.
    • example: "angle" axes should be circularly interpolated
  • Two axes may be jointly interpolated ("mixed") if they
    • The types the both axes are equal.
    • They're part of an axis group (as X and Y above), which implies they share a type.

Proposal D

Proposal C associates a transformation with every axis (or group of axes). This proposal associates transformations with input and output axes. A convenient (necessary?) addition to make this clear is to give identities to the "raw" or "data" axis dimensions. In this example we name these dim_$i, as are the defaults for xarray.

In the examples below, we assume the data are stored in XYZC order and are permuted to XYCZ order in order to be imported into ImageJ (similar to the Proposal C examples).

{ "transforms" : [
    {
        "type" : "affine",
        "affine" : [2.0, 0.0, 0.0, 10.0,
                    0.0, 0.5, 0.0 -5.0,
                    0.0, 0.0, 1.1 0.1 ],
        "inputs" : [
            { 
                "label" : "dim_0",
                "type" : "data",
                "index" : 0,
                "unit" : null
            },
            { 
                "label" : "dim_1",
                "type" : "data",
                "index" : 1,
                "unit" : null
            },
            { 
                "label" : "dim_2",
                "type" : "data",
                "index" : 2,
                "unit" : null
            }
        ],
        "outputs" : [
            { 
                "label" : "x",
                "type" : "space",
                "unit" : "um"
            },
            { 
                "label" : "y",
                "type" : "space",
                "unit" : "um"
            },
            { 
                "label" : "z",
                "type" : "space",
                "unit" : "um"
            }
        ]
    },
    {
        "type":"identity",
        "inputs" : [
            { 
                "label" : "dim_3",
                "type" : "data",
                "index" : 3,
                "unit" : null
            }
        ],
        "outputs" : [
            { 
                "label" : "c",
                "type" : "channel",
                "unit" : null
            }
        ]
    }
]
}

Other considerations and options

Consider labeling the data axes outside the list of transformations, like this:

"dataAxes" : [
    { "label" : "i", },
    { "label" : "j", },
    { "label" : "k", },
    { "label" : "l", }
],
"outputAxes" : [
    { 
        "label" : "x",
        "type" : "space",
        "unit" : "um"
    },
    { 
        "label" : "y",
        "type" : "space",
        "unit" : "um"
    },
    { 
        "label" : "c",
        "type" : "channel",
        "unit" : "none"
    },
    { 
        "label" : "z",
        "type" : "space",
        "unit" : "um"
    }
],
"transforms" : [
    {
        "type" : "affine",
        "affine" : [2.0, 0.0, 0.0, 10.0,
                    0.0, 0.5, 0.0 -5.0,
                    0.0, 0.0, 1.1 0.1 ],
        "inputs" : [ "i", "j", "l" ],
        "outputs" : [ "x", "y", "z" ]
    },
    {
        "type" : "identity",
        "inputs" : [ "k" ],
        "outputs" : [ "c" ]
    }
]

Or we could forget about the "dataAxes" and go with indexes of the input like this:

"transforms" : [
    {
        "type" : "affine",
        "affine" : [2.0, 0.0, 0.0, 10.0,
                    0.0, 0.5, 0.0 -5.0,
                    0.0, 0.0, 1.1 0.1 ],
        "inputIndexes" : [ 0, 1, 2 ],
        "outputs" : [ "x", "y", "z" ]
    },
    {
        "type" : "identity",
        "inputIndexes" : [ 3 ],
        "outputs" : [ "c" ]
    }
]

where the types of the axes are inferred.

How should we handle cases in which transforms are applied to non-sequential axes, XYCZ, for example.

"transforms" : [
    {
        "type" : "affine",
        "affine" : [2.0, 0.0, 0.0, 10.0,
                    0.0, 0.5, 0.0 -5.0,
                    0.0, 0.0, 1.1 0.1 ],
        "inputIndexes" : [ 0, 1, 3 ],
        "outputs" : [ "x", "y", "z" ]
    },
    {
        "type" : "identity",
        "inputIndexes" : [ 2 ],
        "outputs" : [ "c" ]
    }
]

The default output axes should be the same as the input axes, specifically XYCZ. Consider allowing overriding the defaults with an outputIndexes attribute. For example, this specification:

"transforms" : [
    {
        "type" : "identity",
        "inputIndexes" : [ 0, 1, 3 ],
        "outputIndexes" : [ 2, 1, 0 ],
        "outputs" : [ "x", "y", "z" ]
    },
    {
        "type" : "identity",
        "inputIndexes" : [ 2 ],
        "outputIndexes" : [ 3 ],
        "outputs" : [ "c" ]
    }
]

converts XYCZ to ZYXC, since it's index mapping is

  • 0 -> 2
  • 1 -> 1
  • 3 -> 0
  • 2 -> 3

Shorthands

Unit

units are optional. If not provided, assume they are arbitrary "pixel" or discrete units.

Type

Consider allowing types to be inferred from labels:

label type
x, y, z, X, Y, Z space
c, C channel
t, T time
v, V vector
m, M matrix
theta, rho, phi angle

As a result, this axis specification:

"axes": {
    "labels" : ["x", "y", "c" ],
}

implies:

"axes": {
    "labels" : ["x", "y", "c", "z", "t" ],
    "types" : ["space", "space", "channel" ],
    "units" : [ "pixel", "pixel", "pixel" ]
}

Dimensionality

What should assumptions be if no axes are specified? Here are some ideas:

2D datasets imply:

"axes": {
    "labels" : ["x", "y"],
    "types" : ["space", "space" ],
    "units" : [ "pixel", "pixel" ]
}

3D datasets imply:

"axes": {
    "labels" : ["x", "y", "z"],
    "types" : ["space", "space", "space" ],
    "units" : [ "pixel", "pixel", "pixel" ]
}

4D datasets imply:

"axes": {
    "labels" : ["x", "y", "z", "c" ],
    "types" : ["space", "space", "space", "channel" ],
    "units" : [ "pixel", "pixel", "pixel" ]
}

5D datasets imply:

"axes": {
    "labels" : ["x", "y", "z", "c", "t" ],
    "types" : ["space", "space", "space", "channel", "time" ],
    "units" : [ "pixel", "pixel", "pixel", "pixel" ]
}
Clone this wiki locally