Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better rewrite of NXdata scaling_factor and offset fields #1343

Merged
merged 16 commits into from
Sep 5, 2024
Merged
40 changes: 40 additions & 0 deletions applications/NXmx.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,46 @@
<dim index="4" value="k" required="false"/>
</dimensions>
</field>

<field name="data_scaling_factor" type="NX_NUMBER" optional="true">
<doc>
An optional scaling factor to apply to the values in ``data``.

The elements in ``data`` are often stored as integers for efficiency reasons and need
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
further correction, generating floats. The two fields data_scaling_factor and
data_offset allow linear corrections using the following convention:

.. code-block::

corrected_data = (data + offset) * scaling_factor
paulmillar marked this conversation as resolved.
Show resolved Hide resolved

This formula will derive the corrected value, when necessary.

data_scaling_factor is sometimes known as gain and data_offset is sometimes
known as pedestal or background, depending on the community.

Use these fields to specify constants that need to be applied
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
to the data to correct it to physical values. For example, if the detector gain
is 10 counts per photon and a constant background of 400 needs to be subtracted
off the pixels, specify data_scaling_factor as 0.1 and data_offset as -400 to
specify the required conversion from raw counts to corrected photons. It
is implied processing software will apply these corrections on-the-fly during processing.

The rank of these fields should either be a single value for the full dataset, a
single per-pixel array applied to every image (dimensions (i, j) or (i, j, k)),
or a per-image correction specified with an array whose slowest rank is nP (dimensions
(np, 1), (np, i, j) or (np, i, j, k)).
</doc>
</field>
<field name="data_offset" type="NX_NUMBER" optional="true">
<doc>
An optional offset to apply to the values in data.

See :ref:`data_scaling_factor &lt;/NXmx/ENTRY/DATA/data_scaling_factor-field&gt;` for more information.
</doc>
</field>


</group>

<group type="NXsample">
Expand Down
47 changes: 39 additions & 8 deletions base_classes/NXdata.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -446,21 +446,52 @@
</field>

<!-- Data vs. plot coordinates -->
<field name="scaling_factor" type="NX_FLOAT">
<field name="FIELDNAME_scaling_factor" type="NX_NUMBER">
<doc>
The elements in data are usually float values really. For
efficiency reasons these are usually stored as integers
after scaling with a scale factor. This value is the scale
factor. It is required to get the actual physical value,
when necessary.
An optional scaling factor to apply to the values in any field named ``FIELDNAME``
in this ``NXdata`` group. This can be a :ref:`DATA &lt;/NXdata/DATA-field&gt;` field
(signal or auxiliary signal) or a :ref:`AXISNAME &lt;/NXdata/AXISNAME-field&gt;`
field (axis).

The elements stored in NXdata datasets are often stored as integers for efficiency
reasons and need further correction or conversion, generating floats. For example,
raw values could be stored from a device that need to be converted to values that
represent the physical values. The two fields FIELDNAME_scaling_factor and
FIELDNAME_offset allow linear corrections using the following convention:

.. code-block::

corrected values = (FIELDNAME + offset) * scaling_factor

This formula will derive the values to use in downstream applications, when necessary.
</doc>
</field>
<field name="offset" type="NX_FLOAT">
<field name="FIELDNAME_offset" type="NX_NUMBER">
<doc>
An optional offset to apply to the values in data.
An optional offset to apply to the values in FIELDNAME (usually the signal).

See :ref:`FIELDNAME_scaling_factor &lt;/NXdata/FIELDNAME_scaling_factor-field&gt;` for more information.
</doc>
</field>

<field name="scaling_factor" type="NX_FLOAT" deprecated="Use FIELDNAME_scaling_factor instead">
<doc>
The scaling_factor and FIELDNAME_scaling_factor fields have similar semantics.
However, scaling_factor is ambiguous in the case of multiple signals. Therefore
scaling_factor is deprecated. Use FIELDNAME_scaling_factor instead, even when
only a single signal is present.
</doc>
</field>
<field name="offset" type="NX_FLOAT" deprecated="Use FIELDNAME_offset instead">
<doc>
The offset and FIELDNAME_offset fields have similar semantics.
However, offset is ambiguous in the case of multiple signals. Therefore
offset is deprecated. Use FIELDNAME_offset instead, even when
only a single signal is present.
</doc>
</field>


<!-- Other fields -->
<field name="title">
<doc>
Expand Down
31 changes: 18 additions & 13 deletions manual/source/datarules.rst
Original file line number Diff line number Diff line change
Expand Up @@ -231,21 +231,26 @@ the following table lists the suffixes reserved by NeXus.
reserved suffixes; mask
reserved suffixes; set
reserved suffixes; weights

================== ========================================= =================================
suffix reference meaning
================== ========================================= =================================
``_end`` :ref:`NXtransformations` end points of the motions that start with ``DATASET``
``_errors`` :ref:`NXdata` uncertainties (a.k.a., errors)
``_increment_set`` :ref:`NXtransformations` intended average range through which the corresponding axis moves during the exposure of a frame
``_indices`` :ref:`NXdata` Integer array that defines the indices of the signal field which need to be used in the ``DATASET`` in order to reference the corresponding axis value
``_mask`` .. Field containing a signal mask, where 0 means the pixel is not masked. If required, bit masks are defined in :ref:`NXdetector` ``pixel_mask``.
``_set`` :ref:`target values <target_value>` Target value of ``DATASET``
``_weights`` .. divide ``DATASET`` by these weights [#]_
================== ========================================= =================================
reserved suffixes; scaling_factor
reserved suffixes; offset

=================== ========================================= =================================
suffix reference meaning
=================== ========================================= =================================
``_end`` :ref:`NXtransformations` end points of the motions that start with ``DATASET``
``_errors`` :ref:`NXdata` uncertainties (a.k.a., errors)
``_increment_set`` :ref:`NXtransformations` intended average range through which the corresponding axis moves during the exposure of a frame
``_indices`` :ref:`NXdata` Integer array that defines the indices of the signal field which need to be used in the ``DATASET`` in order to reference the corresponding axis value
``_mask`` .. Field containing a signal mask, where 0 means the pixel is not masked. If required, bit masks are defined in :ref:`NXdetector` ``pixel_mask``.
``_set`` :ref:`target values <target_value>` Target value of ``DATASET``
``_weights`` .. divide ``DATASET`` by these weights [#]_
``_scaling_factor`` :ref:`NXdata` Multiply ``DATASET`` by this factor [#]_
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
``_offset`` :ref:`NXdata` Add this factor to ``DATASET`` [#]_
=================== ========================================= =================================

.. [#] If ``DATASET_weights`` exists and has the same shape as the field,
you are supposed to divide ``DATASET`` by the weights.
you are supposed to divide ``DATASET`` by the weights. Similarly, if `DATASET_scaling_factor` and/or
`DATASET_offset` exist, apply this equation: (``DATASET`` + ``DATASET_offset``) * ``DATASET_scaling_factor``

.. Note that the following line might be added to the above table pending discussion:

Expand Down
Loading