Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better rewrite of NXdata scaling_factor and offset fields #1343

Merged
merged 16 commits into from
Sep 5, 2024
Merged
40 changes: 40 additions & 0 deletions applications/NXmx.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,46 @@
<dim index="4" value="k" required="false"/>
</dimensions>
</field>

<field name="data_scaling_factor" type="NX_FLOAT" optional="true">
<doc>
An optional scaling factor to apply to the values in ``data``.

The elements in ``data`` are often stored as integers for efficiency reasons and need
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
further correction, generating floats. The two fields data_scaling_factor and
data_offset allow linear corrections using the following convention:

.. code-block::

corrected_data = (data + offset) * scaling_factor
paulmillar marked this conversation as resolved.
Show resolved Hide resolved

This formula will derive the corrected value, when necessary.

data_scaling_factor is sometimes known as gain or bias and data_offset is sometimes
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
known as pedestal or background, depending on the community.

Use these fields to specify constants that need to be applied
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
to the data to correct it to physical values. For example, if the detector gain
is 10 counts per photon and a constant background of 400 needs to be subtracted
off the pixels, specify data_scaling_factor as 0.1 and data_offset as -400 to
specify the required conversion from raw counts to corrected photons. It
is implied processing software will apply these corrections on-the-fly during processing.

The rank of these fields should either be a single value for the full dataset, a
single per-pixel array applied to every image (rank (i, j) or (i, j, k)),
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
a per-image correction specified with an array whose slowest rank is nP (rank
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
(np, 1), (np, i, j) or (np, i, j, k)).
</doc>
</field>
<field name="data_offset" type="NX_FLOAT" optional="true">
<doc>
An optional offset to apply to the values in data.

See :ref:`data_scaling_factor &lt;/NXmx/ENTRY/DATA/data_scaling_factor-field&gt;` for more information.
</doc>
</field>


</group>

<group type="NXsample">
Expand Down
42 changes: 34 additions & 8 deletions base_classes/NXdata.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -446,21 +446,47 @@
</field>

<!-- Data vs. plot coordinates -->
<field name="scaling_factor" type="NX_FLOAT">
<field name="FIELDNAME_scaling_factor" type="NX_FLOAT">
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
<doc>
The elements in data are usually float values really. For
efficiency reasons these are usually stored as integers
after scaling with a scale factor. This value is the scale
factor. It is required to get the actual physical value,
when necessary.
An optional scaling factor to apply to the values in FIELDNAME (usually the signal).

The elements stored in NXdata datasets are often stored as integers for efficiency
reasons and need further correction or conversion, generating floats. For example,
raw values could be stored from a device that need to be converted to values that
represent the physical values. The two fields FIELDNAME_scaling_factor and
FIELDNAME_offset allow linear corrections using the following convention:

.. code-block::

corrected values = (FIELDNAME + offset) * scaling_factor

This formula will derive the values to use in downstream applications, when necessary.
</doc>
</field>
<field name="offset" type="NX_FLOAT">
<field name="FIELDNAME_offset" type="NX_FLOAT">
<doc>
An optional offset to apply to the values in data.
An optional offset to apply to the values in FIELDNAME (usually the signal).

See :ref:`FIELDNAME_scaling_factor &lt;/NXdata/FIELDNAME_scaling_factor-field&gt;` for more information.
</doc>
</field>

<field name="scaling_factor" type="NX_FLOAT" deprecated="Use FIELDNAME_scaling_factor instead">
<doc>
scaling_factor is ambiguous in the case of multiple signals, so use
phyy-nx marked this conversation as resolved.
Show resolved Hide resolved
FIELDNAME_scaling_factor instead, which has similar semantics, even
when only a single signal is present.
</doc>
</field>
<field name="offset" type="NX_FLOAT" deprecated="Use FIELDNAME_offset instead">
<doc>
offset is ambiguous in the case of multiple signals, so use
FIELDNAME_offset instead, which has similar semantics, even
when only a single signal is present.
</doc>
</field>


<!-- Other fields -->
<field name="title">
<doc>
Expand Down
7 changes: 6 additions & 1 deletion manual/source/datarules.rst
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,8 @@ the following table lists the suffixes reserved by NeXus.
reserved suffixes; mask
reserved suffixes; set
reserved suffixes; weights
reserved suffixes; scaling_factor
reserved suffixes; offset

================== ========================================= =================================
suffix reference meaning
Expand All @@ -242,10 +244,13 @@ suffix reference meaning
``_mask`` .. Field containing a signal mask, where 0 means the pixel is not masked. If required, bit masks are defined in :ref:`NXdetector` ``pixel_mask``.
``_set`` :ref:`target values <target_value>` Target value of ``DATASET``
``_weights`` .. divide ``DATASET`` by these weights [#]_
``_scaling_factor`` :ref:`NXdata` Multiply ``DATASET`` by this factor [#]_
PeterC-DLS marked this conversation as resolved.
Show resolved Hide resolved
``_offset`` :ref:`NXdata` Add this factor to ``DATASET`` [#]_
================== ========================================= =================================

.. [#] If ``DATASET_weights`` exists and has the same shape as the field,
you are supposed to divide ``DATASET`` by the weights.
you are supposed to divide ``DATASET`` by the weights. Similarly, if `DATASET_scaling_factor` and/or
`DATASET_offset` exist, apply this equation: (``DATASET`` + ``DATASET_offset``) * ``DATASET_scaling_factor``

.. Note that the following line might be added to the above table pending discussion:

Expand Down
Loading