Tool to produce a time calibration HDF5 file #533

Bultako · 2020-10-05T15:57:27Z

This PR adds a new Tool lstchain_create_time_calibration_file

lstchain_create_time_calibration_file --help-all
Generate a HDF5 file with time calibration coefficients

Options
-------

Arguments that take values are actually convenience aliases to full
Configurables, whose aliases are listed on the help line. For more information
on full configurables, see '--help-all'.

--input=<Path> (TimeCalibrationHDF5Writer.input)
    Default: None
    Path to the fits.fz events file or directory to glob
--glob=<Unicode> (TimeCalibrationHDF5Writer.glob)
    Default: '*'
    Filename pattern to glob files in the directory
--output=<Unicode> (TimeCalibrationHDF5Writer.output)
    Default: ''
    Path to the hdf5 time calibration file
--pedestal=<Unicode> (LSTR0Corrections.pedestal_path)
    Default: ''
    Path to the LST pedestal binary file
--max_events=<Int> (TimeCalibrationHDF5Writer.max_events)
    Default: 20000
    Maximum numbers of events to read. Default = 20000
--log-level=<Enum> (Application.log_level)
    Default: 30
    Choices: (0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL')
    Set the log level by value or name.
--config=<Path> (Tool.config_file)
    Default: None
    name of a configuration file with parameters to load in addition to command-
    line parameters

Class parameters
----------------

Parameters are set from command-line arguments of the form:
`--Class.trait=value`. This line is evaluated in Python, so simple expressions
are allowed, e.g.:: `--C.a='range(3)'` For setting C.a=[0,1,2].

TimeCalibrationHDF5Writer options
---------------------------------
--TimeCalibrationHDF5Writer.config_file=<Path>
    Default: None
    name of a configuration file with parameters to load in addition to command-
    line parameters
--TimeCalibrationHDF5Writer.glob=<Unicode>
    Default: '*'
    Filename pattern to glob files in the directory
--TimeCalibrationHDF5Writer.input=<Path>
    Default: None
    Path to the fits.fz events file or directory to glob
--TimeCalibrationHDF5Writer.log_datefmt=<Unicode>
    Default: '%Y-%m-%d %H:%M:%S'
    The date format used by logging formatters for %(asctime)s
--TimeCalibrationHDF5Writer.log_format=<Unicode>
    Default: '%(levelname)s [%(name)s] (%(module)s/%(funcName)s): %(messag...
    The Logging format template
--TimeCalibrationHDF5Writer.log_level=<Enum>
    Default: 30
    Choices: (0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL')
    Set the log level by value or name.
--TimeCalibrationHDF5Writer.max_events=<Int>
    Default: 20000
    Maximum numbers of events to read. Default = 20000
--TimeCalibrationHDF5Writer.output=<Unicode>
    Default: ''
    Path to the hdf5 time calibration file
--TimeCalibrationHDF5Writer.pedestal=<Unicode>
    Default: ''
    Path to drs4 fits pedestal file
--TimeCalibrationHDF5Writer.progress_bar=<Bool>
    Default: True
    show progress bar during processing

LSTEventSource options
----------------------
--LSTEventSource.allowed_tels=<Set>
    Default: set()
    list of allowed tel_ids, others will be ignored. If None, all telescopes in
    the input stream will be included
--LSTEventSource.baseline=<Int>
    Default: 400
    r0 waveform baseline (default from EvB v3)
--LSTEventSource.input_url=<Path>
    Default: None
    Path to the input file containing events.
--LSTEventSource.max_events=<Int>
    Default: None
    Maximum number of events that will be read from the file
--LSTEventSource.multi_streams=<Bool>
    Default: True
    Read in parallel all streams
--LSTEventSource.n_gains=<Int>
    Default: 2
    Number of gains at r0/r1 level

LSTR0Corrections options
------------------------
--LSTR0Corrections.offset=<Int>
    Default: 400
    Define the offset of the baseline
--LSTR0Corrections.pedestal_path=<Unicode>
    Default: ''
    Path to the LST pedestal binary file
--LSTR0Corrections.r1_sample_end=<Int>
    Default: 39
    End sample for r1 waveform
--LSTR0Corrections.r1_sample_start=<Int>
    Default: 3
    Start sample for r1 waveform
--LSTR0Corrections.tel_id=<Int>
    Default: 1
    id of the telescope to calibrate

TimeCorrectionCalculate options
-------------------------------
--TimeCorrectionCalculate.calib_file_path=<Unicode>
    Default: ''
    Path to the time calibration file
--TimeCorrectionCalculate.charge_product=<Unicode>
    Default: 'LocalPeakWindowSum'
    Name of the charge extractor to be used
--TimeCorrectionCalculate.minimum_charge=<Float>
    Default: 200
    Cut on charge. Default 200 ADC
--TimeCorrectionCalculate.n_capacitors=<Int>
    Default: 1024
    Number of capacitors (1024 or 4096). Default 1024.
--TimeCorrectionCalculate.n_combine=<Int>
    Default: 8
    How many capacitors are combines in a single bin. Default 8
--TimeCorrectionCalculate.n_harmonics=<Int>
    Default: 16
    Number of harmonic for Fourier series expansion. Default 16
--TimeCorrectionCalculate.tel_id=<Int>
    Default: 1
    Id of the telescope to calibrate

codecov · 2020-10-05T16:26:50Z

Codecov Report

Merging #533 (e64873c) into master (370ab58) will decrease coverage by 0.23%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #533      +/-   ##
==========================================
- Coverage   37.46%   37.23%   -0.24%     
==========================================
  Files          83       84       +1     
  Lines        7425     7471      +46     
==========================================
  Hits         2782     2782              
- Misses       4643     4689      +46

Impacted Files	Coverage Δ
...ain/tools/lstchain_create_time_calibration_file.py	`0.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 370ab58...e64873c. Read the comment docs.

lstchain/tools/lstchain_create_time_calibration_file.py

maxnoe · 2020-10-06T14:01:41Z

lstchain/tools/lstchain_create_time_calibration_file.py

+ help="Path to the input file containing events (wildcards allowed)",
+ ).tag(config=True)
+
+ output_file = traits.Unicode(


Paths should be ctapipe.core.traits.Path traitlets

maxnoe · 2020-10-06T14:02:59Z

lstchain/tools/lstchain_create_time_calibration_file.py

+ reader = LSTEventSource(input_url=self.path_list[0], max_events=self.max_events)
+ self.lst_r0 = LSTR0Corrections(
+ pedestal_path=self.pedestal_file,
+ config=self.config,


All sub-components of a tool need to be passed parent=self instead of config=self.config to work properly

maxnoe · 2020-10-06T14:03:47Z

lstchain/tools/lstchain_create_time_calibration_file.py

+
+ try:
+ for j, path in enumerate(self.path_list):
+ reader = LSTEventSource(input_url=self.path_list[0], max_events=self.max_events)


Shouldn't the eventsource get path as input? Like this you are only looking at the first file.

maxnoe

All path-like arguments should be changed from traits.Unicode to ctapipe.core.traits.Path.

All sub-components of this tool need to get the tool as parent, not the config.
So everywhere, exchange config=self.config with parent=self.
This is required for the config system to work!

To be able to store the used config in the proveance system, also self.add_component must be called.

So all subcomponents should be created like this:

class MyTool(Tool):
    ...
    def setup(self):
        self.subcomponent = self.add_component(SubComponent(parent=self))

Have a look at the ctapipe stage 1 tool here:

https://github.com/cta-observatory/ctapipe/blob/cf873ce99a42d6785cb9824dc297c8e843fc5a99/ctapipe/tools/stage1.py#L118

Bultako · 2020-11-24T20:21:17Z

I have updated this PR with requested changes (when possible) addressed and also new additions:

LSTR0Corrections and LSTEventSource subcomponents added as requested
output and pedestal arguments are needed as Unicode types by the TimeCorrectionCalculate and LSTR0Corrections classes respectively
input argument declared a Path
file name globbing feature added since it is also present in the script https://github.com/cta-observatory/cta-lstchain/blob/master/lstchain/scripts/lstchain_data_create_time_calibration_file.py#L34
progress bar added

maxnoe · 2020-11-24T20:25:19Z

output and pedestal arguments are needed as Unicode types by the TimeCorrectionCalculate and LSTR0Corrections classes respectively

If these are already traitlets of the corresponding classes, there is no need to repeat them.

Bultako · 2020-11-24T22:51:05Z

You're right.
I have removed the duplicated declaration of args.

rlopezcoto · 2021-02-23T13:30:06Z

Continued in #600

FrancaCassol reviewed Oct 6, 2020

View reviewed changes

lstchain/tools/lstchain_create_time_calibration_file.py Outdated Show resolved Hide resolved

FrancaCassol previously approved these changes Oct 6, 2020

View reviewed changes

maxnoe reviewed Oct 6, 2020

View reviewed changes

maxnoe requested changes Oct 6, 2020

View reviewed changes

Bultako dismissed FrancaCassol’s stale review via 889c3a6 November 24, 2020 20:06

Bultako added 14 commits February 23, 2021 12:50

add initial skeleton file

f5b4c68

add setup

516fdc6

add time calibration process in start

eaa8407

save file in finish

59852a6

minor fixes

243dfdd

add file globbing, pedestal_file and max_events

25518c8

set max_events default value to 20000

61817d6

add subcomponents

0f06319

use consistent arguments names

d267d20

add file globbing

2bc5e01

add progress bar

fdbadd5

remove try / except

d8e45ae

remove duplicated declaration of output and pedestal arguments

d26533b

add i and o aliases for input and output args

e64873c

Bultako mentioned this pull request Feb 23, 2021

Tool to produce a time calibration HDF5 file #600

Closed

rlopezcoto closed this Feb 23, 2021

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tool to produce a time calibration HDF5 file #533

Tool to produce a time calibration HDF5 file #533

Bultako commented Oct 5, 2020 •

edited

Loading

codecov bot commented Oct 5, 2020 •

edited

Loading

maxnoe Oct 6, 2020

maxnoe Oct 6, 2020

maxnoe Oct 6, 2020

maxnoe left a comment •

edited

Loading

Bultako commented Nov 24, 2020

maxnoe commented Nov 24, 2020

Bultako commented Nov 24, 2020

rlopezcoto commented Feb 23, 2021

Tool to produce a time calibration HDF5 file #533

Tool to produce a time calibration HDF5 file #533

Conversation

Bultako commented Oct 5, 2020 • edited Loading

codecov bot commented Oct 5, 2020 • edited Loading

Codecov Report

maxnoe Oct 6, 2020

Choose a reason for hiding this comment

maxnoe Oct 6, 2020

Choose a reason for hiding this comment

maxnoe Oct 6, 2020

Choose a reason for hiding this comment

maxnoe left a comment • edited Loading

Choose a reason for hiding this comment

Bultako commented Nov 24, 2020

maxnoe commented Nov 24, 2020

Bultako commented Nov 24, 2020

rlopezcoto commented Feb 23, 2021

Bultako commented Oct 5, 2020 •

edited

Loading

codecov bot commented Oct 5, 2020 •

edited

Loading

maxnoe left a comment •

edited

Loading