Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP: It is the first very very draft on refactoring the whole lstchain code to
DataWriter
)The first idea is to implement a
LSTProcessor
-Tool analogous to ctapipe's Processor Tool with LST-specificComponent
s inheriting as much as possible from ctapipe classes. The aim here would be to mainly substitute ther0_to_dl1
,dl1_ab
anddl1_to_dl2
scripts to have just one tool that you can feed with different configs regarding your desired analysis step. The new analysis flow could look like:1) R0 to DL1
Processing R0 data up to dl1 with Cat-A calibrations all defined in a base config.yaml. It can optionally write out muons and interleaved events (maybe also allow to only write out the interleaved pedestal events directly for Cat-B calibrations)
2) Reprocess DL1 (DL1ab)
Not only for reprocessing dl1 data (e.g. with different cleaning settings...), but also for applying Cat-B calibrations and including pedestal cleaning. All defined in an additional config_dl1ab.yaml.
3) DL1 to DL2
Process DL1 to DL2 with a specific (src (in)dependet) config_dl2.yaml.
Or directly from R0 to DL2
To process directly from R0 to DL2 (with Cat-B calibs) you can first write out the interleaved events with the 'interleaved-only'-mode and compute the cat-B calibrations with the separate tool. After that you hand in your config, the cat-B calibrations and RF-Models (maybe trained by ctapipe's ML module -
ctapipe-train-...
) to the processor tool and process directly to DL2.Tasks
The tool would basically look like ctapipe's
Processor
tool:with different
Component
s passing and filling theArrayEventContainer
. That includes:LSTInterleavedProcessor
component which process the interleaved pedestal and flatfield events with the CalibrationCalculator. There should also be the possibility of writing only the r1 waveforms to an extra .h5 file to calculate the cat-B coefficients. For that it would be also nice to add a flag in ctapipe_io_lst for the gain selection of calibration events, so we don't have to calibrate at this level.WaveformNSBTuner
component to calculate (based on a target file) and apply NSB on waveform-level to MC-data.LSTCameraCalibrator
component which inherits from ctapipe's CameraCalibrator. It needs to be adapted for applying the cat-B calibrations consisting of either filling the DL1CameraCalibrationContainer and applying them in the ._calibrate_dl1 step or applying them afterwards on peak times and charges if no waveforms are available (for 'dl1ab').LSTImageProcessor
component inheriting from ctapipe's ImageProcessor which uses its own LSTImageCleaner. It needs to be adapted regarding the API of the ImageCleaner to allow pedestal cleaning.LSTMuonProcessor
component inheriting from ctapipe's MuonProcessor adding an own CameraCalibrator instance which can use a different extractor (e.g. GlobalPeakWindowSum) for the muon analysis. It also creates its on DataWriter instance for writing promising muon events to an extra .h5 file.LHFitProcessor
component which computes lhfit parameters, fills its own container an write it to the output file.ShowerProcessor
component. I did not yet dive deep enough into ctapipe's ML module and lstchains dl1_to_dl2 step to know exactly what we can use/inherit from. WIP.A first basic config can be found in lstchain/data/lstchain_base_config.yaml
For contributing please open new PR's to this branch and reference them in the associated tasks above to not overload this PR for reviewers. I would suggest at least one PR per component
Additional tasks:
I'm looking forward to all your ideas and comments. Feel free to edit/adapt the tasks.