A time clustering algorithm for image cleaning

[clara-escanuela]

No description provided.

[maxnoe]

slightly shorter: labels = DBSCAN(...).fit_predict(X).

[maxnoe]

Shouldn't this also use sample_weights=image? Otherwise you are using image only for the initial cut

[clara-escanuela]

I tried sample_weight before. I used this weighting:
$\frac{1}{1+exp\left(-\frac{image+s}{d}\right)}$
It introduces two new parameters but I can add weights as an option

[maxnoe]

Why not simply weight with the image itself? I.e. cluster the photo electrons?

[clara-escanuela]

I have been thinking about this and I think we could introduce a fourth dimension in the cluster, which is 1/SNR, SNR is the signal-to-noise ratio, image/pedestal. Using just image does not work as the algorithm finds clusters with similar data points but if you plot 1/image, we find clusters:

[clara-escanuela]

It does not seem to make any difference though

[maxnoe]

Docs failure is fixed in main, please rebase

[kosack]

Not sure if it's interesting, but they just introduced an improved DBSCAN algorithm in scikit-learn

[kosack]

I think here you should really use the correct pedestal, not some hard-coded value. There is for example

ped = event.mon.tel[7].calibration.pedestal_per_sample

That is per channel, so you need to use the high/low gain switch mask. We probably should properly process simulated pedestals into DL1 information though, so you don't have to do this.

[clara-escanuela]

Maybe I should change the name but what I need is the variance of the reconstructed noise per pixel in units of pe. Is that available?

[kosack]

Yeah, i realized that just after posting... the pedestals themselves are not what you need. In the simulations currently we don't have the variances, but this is something we really need to fix. Either we should properly compute pedestals and not rely on the ones given in the sims (which are technically inputs, not outputs), or see if somewhere the variances are already included or ask for them to be included. That's not just or this PR, but for any "realistic" cleaning method, e.g. the Whipple-10m-style cleaning that is just tailcuts but with the thresholds expressed in pedestal variance units.

This should probably be a separate issue and PR. For now, what you can maybe do is just don't have these pedestal dispersions hard-coded in the class, but rather make them a TelescopeParameter that is configurable. That way we could adapt to multiple prods without having to edit the code. Once we have a "correct" way to read the pedestal dispersions for sims, we can then properly fix it.

[kosack]

You should definitely change the name, since "pedestal" is the mean value of the noise, not the dispersion. Here, you are assuming it's Gaussian, right (even though it's not)? And you are using it as a standard deviation, not variance, so maybe pedestal_std

[kosack]

it also occurs to me that this threshold could also just be a threshold in PE for now (like the tailcuts), which would work fine for simulations, where the pedestal variances don't vary across the FOV or in time. Then later we upgrade it to use pedestal variances once we have those properly implemented for simulations.

[clara-escanuela]

This is the basic implementation of the method. I think we could finish it soon and then if we need to add some more details, which is possible, we could do so. I am just wondering about unit tests

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (main@18832ea). Click here to learn what that means.

❗ Current head 1f94003 differs from pull request most recent head afc1704. Consider uploading reports for the commit afc1704 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2401   +/-   ##
=======================================
  Coverage        ?   92.48%           
=======================================
  Files           ?      234           
  Lines           ?    20005           
  Branches        ?        0           
=======================================
  Hits            ?    18502           
  Misses          ?     1503           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maxnoe]

Can we determine the scale parameters from the information in CameraDescription?

[clara-escanuela]

I am not sure. That parameter needs optimization, it could 2x or 3x the pixel spacing of the camera and teh optimized number depends on the telescope

[Tobychev]

@kosack @maxnoe This PR seems to have picked up quite a few unrelated changes, presumably from an attempt to rebase on main, what is the preferred way to deal with this?

[maxnoe]

I am not sure what happened here.

@clara-escanuela What git commands did you run?

I see issues like this one from time to time but never could figure out what mistake leads to these.

The proper way of updating a branch is (assuming upstream points to cta-observatory/ctapipe and origin points to clara-escanuela/ctapipe)

$ git fetch upstream # get latest changes from main repo
$ git rebase upstream/main
# fix any conflicts that might occure by editing and then running git add / git rebase --continue

I think could locally salvage this branch here by cherry-picking the commits that seem to be originally from @clara-escanuela on this branch after resetting the branch hard to upstream/main.

$ git reset --hard upstream/main
$ git cherry-pick 37e5ea864 55cf8bafb 7e8daf8d4 895ff9774 a9f81e99d 357d02b51 722dfcf17 3c799aaee b9ab65e67 afc170480