Implement the Correlator special suite #328

JamesWrigley · 2021-12-10T18:58:35Z

Finally got around to making a PR out of this 😅 I'm keeping it as a draft for now because there are a couple of (hopefully) minor issues still to fix:

f581c87 needs to be refactored, you can ignore this commit for now.
This uses the high_high_water_mark branch of metropc, which in turn is based on feat/bound_device_features. The high_high_water_mark branch just increases the HWM for the output socket of metropc, because I noticed that sometimes metropc was just dropping outputs and that fixed the issue. It could also be because the special suite is not reading fast enough, but that needs to be investigated. And perhaps we should wait to merge this until the feat/bound_device_features branch is merged too.

To reviewers, even though this is a draft I would suggest starting now because there's... a lot... of code. There's also some auxiliary commits that were needed for the Correlator, so I would suggest reviewing each commit individually because they're fairly atomic (it'll make more sense that way).

~~Note: this PR depends on #326, so you should ignore commits 1581c1b and 479e93a.~~

lgtm-com · 2021-12-10T19:36:05Z

This pull request introduces 1 alert when merging 5dbc0f1 into 2331f1d - view on LGTM.com

new alerts:

1 for Inefficient regular expression

lgtm-com · 2021-12-20T10:50:57Z

This pull request introduces 1 alert when merging e5be471 into 2331f1d - view on LGTM.com

new alerts:

1 for Inefficient regular expression

JamesWrigley · 2021-12-20T10:58:56Z

Hmm, it's failing because the CI can't install metropc. Currently it's using an SSH URL but the repo is private anyway so using HTTPS won't help. @philsmt, is there a timeline to make metropc public?

lgtm-com · 2021-12-20T17:14:49Z

This pull request introduces 1 alert when merging 4288618 into f8d225d - view on LGTM.com

new alerts:

1 for Inefficient regular expression

philsmt · 2022-01-10T13:04:22Z

extra_foam/utils.py

+                               metadata for plotting.
+    """
+    xr_attrs = {
+        "xlabel": xlabel,


When receiving data as an xarray.DataArray (as you create here), there is already a documented protocol in metropc to use the coordinate label(s) as X axis label in 1D, whereas the y label is passed as an attribute (like here).

Would you mind using this instead of an xlabel attribute for compatibility?

Yeah sure, that should be fine.

lgtm-com · 2022-02-03T18:13:21Z

This pull request introduces 1 alert when merging c27cd0c into 1489977 - view on LGTM.com

new alerts:

1 for Inefficient regular expression

lgtm-com · 2022-02-16T11:44:18Z

This pull request introduces 1 alert when merging 46086ae into 86177ea - view on LGTM.com

new alerts:

1 for Inefficient regular expression

lgtm-com · 2022-02-17T15:29:27Z

This pull request introduces 2 alerts when merging e685bbe into 621f1e5 - view on LGTM.com

new alerts:

1 for Unused import
1 for Inefficient regular expression

lgtm-com · 2022-03-04T10:51:29Z

This pull request introduces 2 alerts when merging 2759386 into 7b704c3 - view on LGTM.com

new alerts:

1 for Unused import
1 for Inefficient regular expression

lgtm-com · 2022-03-16T19:57:07Z

This pull request introduces 4 alerts when merging 4b8f0e0 into 56f56f5 - view on LGTM.com

new alerts:

2 for Unused import
1 for Unused local variable
1 for Inefficient regular expression

This is to match what the DataTransformer does.

This makes it easier to change the settings from within special suites, without having to dig into _SpecialAnalysisBase code.

lgtm-com · 2022-05-06T16:37:33Z

This pull request introduces 4 alerts when merging 3a135f8 into b9e30b5 - view on LGTM.com

new alerts:

2 for Unused import
1 for Unused local variable
1 for Inefficient regular expression

This is a very simple implementation of something that can hopefully be improved and cleaned up in the future: support for context-defined annotation widgets. Each ROI is defined as a parameter, they are added to a View with an argument to the Image_WithRoi type, which is read by the Correlator and thus creates an ROI. When the ROI is changed, an updated parameter is sent to metropc and the Correlator modifies the context file with the new ROI parameters (position, height, etc).

Also added a warning dialog that appears when the user tries to move a ROI when there's a syntax error in the code (which libcst will fail to parse).

Among other things, this makes it possible to downsample all series being plotted.

This is to allow LinearROIs on images.

This is in preparation for using another library that's most up-to-date on conda.

- Replace np.warnings with the warnings module - Replace np.float with float - Pass enum values instead of the enums themselves to redis functions - Drop support for saving float arrays PNG images and expecting them to be restored with the same values. Seems that Pillow dropped support for it.

Lots of minor fixes to the tests, and fixed one bug in the correlator that caused an extra nan value to be added to the error arrays for each series.

philsmt · 2023-07-28T06:58:37Z

We should probably aim to point this at metropc's main branch 🤔

JamesWrigley · 2023-07-28T07:29:24Z

I think you pressed the wrong button there 😄

philsmt · 2023-07-28T07:32:37Z

I did, apologies 😳

JamesWrigley requested review from tmichela, philsmt and CammilleCC December 10, 2021 18:58

JamesWrigley self-assigned this Dec 10, 2021

JamesWrigley force-pushed the correlator branch from 32d2fc5 to e5be471 Compare December 20, 2021 10:12

JamesWrigley force-pushed the correlator branch from e5be471 to 4288618 Compare December 20, 2021 16:36

philsmt reviewed Jan 10, 2022

View reviewed changes

JamesWrigley force-pushed the correlator branch from 4288618 to c27cd0c Compare February 3, 2022 17:35

JamesWrigley force-pushed the correlator branch from c27cd0c to 46086ae Compare February 16, 2022 11:03

JamesWrigley force-pushed the correlator branch from 46086ae to e685bbe Compare February 17, 2022 14:45

JamesWrigley force-pushed the correlator branch from e685bbe to 2759386 Compare March 4, 2022 10:12

JamesWrigley force-pushed the correlator branch from 2759386 to 4b8f0e0 Compare March 16, 2022 18:03

JamesWrigley force-pushed the correlator branch from 4b8f0e0 to 8485c45 Compare April 6, 2022 18:25

JamesWrigley added 4 commits May 6, 2022 17:55

Remove the 'category' abstraction from the special suites

f0cfea4

Implement SimpleSequence.resize()

91516f7

Change data_with_assembled() to save the train ID as "train_id"

cf9405a

This is to match what the DataTransformer does.

Add the hostname/port/client type to the special suite interface

03eaa9e

This makes it easier to change the settings from within special suites, without having to dig into _SpecialAnalysisBase code.

JamesWrigley force-pushed the correlator branch from 8485c45 to 3a135f8 Compare May 6, 2022 15:55

JamesWrigley added 2 commits July 20, 2022 18:00

Implement the Correlator special suite

653db4c

Add tests for the Correlator

b463e3d

JamesWrigley added 20 commits July 27, 2023 14:14

Add support for labels to RectROI

deba71c

Add margin symbols for context errors

ab0d14b

Also added a warning dialog that appears when the user tries to move a ROI when there's a syntax error in the code (which libcst will fail to parse).

WIP support for downsampling large vectors

c41b08a

Add support for images with dynamic aspect ratios

ac7f525

Show image axes by default

f68e4f2

WIP implementation of binning for motor scans

e0ec1fb

Implement curve fitting for individual series

148efb7

Add an ElidedLabel class to display long context paths more neatly

388bc94

Allow having separate X values for each series

fbfd927

Among other things, this makes it possible to downsample all series being plotted.

Add support for specifying the beam width of error bars

3ffbe6e

Make the ROI creation code more flexible

0a82b1c

This is to allow LinearROIs on images.

Allow setting ROIs on AspectUnlocked views

534db2f

Add a utility method for getting slice objects from LinearROI

74312f5

Add support for action views in the interface

d4d6ebb

Implement a get_icon_path() helper

b0507aa

Add live validation to the correlator editor

d5ac51f

Install PyQt libraries through conda

f76d605

This is in preparation for using another library that's most up-to-date on conda.

Update correlator tests

46ff072

Lots of minor fixes to the tests, and fixed one bug in the correlator that caused an extra nan value to be added to the error arrays for each series.

JamesWrigley force-pushed the correlator branch from ab9acf4 to 46ff072 Compare July 27, 2023 17:07

philsmt closed this Jul 28, 2023

JamesWrigley reopened this Jul 28, 2023

JamesWrigley and others added 4 commits July 28, 2023 12:14

fixup! Add live validation to the correlator editor

b382e29

Merge branch 'dev' into correlator

752a459

wget doesn't like downloading to path containing ..

7ef53f6

Update metropc to master branch

e497e47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement the Correlator special suite #328

Implement the Correlator special suite #328

JamesWrigley commented Dec 10, 2021 •

edited

Loading

lgtm-com bot commented Dec 10, 2021

lgtm-com bot commented Dec 20, 2021

JamesWrigley commented Dec 20, 2021

lgtm-com bot commented Dec 20, 2021

philsmt Jan 10, 2022

JamesWrigley Jan 11, 2022

lgtm-com bot commented Feb 3, 2022

lgtm-com bot commented Feb 16, 2022

lgtm-com bot commented Feb 17, 2022

lgtm-com bot commented Mar 4, 2022

lgtm-com bot commented Mar 16, 2022

lgtm-com bot commented May 6, 2022

philsmt commented Jul 28, 2023

JamesWrigley commented Jul 28, 2023

philsmt commented Jul 28, 2023

Implement the Correlator special suite #328

Are you sure you want to change the base?

Implement the Correlator special suite #328

Conversation

JamesWrigley commented Dec 10, 2021 • edited Loading

lgtm-com bot commented Dec 10, 2021

lgtm-com bot commented Dec 20, 2021

JamesWrigley commented Dec 20, 2021

lgtm-com bot commented Dec 20, 2021

philsmt Jan 10, 2022

Choose a reason for hiding this comment

JamesWrigley Jan 11, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Feb 3, 2022

lgtm-com bot commented Feb 16, 2022

lgtm-com bot commented Feb 17, 2022

lgtm-com bot commented Mar 4, 2022

lgtm-com bot commented Mar 16, 2022

lgtm-com bot commented May 6, 2022

philsmt commented Jul 28, 2023

JamesWrigley commented Jul 28, 2023

philsmt commented Jul 28, 2023

JamesWrigley commented Dec 10, 2021 •

edited

Loading