Cubeviz spectral extraction through plugin

[kecnry]

Description

This pull request replaces the glue automatic cube collapse with the spectral extraction plugin and the auto-updating plugin results functionality introduced in #2680. This is being done both for internal self-consistency but also to properly handle different resulting units depending on the function (which can then be sent through #2781 to "translate" between surface brightness and flux, as needed).

Updated cubeviz export_data docs

auto/updating-extraction

Extracted spectrum is now a separate data entry from the flux cube. Creating a spatial subset automatically runs through the spectral extraction plugin, resulting in an auto-updating data product.

Screen.Recording.2024-05-02.at.2.28.39.PM.mov

gaussian smooth

If spectral smooth is selected, then the output shape depends on the input (cubes result in cubes, with NO automatic extraction, spectra result in spectra). Spatial smoothing always results in a cube (with NO automatic extraction).

Screen.Recording.2024-05-02.at.2.27.31.PM.mov

model fitting

Cube fitting option is moved to the top and changes the available options for input data. Spatial subset selection is gone in place of choosing the auto-extracted spectrum. Model parameter values for cubes are estimated based on an on-the-fly extraction using mean.

Screen.Recording.2024-05-02.at.2.31.57.PM.mov

line analysis

Spatial subset selection is gone in place of choosing the auto-extracted spectrum.

Screen.Recording.2024-05-02.at.2.32.48.PM.mov

moment maps

Continuum subtraction now allows to choose which existing 1d spectrum should be used to visualize the continuum inputs. Note that the continuum is ultimately computed with respect to the full input cube.

Screen.Recording.2024-05-02.at.2.35.11.PM.mov

This PR accomplishes this by doing the following:

• bumps the pin of astropy to 5.3.2, since this plugin is now required for core-functionality of cubeviz
• no longer loads the cube into the spectrum-viewer during load.
• removes cubes from the data menu for the spectrum-viewer. All spectra are now separate 1D spectrum entities rather than representations of the cube and must be sent through the spectral extraction plugin.
• runs the entire cube through the plugin (with default options) during load, similar to specviz2d, resulting in a new data entry/layer labeled "Spectrum (sum)" (label up for debate, using new plugin default label logic)
• adds the collapse function to the default proposed label for extraction entries
• creates auto-updating entries sent through the plugin (with default options) when creating a spatial subset, labeled "Spectrum (Subset X, sum)" (label up for debate, using new plugin default label logic)
• collapsed spectrum adopts the initial color based on the input subset
• no longer shows spatial subsets in spectrum-viewer (legend and plot options)
• removes collapse function from plot options. The function will now be per-spectrum and controlled in the spectral extraction plugin. The viewer can now contain spectra of mixed collapse functions, but changing the function requires re-extraction and can no longer be changed on-the-fly.
• removes line styling options for spatial subsets in plot options
• removes spatial subset options from all plugins that take a 1d spectrum as input and updates the dataset entries to require 1d spectra directly (instead of cubes)
  • spectral extraction: generalized to work on any flux cube, by generalizing the is_flux_cube filter (so that model fitting can make use of collapsing functionality for smoothed cubes, etc)
  • gaussian smooth: allow both cubes and 1d spectra as input. If cube, allow selection spatial or spectral, resulting in a cube output, but no automatic extraction (could be added in the future). If spectrum, hide "mode" dropdown (only allow spectral). Add to viewer selection should automatically update based on the input dataset.
  • model fitting: move "fit to cube" toggle to top and change the filter on the datasets depending on the selection. Remove "cube_fit" from both the default proposed output label and the results table (since will now be obvious based on the input label).
  • model fitting: parameter estimates for model components now go through spectral extraction logic (entire cube, mean, all other defaults) on-the-fly.
  • model fitting: allow any flux cube as input (but exclude non-flux cubes). Previously you could fit (or at least attempt to fit) any cube in the flux viewer (whether it was a flux cube or not).
  • line analysis: swap "spatial region" out with a data dropdown (1d spectra only)
  • moment maps: new continuum_dataset dropdown to select which 1d spectrum to use for determining the continuum (for visualization only). Note this is only shown when more than one option exist - so we could add additional logic to filter the choices to match the input cube and with no spatial subsets as input, in which case multiple entries would only show if there are multiple manual extractions which might then be useful for the user to switch between. Or we could hide this entirely from the user and just take the first match (effectively the same as previous behavior).
• removes spatial-spectral shadowing logic
• moves plugin higher up in tray
• removes "degrees" checks that used to exist when first loading the cube into the spectrum-viewer
• deprecates REMOVES SUPPORT for function and spatial_subset kwargs to get_data to get a spectrum from a cube. spatial_subset can still be passed in cubeviz/imviz to get a masked cube/image, but spectral extraction must now go through the dedicated plugin. This might qualify for a major version bump (4.0 instead of 3.11 milestone)
• delete auto-updated entry when subset is deleted (behavior may need to change in the future if/when we allow user-created auto-updating products)
• updates tests (50 29 18 17 14 40 35 17 14 13 11 6 3 0 broken 😭)
• update notebooks
• update docs
• stretch/follow-up: shows indicator in legend/data-menu if live-update becomes out of date
  • alternatively, should failed auto-update should delete entry

TODO:

• fix (again) the initial slice location
• fix (likely cause of buggy behavior reported by @javerbukh): jdaviz/jdaviz/core/helpers.py:586: UserWarning: Not able to get Spectrum (sum) returned with subset Subset 1 applied of type <class 'specutils.spectra.spectrum1d.Spectrum1D'>.
• ensure no deprecated access to get_data is being used internally (currently resulting in deprecation logger messages)
• ensure cubeviz.specviz.get_spectra() works with spatial and/or spectral subsets added
• follow-up ticket to refactor filters on layer/dataset component
• follow-up ticket to fix scroll parentElement warnings in dev console
• follow-up ticket to move cached properties in cubeviz viewers to mixins
• include 2680 in changelog entry?

Change log entry

• Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
  list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
  should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• Do the proposed changes follow the STScI Style Guides?
• Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• Did the CI pass? If not, are the failures related?
• Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
• After merge, any internal documentations need updating (e.g., JIRA, Innerspace)?

[pllim]

I got a traceback from this branch. Workflow to reproduce:

1. Run Cubeviz example notebook to the end.
2. Scroll back up and use the UI trashcan to delete Subset 4. OK here.
3. Now use the same UI to delete Subset 3.

I get a warning saying that spectrum tied to Subset 3 is removed, but I still see Subset 3 on image viewers and Subsets drop-down. When I open the log in Lab, I see this traceback:

File ipyvue/VueTemplateWidget.py:60, in Events._handle_event(self, _, content, buffers)
     58     getattr(self, "vue_" + event)(data, buffers)
     59 else:
---> 60     getattr(self, "vue_" + event)(data)

File glue_jupyter/widgets/subset_select_vuetify.py:107, in SubsetSelect.vue_remove_subset(self, index)
    106 def vue_remove_subset(self, index):
--> 107     self.data_collection.remove_subset_group(self.data_collection.subset_groups[index])

File glue/core/data_collection.py:288, in DataCollection.remove_subset_group(self, subset_grp)
    286 self._subset_groups.remove(subset_grp)
    287 for s in subset_grp.subsets:
--> 288     s.delete()
    289 subset_grp.unregister(self.hub)

File glue/core/subset.py:244, in Subset.delete(self)
    242 if dobroad:
    243     msg = SubsetDeleteMessage(self)
--> 244     self.data.hub.broadcast(msg)
    246 Registry().unregister(self, group=self.data)

File glue/core/hub.py:215, in Hub.broadcast(self, message)
    213 logging.getLogger(__name__).info("Broadcasting %s", message)
    214 for subscriber, handler in self._find_handlers(message):
--> 215     handler(message)

File jdaviz/app.py:454, in Application._on_subset_delete_message(self, msg)
    453 def _on_subset_delete_message(self, msg):
--> 454     self._remove_live_plugin_results(trigger_subset=msg.subset)
    455     self._on_layers_changed(msg)

File jdaviz/app.py:441, in Application._remove_live_plugin_results(self, trigger_data_lbl, trigger_subset)
    437 for data, plugin_inputs in self._iter_live_plugin_results(trigger_data_lbl, trigger_subset):
    438     self.hub.broadcast(SnackbarMessage(
    439         f"Removing {data.label} due to deletion of {trigger_subset.label if trigger_subset is not None else trigger_data_lbl}",  # noqa
    440         sender=self, color="warning"))
--> 441     self.vue_data_item_remove({'item_name': data.label})

File jdaviz/app.py:2213, in Application.vue_data_item_remove(self, event)
   2211     self._reparent_subsets(data, new_parent=orient)
   2212 else:
-> 2213     self._reparent_subsets(data)
   2215 # Make sure the data isn't loaded in any viewers and isn't the selected orientation
   2216 for viewer_id, viewer in self._viewer_store.items():

File jdaviz/app.py:2011, in Application._reparent_subsets(self, old_parent, new_parent)
   2009 for subset in subset_group.subsets:
   2010     subset_message = SubsetUpdateMessage(sender=subset)
-> 2011     self.hub.broadcast(subset_message)

File glue/core/hub.py:215, in Hub.broadcast(self, message)
    213 logging.getLogger(__name__).info("Broadcasting %s", message)
    214 for subscriber, handler in self._find_handlers(message):
--> 215     handler(message)

File jdaviz/configs/default/plugins/subset_plugin/subset_plugin.py:129, in SubsetPlugin._on_subset_update(self, *args)
    127 if 'Create New' in self.subset_selected:
    128     return
--> 129 self._get_subset_definition(*args)
    130 subset_to_update = self.session.edit_subset_mode.edit_subset[0]
    131 self.subset_select._update_subset(subset_to_update, attribute="type")

File jdaviz/configs/default/plugins/subset_plugin/subset_plugin.py:322, in SubsetPlugin._get_subset_definition(self, *args)
    319 self.glue_state_types = []
    320 self.subset_states = []
--> 322 self._unpack_get_subsets_for_ui()

File jdaviz/configs/default/plugins/subset_plugin/subset_plugin.py:168, in SubsetPlugin._unpack_get_subsets_for_ui(self)
    165     return
    167 include_sky_region = bool(self.display_sky_coordinates)
--> 168 subset_information = self.app.get_subsets(self.subset_selected,
    169                                           simplify_spectral=False,
    170                                           use_display_units=True,
    171                                           include_sky_region=include_sky_region)
    173 _around_decimals = 6  # Avoid 30 degrees from coming back as 29.999999999999996
    174 if not subset_information:

File jdaviz/app.py:1019, in Application.get_subsets(self, subset_name, spectral_only, spatial_only, object_only, simplify_spectral, use_display_units, include_sky_region)
   1017     return all_subsets[subset_name]
   1018 elif subset_name:
-> 1019     raise ValueError(f"{subset_name} not in {all_subset_names}")
   1020 else:
   1021     return all_subsets

ValueError: Subset 3 not in ['Subset 1', 'Subset 2']

[javerbukh]

@pllim The version of the example notebook I'm using does not create 4 subsets. Do you have the latest version?

[pllim]

@javerbukh , run all the cells. The second half of the notebook loads regions into the app from API.

[pllim]

Specifically this one:

# photutils aperture
my_aper = CircularAperture((15, 10), r=5)

# regions shape
my_reg = CirclePixelRegion(center=PixCoord(x=35, y=35), radius=10)

my_regions = [my_aper, my_reg]
cubeviz.load_regions(my_regions)

[javerbukh]

I only get 2 subsets when running all the cells.

[kecnry]

I can reproduce with just 2 manually created spatial subsets (but oddly not with a single spatial subset, which was previously my test case, and also oddly seems to be isolated to the last added subset). I'll look into and try to fix it - hopefully shouldn't hold up continuing reviews in the meantime! 🤞

[pllim]

I only get 2 subsets when running all the cells.

Oh oops... sorry, I forgot I also drawn 2 with my hands before that.

[javerbukh]

I found another subset related bug (so sorry) and confirmed it is not on main:

1. Create a spectral subset
2. Unit convert to Angstrom
3. Go to subset tools and change high or low value of subset
4. Update

I see a UnitConversionError: 'Angstrom' (length) and '' (dimensionless) are not convertible traceback originating from:

File jdaviz/configs/default/plugins/subset_plugin/subset_plugin.py#line=409, in SubsetPlugin.vue_update_subset(self, *args)
    408         if self.spectral_display_unit != base_units:
    409             d_val = d_val*u.Unit(self.spectral_display_unit)
--> 410             d_val = d_val.to(u.Unit(base_units))
    411             d_val = d_val.value
    413 if float(d_att["orig"]) != d_val:

[javerbukh]

Looks like the previously mentioned bug is only in cubeviz and not specviz. I also confirmed that this PR has the same behavior in other configs as on main, although I did notice that subsets are apparently not working in mosviz. I'll check if there is already a ticket for that and if not, create one.

[kecnry]

The "re-parenting" traceback when deleting the last-created (but not first) subset, should now be fixed.

[kecnry]

I see a UnitConversionError: 'Angstrom' (length) and '' (dimensionless) are not convertible traceback originating from

This one should also now be fixed.

[javerbukh]

Can confirm that the bug I was seeing is fixed, thanks!

[pllim]

Let's get it in then before Wednesday. Please don't forget the change log. Thanks!

[kecnry]

This is now combined with #2781 and I tried to add changelog entries that would cover the breaking changes without getting too verbose. If anyone has comments on that, let me know, otherwise I can merge once CI is green.

[pllim]

LGTM.

Is there any follow-up needed for all the JDAT notebooks out there?

[kecnry]

I'm not sure, I did go through and update the example notebooks. I'll create a general ticket for updating JDAT for any/all breaking changes in 4.0 - that's a good idea.

[pllim]