Test mx_bluesky 1.0.0 as hyperion

[DominicOram]

• Running dls_dev_env.sh from deployment results in This script should be run from the 'mx-bluesky' directory - fixed in Hotfixes found from testing on the beamline #477
• Issue on deploy of:

ERROR: while parsing the following warning configuration:

  ignore::sqlalchemy.exc.MovedIn20Warning

This error occurred:

Traceback (most recent call last):
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1917, in parse_warning_filter
    category: type[Warning] = _resolve_warning_category(category_)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1955, in _resolve_warning_category
    m = __import__(module, None, None, [klass])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'sqlalchemy'

Fixed by ignoring this error in pyproject.toml - already on main

• Failed to build confluent-kafka as before. Solution is to install this package using a RHEL8 machine but everything else using i03-control - known issue that will ultimately be fixed by containerizing
• Panda issue:

ERROR:root:device `coros` raised unexpected exception RuntimeError
Traceback (most recent call last):
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/fastcs/panda/_hdf_panda.py", line 41, in connect
    await fill_pvi_entries(self, self._prefix + "PVI", timeout=timeout, mock=mock)
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/_pvi.py", line 301, in fill_pvi_entries
    await _get_pvi_entries(root_entry, timeout=timeout)
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/_pvi.py", line 257, in _get_pvi_entries
    await _get_pvi_entries(sub_entry)
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/_pvi.py", line 260, in _get_pvi_entries
    _verify_common_blocks(entry, entry.common_device_type)
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/_pvi.py", line 102, in _verify_common_blocks
    raise RuntimeError(
RuntimeError: sub device `create_directory:<class 'typing._GenericAlias'>` was not provided by pvi

Fixed by updating the panda ioc - @olliesilvester is discussing with @evalott100 and @coretl on better ways we might find out about this dependency upfront.

• Issue around sample_shutter:

  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/src/mx_bluesky/hyperion/device_setup_plans/utils.py", line 39, in wrapped_plan
    yield from plan_to_run
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/src/mx_bluesky/hyperion/experiment_plans/robot_load_then_centre_plan.py", line 269, in robot_load_then_centre_plan
    yield from pin_centre_then_xray_centre_plan(
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/src/mx_bluesky/hyperion/experiment_plans/pin_centre_then_xray_centre_plan.py", line 84, in pin_centre_then_xray_centre_plan
    yield from ispyb_activation_wrapper(_pin_centre_then_xray_centre_plan(), parameters)
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 358, in run_wrapper
    yield from contingency_wrapper(plan, except_plan=except_plan, else_plan=close_run)
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 604, in contingency_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/src/mx_bluesky/hyperion/experiment_plans/pin_centre_then_xray_centre_plan.py", line 78, in _pin_centre_then_xray_centre_plan
    yield from detect_grid_and_do_gridscan(
  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/src/mx_bluesky/hyperion/experiment_plans/grid_detect_then_xray_centre_plan.py", line 170, in detect_grid_and_do_gridscan
    sample_shutter=composite.sample_shutter,
                   ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'RobotLoadThenCentreComposite' object has no attribute 'sample_shutter'

Fixed in #477

• We're now using properties from https://daq-config.diamond.ac.uk/ - we need to have a think about this and discuss with the scientists - Demo and discuss daq-config server to scientists  #478
• Got error of:

  File "/dls_sw/i03/software/bluesky/mx_bluesky_1.0.0/hyperion/src/mx_bluesky/hyperion/external_interaction/callbacks/rotation/nexus_callback.py", line 61, in activity_gated_event
    assert self.writer, "Nexus writer not initialised"
AssertionError: Nexus writer not initialised

This was due to missing brackets on a call to model_dump_json, we should have caught this in a test - Fixed in #477

• Aperture is always going into ispyb as ROBOT_LOAD - https://ispyb.diamond.ac.uk/dc/visit/cm37235-4/id/15008409. This was due to missed early return in _safe_move_within_datacollection_range so selected_aperture not getting set. Needs tests. Also maybe we should actually read the state rather than hold it? - Fixed in Make sure that the aperture scatterguard position is set correctly  dodal#784 follow up in Read aperture/scatterguard hardware on read dodal#782
• hyperion_params were coming through to zocalo callback as a string in the start method and failing, had to json load them. - Fixed in Hotfixes found from testing on the beamline #477 by putting the zocalo environment in the emitted doc so there's no need to load all the params. This matches what the XRC does.
• Issue with rotation at chi 30 https://graylog.diamond.ac.uk/messages/graylog-daq-mx-hyperion_21/252aebe1-6f88-11ef-9ed7-eaefa503891f - Needs more testing at next beam time

[coretl]

Fixed by updating the panda ioc - @olliesilvester is discussing with @evalott100 and @coretl on better ways we might find out about this dependency upfront.

Two ideas:

1. We are more explicit about the PandA IOC version requirements in the changelog
2. We annotated PandA with a custom error string containing the IOC version requirement so that it comes out in the error message.

Are either/both of those useful?

[DominicOram]

I think both would be good. For 2 are you suggesting something like the ophyd-async device reading the version from the IOC and throwing an error on connection if it's unexpected?

If we were going to do this we would have to do some clever comparison. For example, we're now using the newest IOC, which seems to work fine with the older and newer ophyd-async panda

[coretl]

I think both would be good. For 2 are you suggesting something like the ophyd-async device reading the version from the IOC and throwing an error on connection if it's unexpected?

No, something much dumber, just an error string we could put in the device that it will echo on error, so that instead of reporting "create_directory" was not provided by pvi, it would report "create_directory" was not provided by pvi: is PandABlocks-ioc at least version x.x?

[DominicOram]

My main concern is that we get issues where version X is required because of some change that would not cause an exception at connect but will cause subtle behaviour issues when it's actually used. If you think the chance of this is low then happy to just add the version string to any errors coming out of connect

[coretl]

Chance should be low of that, we haven't had any issues like that so far