server connection loss during consecutive bias steps

[csierra2]

We were running a noise vs bias voltage script for a range of 20 bias points, and also took bias steps at each point. The script was basically this:

bias_points = np.linspace(0,19.9, 20)[::-1]
bgs = range(12)
biases = np.ones(bgs)

for p in bias_points:
        bias_to_volt_arr(
                S,
                cfg,
                p*biases,
                bias_groups=bgs,
                overbias=False,
        )
        time.sleep(step_wait*60)
        bs_op.take_bias_steps(S,cfg,bgs=bgs)
        time.sleep(60)
        sdl.noise.take_noise(S,cfg,acq_time=30,show_plot=False,save_plot=True)

with an overbias step just before.

This measurement ran fine for our first slot (for which there were only 6 bias lines), but failed 3 consecutive times for our other slot (12 bias lines). Each time the errors were slightly different but all seemed related to an epics server crash and required hammering to bring the slot back up. This crash also happened around the 10th data point every time.

After removing the bias steps line from each point, the measurement ran without issues.

First error:

[ 2023-01-09 21:13:39 ]  BG 11: 11.52
[ 2023-01-09 21:14:40 ]  Waiting 3 sec after switching to hcm
[ 2023-01-09 21:14:43 ]  Input downsample factor is None. Using value already in pyrogue: 1
[ 2023-01-09 21:14:43 ]  FLUX RAMP IS DC COUPLED.
[ 2023-01-09 21:14:46 ]  caput smurf_server_s4:AMCc:SmurfProcessor:Unwrapper:reset 1
[ 2023-01-09 21:14:46 ]  caput smurf_server_s4:AMCc:SmurfProcessor:Filter:reset 1
[ 2023-01-09 21:14:48 ]  Writing to file : /data/smurf_data/20230109/ufm_uv31/1673296411/outputs/1673298888.dat
[ 2023-01-09 21:14:48 ]  /data/smurf_data/20230109/ufm_uv31/1673296411/outputs/1673298888_mask.txt
[ 2023-01-09 21:14:48 ]  Writing frequency mask.
[ 2023-01-09 21:15:01 ]  Command failed: smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:SysgenCryo:Base[5]:CryoChannels:centerFrequencyArray
[ 2023-01-09 21:15:01 ]  Retry attempt 1 of 5
[ 2023-01-09 21:15:10 ]  Command failed: smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:SysgenCryo:Base[6]:CryoChannels:centerFrequencyArray
[ 2023-01-09 21:15:10 ]  Retry attempt 1 of 5
[ 2023-01-09 21:15:22 ]  Command failed: smurf_server_s4:AMCc:SmurfProcessor:SOStream:SOFileWriter:session_id
[ 2023-01-09 21:15:22 ]  Retry attempt 1 of 5
[ 2023-01-09 21:15:24 ]  Session id: 1673298923
^AdCA.Client.Exception...............................................
    Warning: "Virtual circuit unresponsive"
    Context: "localhost:44239"
    Source File: ../tcpiiu.cpp line 919
    Current Time: Mon Jan 09 2023 21:22:57.516943853
..................................................................
cannot connect to smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:LutCtrl:Ctrl:EnableCh
cannot connect to smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacCtrlRegCh[15]
cannot connect to smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacCtrlRegCh[16]
cannot connect to smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[15]
cannot connect to smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:RtmSpiMax:TesBiasDacDataRegCh[16]

Second error (same as third?):

Traceback (most recent call last):
  File "/sodetlib/sodetlib/operations/bias_steps.py", line 1179, in take_bias_steps
    play_bias_steps_waveform(
  File "/sodetlib/sodetlib/operations/bias_steps.py", line 122, in play_bias_steps_waveform
    _play_tes_bipolar_waveform(S, bias_group, sig)
  File "/sodetlib/sodetlib/operations/bias_steps.py", line 59, in _play_tes_bipolar_waveform
    S.set_rtm_arb_waveform_continuous(1)
  File "/usr/local/src/pysmurf/python/pysmurf/client/command/smurf_command.py", line 3559, in set_rtm_arb_waveform_continuous
    self._caput(
  File "/usr/local/src/pysmurf/python/pysmurf/client/command/smurf_command.py", line 97, in _caput
    epics.caput(cmd, val, wait=wait_done, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/epics/__init__.py", line 105, in caput
    return thispv.put(value, wait=wait, timeout=timeout)
  File "/usr/local/lib/python3.8/dist-packages/epics/pv.py", line 47, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/epics/pv.py", line 652, in put
    return ca.put(self.chid, value,
  File "/usr/local/lib/python3.8/dist-packages/epics/ca.py", line 613, in wrapper
    raise ChannelAccessException(fmt % (fcn.__name__,
epics.ca.ChannelAccessException: put() timed out waiting 'smurf_server_s4:AMCc:FpgaTopLevel:AppTop:AppCore:RtmCryoDet:LutCtrl:Ctrl:Continuous' to connect (59 seconds)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "noise_vs_bias_20230105.py", line 87, in <module>
    bs_op.take_bias_steps(S,cfg,bgs=bgs)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/pub.py", line 50, in wrapper
    rv = func(S, *args, **kwargs)
  File "/sodetlib/sodetlib/operations/bias_steps.py", line 1191, in take_bias_steps
    sdl.stream_g3_off(S)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/pub.py", line 50, in wrapper
    rv = func(S, *args, **kwargs)
  File "/sodetlib/sodetlib/stream.py", line 216, in stream_g3_off
    sess_id = reg.g3_session_id.get()
  File "/sodetlib/sodetlib/util.py", line 543, in get
    return self.S._caget(self.addr, **kw)
  File "/usr/local/src/pysmurf/python/pysmurf/client/command/smurf_command.py", line 194, in _caget
    raise RuntimeError("epics failed to respond")
RuntimeError: epics failed to respond

[jlashner]

Is this the actual code you were running? This line:

bgs = range(12)
biases = np.ones(bgs)

should actually be something like this

bgs = np.arange(12)
biases = np.ones(len(bgs))

Not sure if this is actually the issue, but the first one results in this:

>>> biases = np.ones(bgs)
>>> biases
array([], shape=(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11), dtype=float64)

which epics might interpret in some weird way.

[csierra2]

No sorry that part was from memory, here is the full script we were running (with the first slot part commented out):

import numpy as np
import time

import sodetlib as sdl
from sodetlib.det_config import DetConfig
import sodetlib.noise as so_noise
import sodetlib.operations.bias_steps as bs_op
from sodetlib.operations.bias_dets import bias_to_rfrac, bias_to_volt_arr

bgs_uv8 = [6,7,8,9,10,11]
bgs_uv31 = [0,1,2,3,4,5,6,7,8,9,10,11]

bias_points = np.linspace(0,19.9, 20)[::-1]
overbias_wait = 5 #min
step_wait = 1 #min (use larger values for larger steps)

'''
### Uv8 ###
bgs = bgs_uv8
biases = np.array([0,0,0,0,0,0,1,1,1,1,1,1])
S = cfg.get_smurf_control(
        dump_configs=True, 
        make_logfile=False, 
        apply_dev_configs=True, 
        load_device_tune=True
)

bias_to_volt_arr(
        S, 
        cfg, 
        bias_points[0]*biases, 
        bias_groups=bgs,
        overbias=True,
)
time.sleep(overbias_wait*60)
bs_op.take_bias_steps(S,cfg,bgs=bgs)
time.sleep(60)
sdl.noise.take_noise(S,cfg,acq_time=30,show_plot=False,save_plot=True)

for p in range(1,len(bias_points)):
        bias_to_volt_arr(
                S, 
                cfg, 
                bias_points[p]*biases, 
                bias_groups=bgs,
                overbias=False,
        )
        time.sleep(step_wait*60)
        bs_op.take_bias_steps(S,cfg,bgs=bgs)
        time.sleep(60)
        sdl.noise.take_noise(S,cfg,acq_time=30,show_plot=False,save_plot=True)
'''

### Uv31 ###
bgs = bgs_uv31
biases = np.ones_like(bgs)
cfg = DetConfig()
cfg.load_config_files(slot=4)

S = cfg.get_smurf_control(
        dump_configs=True, 
        make_logfile=False, 
        apply_dev_configs=True, 
        load_device_tune=True
)

bias_to_volt_arr(
        S, 
        cfg, 
        bias_points[0]*biases, 
        bias_groups=bgs,
        overbias=True,
)
time.sleep(overbias_wait*60)
bs_op.take_bias_steps(S,cfg,bgs=bgs)
time.sleep(60)
sdl.noise.take_noise(S,cfg,acq_time=30,show_plot=False,save_plot=True)

for p in range(1,len(bias_points)):
        bias_to_volt_arr(
                S, 
                cfg, 
                bias_points[p]*biases, 
                bias_groups=bgs,
                overbias=False,
        )
        time.sleep(step_wait*60)
        bs_op.take_bias_steps(S,cfg,bgs=bgs)
        time.sleep(60)
        sdl.noise.take_noise(S,cfg,acq_time=30,show_plot=False,save_plot=True)