Ask Prefect re network connection error

[leewujung]

In the Shimada deployment we continue to run into connection issue due to internet problems. It seems that Prefect requires pinging and getting responses from the runs continuously, and would hang (stuck at “running” status, rather than “failed”) if the connection was not able to be reached within a certain short period of time.

We should try to see if we can connect with Prefect to see:

• if there are parameters to change to lengthen the communication intervals for which the cloud dash board requires to hear back from the runs
• if there are retry or timeout mechanisms that can be set up for killing the hanging runs