[TASK]: Improvements on workflow status tracking

[mwdchang]

This relates to several issues raised in: #2580

Summary

Currently in workflow, each operator tracks its own run status, for example:

• A simulator node runs a long simulation, it is responsible for polling and updating its own state

This works, but however it means in the higher level graph view we don't really have a good sense of progress, and we will not see updates until we go into the drilldown view to complete the pending transactions.

We should brainstorm a few ideas of how we can accomplish this, and provide some estimate of the effort needed.

Food for thoughts:

• Currently we have a polling mechanism for checking progress, we should try to keep the implementation agnostic so we can move to use MQ/SSE if needed
• We probably want to have different update/polling frequencies in the graph-view vs drilldown view
• Because the drilldown-view is not a separate page but an extension of the graph-view, we want to avoid double updates or race conditions, or at least, we want to keep the state/output consistent.

Thoughts - DC

Initially the operators were created with the idea of having to define a run() method, that will completely incapsulate the logic needed to produce an output. We may want to see if we can revisit similar ideas, and make these type of methods available in both the graph-view and the drilldown-view. For example, maybe in the graph-view we would have something like (pseudo code):

set_interval(4000):
  for pending_task in extract_pending_task(workflow):
    get_operator(pending_task).run(pending_task.input, pending_task.state);

and within the drilldown

get_operator(node).run(node.input, node.state);

CC @YohannParis @liunelson