task: persist simulation states in the UI

[blanchco]

Description

• This PR has a few assumptions included, but its a basis to integrate the intermediate results into the HMI
• Feat/pika pyciemss-service#9 Looking at this PR it looks like the progress that is going to be a float between 0 and 1 AND for now only on the calibrate functionality in pyciemss.
• For now I'm assuming we will get a progress attribute when we get the job status, this can be changed once this is more fleshed out
• We can now persist the simulation/calibration state when we leave and come back to a workflow. This is done by adding a new attribute called simulationsInProgress on the node state which is an array of simulationIds. When queuing/running these simulations they will be added to the simulationsInProgress node. On mounting the component it will poll for the jobs. When the simulations are complete, they will be deleted from the simulationsInProgress array.

As @mwdchang mentioned this polling will be changed for an event sourcing architecture - will be looking into websockets.

Screen.Recording.2023-08-08.at.9.49.11.AM.mov

Screen.Recording.2023-08-08.at.9.50.48.AM.mov

References #(issue)
#1634

[mwdchang]

There might be some confusion here:

• Polling for simulation result is fine, results do not leverage message queues
• Intermediates uses message queues and we want to use event-sourcing to listen to updates

[blanchco]

Just to clarify this PR a bit - it was initially an initiative to add intermediate results. Since the BE isn't prepared yet the added changes are just a refactor around polling as well as persisting simulation results in the UI.

[YohannParis]

I do believe we should use SSE for the this, as multiple window/user could look at this.

[blanchco]

I do believe we should use SSE for the this, as multiple window/user could look at this.

I discussed with @mwdchang a little earlier today - I'm looking into this PR this file specifically and how it can be used for the intermediate results here

[blanchco]

I do believe we should use SSE for the this, as multiple window/user could look at this.

I discussed with @mwdchang a little earlier today - I'm looking into this PR this file specifically and how it can be used for the intermediate results here

But just an FYI this PR is not for the intermediate results but instead the persisting of simulation jobs. Intermediate results will follow

[mwdchang]

Two things

1. You have some cases where the functional parameters have variable types plus defaults. e.g. string | string[] = '' ... personally I think this makes things a lot more difficult to comprehend than necessary - I prefer functions to be more restrictive than being flexible, makes it a lot easier to use, to test and to maintain. In these cases I think we should either reduce it down to string, or string[] ... let me know your thoughts.

2. Your return patterns are weird. handleSimulationsInProgress returns "state", but then the return is fed into "getStatus" which expect a list of runIds. These two things are not compatible, am I missing something here?

[blanchco]

Two things

1. You have some cases where the functional parameters have variable types plus defaults. e.g. string | string[] = '' ... personally I think this makes things a lot more difficult to comprehend than necessary - I prefer functions to be more restrictive than being flexible, makes it a lot easier to use, to test and to maintain. In these cases I think we should either reduce it down to string, or string[] ... let me know your thoughts.
2. Your return patterns are weird. handleSimulationsInProgress returns "state", but then the return is fed into "getStatus" which expect a list of runIds. These two things are not compatible, am I missing something here?

@mwdchang

1. Makes sense - It will now always take in a string[] (run ids) and have simplified the add/delete function to only accommodate this.
2. Agreed. It worked but that pass-thru function was a little too flexible to make any sense. I've fixed this up so I hope it makes more sense now. Add/Delete return a state, Query returns a list of run ids

[blanchco]

I've also stopped the poller when unmounting the component

[YohannParis]

All those node having the same progress-bar and code is in big need of refactoring.
Why not leveraging the tera-worflow-node.vue

[blanchco]

All those node having the same progress-bar and code is in big need of refactoring. Why not leveraging the tera-worflow-node.vue

@YohannParis I agree that this is in need of a refactor here, but I think we would need a new component to encapsulate all of the simulation/calibration/ensemble nodes since a lot of the code is shared between those components only. Workflow nodes such as model & dataset do not need this progress/polling functionality within them so Im not sure it would make sense to put it on the tera-workflow-node.

Do you think I should include this refactor as a part of this PR? I'm a little worried of doing too much in this PR alone. (Persisting simulation status + Intermediate results + refactor of simulation/calibration/ensemble nodes)

[mwdchang]

@YohannParis @blanchco Let's leave progress-bar refactor for another time, arguably it might have been introduced a tad too early 😄

• It isn't clear that all the engined (julia/ciemss), or even individual operations will have the same behaviour when it comes to relaying progress information
• Given some of the conversations we have had in the last 2 days or so, I think there are still some confusion over the notions of progress, 'simulation-poll', and intermediates. Both intermediate and simulations-polls can denote some sense of "progress" and we should sync up on what we want to convey before undertaking large refactoring efforts.

[mwdchang]

Didn't test all the cases, but looks fine in general, thanks