structuring pipeline from R to Vue

[cnell-usgs]

A few considerations for how the data processing pipeline outputs interface with Vue.

File hosting
To avoid committing large .svg or .csv files associated with each time period, we need to figure out where to store these files that are accessible during the jenkins build, and still be publicly available. Can we push to S3 and access from there during the build?

Re-generating the page with different time periods/datasets
When we have multiple time periods, how can we access files from past time periods (and not just the default data that is used in the initial build)? How do we dynamically store the information about other dates/iterations in a way that is accessible in vue?

[cnell-usgs]

Related commentary #26

[lindsayplatt]

Re: different time periods. I think the pipeline can handle building datasets for different dates and then pushing those to whereever we are hosting the files (whatever the solution to the first thing above) by using some kind of task table based on a set of dates. Maybe the dates we use are somewhat manually generated?

[lindsayplatt]

Targets can very easily handle file targets with different names based on a date, see an example where Jake did this for a model on GitLab here. Still don't know a great way to do it for scipiper other than what Cee did in #62 with https://github.com/USGS-VIZLAB/gw-conditions/pull/62/files#diff-1d5ecb3771321ff41a9eeb1fedbd7bd43f56570dcad348ac26592784d254000b.

[lindsayplatt]

^ the scipiper way we could implement this would be to use a task table with one task and have that file target get named dynamically. I think we would only want to build one time period per pipeline build but we could include all in the task table - we don't want to build old ones if they already exist on S3 (but we could also go back and edit the dates to rebuild them if we wanted to)