2021 01 19.devel

• Peter's incite on Summit:

  • setup some testing dedicated infrastructure
  • problems mostly env related
  • environment setup
  • how do we make RCT work out of the box?
    • research vs. consultant vs. solid product
  • some error modes seem workload related
  • need to target the different set of problems being faced
    • deployment
    • scaling
    • task stability
    • config variability
    • etc.
  • approach:
    • provided stable environment setup for the user
    • regular small scale integration tests, including a full example
    • leave production testing to group

• PR

  • merge multi-DVM

• RE

  • fault tolerance

• storage

• Task Description

  • V3 it is

• release

  • RP/RU Jan:
    • unit -> task
      • @aymen: run example
    • Multi-DVM
    • task fixes

• XSEDE allocation:

  • TODO Shantenu: draft extension

• roadmap update for stack

  • NoDB
  • conda pack: TODO IP, MTi

• termination in function executor

  • TODO AY: schedule call for Wednesday
  • TODO AM, MTi: discuss verify / prepare

• TODO Agenda: futures

• TODO AGENDA RCT

  • data archival: where to store raw data
  • Github LFS: 60,-- per 50GB / yr
    • allows to version, annotate, ...
  • Pylon: free
    • unstructured, structure difficult to enforce
  • own server: maintenance, how to host, how to share?
  • TODO ALL: clarify GIT-LFS usage mode
  • TODO ALL: propose gdrive data organisation
  • TODO AGENDA: follow-up
  • TODO AGENDA: discussions
  • GH is not cheap
    • glacier is cheapest
  • data curation is a major issue
    • we need to prescribe structure
  • try with gdrive as cheap reasonable option
  • gdrive allows fuse
    • TODO AM : clean / document Matteo's script
    • TODO MTu: document structure
  • conclusion: we favor gdrive

TODOs