flux-filemap/stage-in use case

[grondo]

@ardangelo asks in #4572:

We're using flux-filemap to ship files from the broker node to the non-broker nodes, but we still needed a way to get the file from the frontend where we're running our debugger tools to the broker node.

Although currently, we only are supporting running our tools from inside the flux start shell. Could we add files to the filemap directly in that case without worrying that the broker would be running somewhere else?

I'm pretty confident that flux-filemap(1) and/or the shell stage-in plugin will work for this use case, but it will be good to capture any discussion in a Q&A here for future reference.

[grondo]

@ardengelo, what is meant by "non-broker nodes" above. Do you mean rank > 0?

AFAIK, flux-filemap always maps the files on the rank 0 broker. The files may then be extracted on any other rank using the overlay. Currently Flux always has a broker per node in the system instance and for each job, so the files should be available via filemap on all nodes of any job. The key is whether the files are available to be mapped into the rank 0 broker...

[garlick]

Note also that filemap requires "instance owner" privileges, so users cannot map files in the system instance of Flux for use by direct run jobs (flux run). It is intended to work within a flux alloc or flux batch subinstance. For example, you would flux alloc some nodes, then from the shell spawned on the subinstance's rank 0 node, interactively start a job under the debugger with flux run.

[ardangelo]

Our current workflow for running tools under Flux looks like this:

• Start a Flux instance with flux start either under srun or not
• Inside the Flux shell, start the tool and launch a Flux job via the submission system
• During tool launch, ship files to rank 0, map them with filemap by running flux exec -r 0 flux filemap map
• Distribute files to ranks > 0 using flux exec -r <non-0 ranks> flux filemap get

Is the Flux shell always running on the broker node, rank 0? If so, then we could skip shipping to rank 0, right? As the tool would be running on rank 0 in the first place, along with its files.
Regarding flux alloc / flux batch, using filemap is working for us when run under flux start. Does flux start give the same "instance owner" privileges with regards to filemap?

[garlick]

Is the Flux shell always running on the broker node, rank 0?

Yes

Does flux start give the same "instance owner" privileges with regards to filemap?

Yes

So maybe the use case is handled? Let us know if there are other gotchas for this use case.

[ardangelo]

Great to know. We may come back to this in the future if we support attaching from outside the Flux instance, but for the current workflow this should be handled. Thanks!