Reading files with uproot and xrootd: how to set timeout?

[andreypz]

Hello,

I would like to use processor.run_uproot_job() method and open multiple files located on xrootd. It works for a few files, but crashes when I try more.
I think it's related to the fact the the xrd.timeout is not set (I can see that by printing out the value in uptoot/source/xrootd.py - it returns None). So my question is how can I set it in processor.run_uproot_job()?

[lgray]

@andreypz It would be helpful to see how things are crashing in your case. It could be due to a variety of things beyond the timeout. Typically the lack of a timeout causes stalls and not crashes. In particular you should try telling run_uproot_job to skip bad files: https://github.com/lgray/hgg-coffea/blob/master/runner.py#L355.

To make a timeout really sticky you need to set it when the dask/parsl/etc worker job is started. If you are running with Dask you should add the timeouts via dask to the worker environment. You need to set XRD_REQUESTTIMEOUT and XRD_STREAMTIMEOUT to some reasonable values. For the local executors you can set the environment variables yourself or use the xrootdtimeout executor config option.

run_uproot_job(..., executor_args={"xrootdtimeout": Nseconds, ...}, ...)

[lgray]

Setting "skipbadfiles": True does not work for me. It crashes with:

    output = processor.run_uproot_job(file_list,
  File "/.automount/home/home__home1/institut_3a/andreypz/work/vjets/gen-scripts/geneno/lib/python3.8/site-packages/coffea/processor/executor.py", line 1222, in run_uproot_job
    processor_instance.postprocess(wrapped_out["out"])
TypeError: 'NoneType' object is not subscriptable

@nsmith- bug?

[nsmith-]

I think that happens when all files are skipped, hence the output is null. Could be handled better

[andrzejnovak]

Yup that's a common question that comes up

[jpivarski]

I don't know how many slashes are needed in an XRootD URL, but Uproot does not modify that.

One thing to know about XRootD timeouts is that they must be integers, and hence whole numbers of seconds. If you're trying to set a fraction of a second, it probably gets cast as an int and therefore becomes 0, which I think means "no timeout."

[andreypz]

Thanks @lgray , changing to two slashes seems to fix this one. Strange that three slashes worked fine for all other files I run over. Anyway, turns out timeout is irrelevant for this issue.