How to reduce the frequency of task status lines in the log file

[cvyaru]

Hello,

I started to use NextFlow with Az. Batch. I'm facing an issue with very large log files due to status lines that are saved every 10 sec for each task.

This is an example:
Oct-21 11:44:42.172 [Task monitor] DEBUG n.c.azure.batch.AzBatchTaskHandler - [AZURE BATCH] Task status pred (7) taskId=nextflow.cloud.azure.batch.AzTaskKey(job-79276a5a2b574004e3f0-pred, nf-8ab08a3a0d70a7cacb79dbddfbe65479); running=false
Oct-21 11:44:42.195 [Task monitor] DEBUG n.c.azure.batch.AzBatchTaskHandler - [AZURE BATCH] Task status pred (9) taskId=nextflow.cloud.azure.batch.AzTaskKey(job-79276a5a2b574004e3f0-pred, nf-50126bf7a817bdff7f4aaf040a72669a); running=false
Oct-21 11:44:42.219 [Task monitor] DEBUG n.c.azure.batch.AzBatchTaskHandler - [AZURE BATCH] Task status pred (10) taskId=nextflow.cloud.azure.batch.AzTaskKey(job-79276a5a2b574004e3f0-pred, nf-fb8617941812bd733b2162d1d4397f56); running=false

Could you please indicate how to vary the frequency of these lines? I tried with executor.pollInterval, but it is not the right one. Thanks!

[bentsherman]

pollInterval is the correct setting:

executor.pollInterval = '60s'

Just note that this is the interval between checking all tasks, not each task. So you will see a wall of status updates every 60 seconds.

[cvyaru]

Hi @bentsherman, thanks for the quick reply. I note that I did not describe the situation properly. The lines I mentioned above keep being saved in the log for all the tasks that are NOT running. This is because I have more tasks than available nodes, in Az. Batch, for this job. In jobs where all tasks can run in parallel, those specific lines are not saved, after all tasks are active, and the ones below are then printed following the frequency fixed in pollInterval as you pointed out.

Oct-21 14:25:11.596 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor azurebatch > tasks to be completed: 4 -- submitted tasks are shown below
~> TaskHandler[id: 1; name: pred (1); status: RUNNING; exit: -; error: -; workDir: az://work/09/f00e370b9390bee93a0ed7f7ea1091]
~> TaskHandler[id: 2; name: pred (2); status: RUNNING; exit: -; error: -; workDir: az://work/9e/60782acdfd8bba1c28b6cc8b0cf091]
~> TaskHandler[id: 4; name: pred (4); status: RUNNING; exit: -; error: -; workDir: az://work/fa/279661208598c0896c40cfcd32b6ef]
~> TaskHandler[id: 3; name: pred (3); status: RUNNING; exit: -; error: -; workDir: az://work/09/e922ba2442dd5cd9391aa00489e506]
Oct-21 14:30:11.604 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor azurebatch > tasks to be completed: 4 -- submitted tasks are shown below
~> TaskHandler[id: 1; name: pred (1); status: RUNNING; exit: -; error: -; workDir: az://work/09/f00e370b9390bee93a0ed7f7ea1091]
~> TaskHandler[id: 2; name: pred (2); status: RUNNING; exit: -; error: -; workDir: az://work/9e/60782acdfd8bba1c28b6cc8b0cf091]
~> TaskHandler[id: 4; name: pred (4); status: RUNNING; exit: -; error: -; workDir: az://work/fa/279661208598c0896c40cfcd32b6ef]
~> TaskHandler[id: 3; name: pred (3); status: RUNNING; exit: -; error: -; workDir: az://work/09/e922ba2442dd5cd9391aa00489e506]

I want to control the frequency, or even prevent completely the status report for tasks that are not running yet.

[bentsherman]

I guess we could demote the status logging to trace, let me draft a PR

[bentsherman]

#5416