Memory leak?

[BolekZapiec]

I'm running a FIJI macro on a dedicated Ubuntu machine that has a 2 step Unet segmentation. It all works on single images but when I run a macro on a folder, every 7-8 image I get the error below. Running garbage collection doesn't help - only quiting and relaunching Fiji - I've reproduced this behavior 5 times now on different sets of images.

Please let me know if there's anything I can do to help identify the error or allow it to run across a whole folder without restarting the run each time.

(Fiji Is Just) ImageJ 2.0.0-rc-69/1.52n; Java 1.8.0_172 [64-bit]; Linux 4.15.0-47-generic; 15317MB of 30000MB (51%)

java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at ij.macro.Functions.call(Functions.java:4363)
at ij.macro.Functions.getStringFunction(Functions.java:274)
at ij.macro.Interpreter.getStringTerm(Interpreter.java:1407)
at ij.macro.Interpreter.getString(Interpreter.java:1385)
at ij.macro.Interpreter.doStatement(Interpreter.java:324)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.runUserFunction(Interpreter.java:365)
at ij.macro.Interpreter.doStatement(Interpreter.java:276)
at ij.macro.Interpreter.doIf(Interpreter.java:1044)
at ij.macro.Interpreter.doStatement(Interpreter.java:291)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.doStatement(Interpreter.java:315)
at ij.macro.Interpreter.doFor(Interpreter.java:588)
at ij.macro.Interpreter.doStatement(Interpreter.java:297)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.runUserFunction(Interpreter.java:365)
at ij.macro.Interpreter.doStatement(Interpreter.java:276)
at ij.macro.Interpreter.doStatements(Interpreter.java:256)
at ij.macro.Interpreter.run(Interpreter.java:152)
at ij.macro.Interpreter.run(Interpreter.java:91)
at ij.macro.Interpreter.run(Interpreter.java:102)
at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
at ij.IJ.runMacro(IJ.java:148)
at ij.IJ.runMacro(IJ.java:137)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1107)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1103)
at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1054)
at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1103)
at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:147)
at org.scijava.script.ScriptModule.run(ScriptModule.java:160)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:238)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Macro canceled
at ij.Macro.abort(Macro.java:76)
at ij.IJ.error(IJ.java:691)
at de.unifreiburg.unet.SegmentationJob.run(SegmentationJob.java:737)
at de.unifreiburg.unet.SegmentationJob.processHyperStack(SegmentationJob.java:620)

[BolekZapiec]

I didn't post the actual error dialog for the previous message it is:

While the above message is the more frequent one, on the same 6-8 image cycle I sometimes instead get the message below. It's less frequent but seems to happen in lieu of the above message (interchangeably with the above message). As previously, restarting fiji and rerunning with same settings fixes it - rerunning it without restarting Fiji results in the same error.
The specs:
Ubuntu 18.04.1 LTS
i9-9900k 3.6Ghz
32GB RAM
2x EVGA GeForce RTX 2080 Ti 11264MB (not linked)
(one training is running on one GPU via network, macro is running locally on IJ on other GPU)
TileSize setting is: 1396x1396 which results in 6.5GB being used when running - I had it higher before but brought it down thinking that might be the problem. Happens at same frequency since It did with larger tile size / more memory being used.

Any ideas or advice? Thanks!

(Fiji Is Just) ImageJ 2.0.0-rc-69/1.52n; Java 1.8.0_172 [64-bit]; Linux 4.15.0-47-generic; 15761MB of 30000MB (52%)

java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at ij.macro.Functions.call(Functions.java:4363)
at ij.macro.Functions.getStringFunction(Functions.java:274)
at ij.macro.Interpreter.getStringTerm(Interpreter.java:1407)
at ij.macro.Interpreter.getString(Interpreter.java:1385)
at ij.macro.Interpreter.doStatement(Interpreter.java:324)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.runUserFunction(Interpreter.java:365)
at ij.macro.Interpreter.doStatement(Interpreter.java:276)
at ij.macro.Interpreter.doIf(Interpreter.java:1044)
at ij.macro.Interpreter.doStatement(Interpreter.java:291)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.doStatement(Interpreter.java:315)
at ij.macro.Interpreter.doFor(Interpreter.java:588)
at ij.macro.Interpreter.doStatement(Interpreter.java:297)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.runUserFunction(Interpreter.java:365)
at ij.macro.Interpreter.doStatement(Interpreter.java:276)
at ij.macro.Interpreter.doStatements(Interpreter.java:256)
at ij.macro.Interpreter.run(Interpreter.java:152)
at ij.macro.Interpreter.run(Interpreter.java:91)
at ij.macro.Interpreter.run(Interpreter.java:102)
at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
at ij.IJ.runMacro(IJ.java:148)
at ij.IJ.runMacro(IJ.java:137)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1107)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1103)
at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1054)
at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1103)
at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:147)
at org.scijava.script.ScriptModule.run(ScriptModule.java:160)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:238)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Macro canceled
at ij.Macro.abort(Macro.java:76)
at ij.IJ.error(IJ.java:691)
at de.unifreiburg.unet.SegmentationJob.run(SegmentationJob.java:737)
at de.unifreiburg.unet.SegmentationJob.processHyperStack(SegmentationJob.java:620)
... 41 more

[ThorstenFalk]

So the problematic process runs locally? I am not aware of a memory leak between calls to caffe_unet. Since each call spawns a completely independent process, and the next should only be able to start when the previous one finished, the GPU should be completely free between two segmentations. The only thing I could imagine is that the different calls in the macro try to execute in parallel. This should show up in nvidia-smi top and ps as multiple caffe_unet processes. I am currently on vacation, but will look into it when I'm back in two weeks.

[BolekZapiec]

Hope my previous post didn't put a dent in your vacation :-)
After some testing it seems to be isolated to running the script locally on Fiji on ubuntu as I didn't encounter the problem running it on Windows and remotely tasking caffe on the Ubuntu box. Perhaps it's an issue with my Fiji ubuntu install. I'll follow up if I figure out the underlying cause.