Accumulating Two-Stage Form Compiler memory

[lindsayad]

Hi, we have an HDG Navier-Stokes application in which we perform a sequence of SNES solves with lower and lower viscosity using the previous solution as the initial guess for the next solve. I am running into OOM errors on my cluster. Using tracemalloc, the memory accumulation appears to be occurring due to

/home/lindad/firedrake/src/firedrake/firedrake/tsfc_interface.py:95: size=87.2 MiB, count=592773, average=154 B 

Do you have any suggestions on how to clear this memory? I added a gc.collect() at the end of each solve but that doesn't appear to have an effect on this.

[connorjward]

It might be the in-memory cache here: https://github.com/firedrakeproject/firedrake/blob/master/firedrake/tsfc_interface.py#L57. In which case you want to do

TSFCKernel._cache.clear()

this is what Thetis do in their tests.

[lindsayad]

Unfortunately that did not fix it. I wish I could see more of a stack trace from tracemalloc than

196380 memory blocks: 29662.4 KiB
  File "/home/lindad/firedrake/src/firedrake/firedrake/tsfc_interface.py", line 95
    return cls._cache.setdefault((key, comm.py2f()), pickle.loads(val))

[wence-]

These parameters shouldn't change the form (if you wrapped them in Constant), so you wouldn't make more kernels.

[lindsayad]

No longer getting the _func_cache errors, I'm still seeing the growth from the tsfc_interface every solve even with wrapping $nu$ in Constant. So I must still be doing something wrong

[lindsayad]

What do you guys use for memory profiling? I know your site suggests using memory_profiler. However, I am not finding its output (attached) to be helpful in tracking down where all the memory allocation is. And tracemalloc just gives me a single frame for the tsfc_interface.py.

profiler.log

[connorjward]

I don't do a tremendous amount of memory profiling so I don't know another tool to recommend. @Ig-dolci you have been debugging some memory leaks recently, do you have any memory-profiling tool recommendations?

Looking through your code it looks like you reuse the same mesh hierarchy for each problem. Ideally that should be fine but we do cache a fair amount of things on the mesh so this might be where you are running out of room. It may be informative to try moving the mesh construction code into the loop to see if that stops this issue, which would point us to where the problem is.

[Ig-dolci]

I am using memory_profiler, which has worked for my memory leak investigation. @lindsayad Are you using NonlinearVariationalSolver?

[lindsayad]

Hi @Ig-dolci, yes I am