BUG: double free errors

[tylerjereddy]

Probably there should never be double free C++ errors, regardless of the input data, if the memory handling is safe throughout the library and its dependency stack.

Here is a reproducer using nothing more than NumPy, pytest, and the hypothesis[numpy] property-based testing library:

Run the test below with version 1.0 of memsurfer with: pytest reproducer.py -s

import numpy as np
import memsurfer
import hypothesis
from hypothesis import given
from hypothesis import strategies as st
from hypothesis.extra import numpy as hynp


@given(
      # generate the leaflet coords
      hynp.arrays(np.float64, (100, 3)),
      # and the box coords
      hynp.arrays(np.float64, (2, 3)), 
      # and the resnames
      hynp.arrays(np.str_, 100),
        )
def test_memsurfer_double_free(leaflet_coords, bbox, labels):
    # this test causes memsurfer or one of its
    # dependencies to produce C++ "double free"
    # errors
    memsurfer.Membrane.compute(leaflet_coords,
                               labels=labels,
                               bbox=bbox,
                               periodic=False)

It will hard crash:

reproducer.py free(): double free detected in tcache 2
Fatal Python error: Aborted

Current thread 0x00007f230f663b80 (most recent call first):
  File "/autofs/nccs-svm1_home1/treddy/github_projects/spack/opt/spack/linux-rhel8-zen/gcc-8.3.1/memsurfer-1.0-36bovnle23ei6dcloohpm7bnpjminva5/lib/python3.7/site-packages/memsurfer-1.0.0-py3.7-linux-x86_64.egg/memsurfer/membrane.py", line 160 in compute_approx_surface
  File "/autofs/nccs-svm1_home1/treddy/github_projects/spack/opt/spack/linux-rhel8-zen/gcc-8.3.1/memsurfer-1.0-36bovnle23ei6dcloohpm7bnpjminva5/lib/python3.7/site-packages/memsurfer-1.0.0-py3.7-linux-x86_64.egg/memsurfer/membrane.py", line 300 in compute
  File "/autofs/nccs-svm1_home1/treddy/voronoi_pilot2/reproducer.py", line 24 in test_memsurfer_double_free
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/core.py", line 523 in test
  File "/autofs/nccs-svm1_home1/treddy/voronoi_pilot2/reproducer.py", line 10 in test_memsurfer_double_free
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/core.py", line 599 in run
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/executors.py", line 52 in default_new_style_executor
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/core.py", line 603 in execute_once
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/core.py", line 648 in _execute_once_for_engine
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/internal/conjecture/engine.py", line 137 in __stoppable_test_function
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/internal/conjecture/engine.py", line 155 in test_function
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/internal/conjecture/engine.py", line 989 in cached_test_function
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/internal/conjecture/engine.py", line 519 in generate_new_examples
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/internal/conjecture/engine.py", line 818 in _run
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/internal/conjecture/engine.py", line 420 in run
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/core.py", line 724 in run_engine
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/hypothesis/core.py", line 1050 in wrapped_test
  File "/autofs/nccs-svm1_home1/treddy/voronoi_pilot2/reproducer.py", line 10 in test_memsurfer_double_free
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/autofs/nccs-svm1_sw/andes/python/3.7/anaconda-base/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/ccs/home/treddy/.local/andes/anaconda3/3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 163 in main
  File "/sw/andes/python/3.7/anaconda-base/bin/pytest", line 11 in <module>
Aborted (core dumped)

[tylerjereddy]

Was built on andes node with spack as follows:

cd /ccs/home/treddy/github_projects/spack
git checkout 2421d90
. share/spack/setup-env.sh
spack install --no-checksum memsurfer +osmesa %gcc@8.3.1

[ctlee]

This may be related to the poisson reconstruction python binding interface mmolero/pypoisson#13

[bhatiaharsh]

@tylerjereddy , thanks for opening the issue and @ctlee , thanks for connecting it with the pypoisson issue.

There appears to be a memory bug in the original code for python reconstruction that slips through the cracks in older compilers, but it appears that gcc>8 is not letting it through. I spent quite some time digging down the cause, and I suspect really fixing it requires an overhaul of the original cpp code (see my comment here.)

Until then, I tried the code again with gcc@7.5.0 and I was able to successfully run.

I will leave this issue open but add to ReadMe the need to stick with gcc@7 until someone figures this out.