Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problem with python multiprocessing? #43

Open
halfluke opened this issue Jul 12, 2021 · 1 comment
Open

problem with python multiprocessing? #43

halfluke opened this issue Jul 12, 2021 · 1 comment

Comments

@halfluke
Copy link

Traceback (most recent call last):
File "/usr/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.9/multiprocessing/util.py", line 224, in call
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.9/multiprocessing/pool.py", line 692, in _terminate_pool
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/usr/lib/python3.9/multiprocessing/pool.py", line 674, in _help_stuff_finish
inqueue._reader.recv()
File "/usr/lib/python3.9/multiprocessing/connection.py", line 256, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "/usr/lib/python3.9/multiprocessing/managers.py", line 933, in RebuildProxy
return func(token, serializer, incref=incref, **kwds)
File "/usr/lib/python3.9/multiprocessing/managers.py", line 981, in AutoProxy
proxy = ProxyType(token, serializer, manager=manager, authkey=authkey,
File "/usr/lib/python3.9/multiprocessing/managers.py", line 783, in init
self._incref()
File "/usr/lib/python3.9/multiprocessing/managers.py", line 837, in _incref
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib/python3.9/multiprocessing/connection.py", line 507, in Client
c = SocketClient(address)
File "/usr/lib/python3.9/multiprocessing/connection.py", line 635, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused

Connection refused to what? It's a local file I am running dumpsterdiver against...
I found the reference to multiprocessing in core.py and I removed it and now it works.
What's wrong with python multiprocessing module??

@halfluke
Copy link
Author

apparently solved by putting time.sleep(1) after the "while" in core.py
def mp_handler(queue, result, settings):
# depending on your hardware the DumpsterDiver will use all available cores for
# parallel processing

p = multiprocessing.Pool(multiprocessing.cpu_count())
while queue.qsize():
    time.sleep(1)
    p.apply_async(worker, (queue,result,settings))
queue.join()

Still curious to know if it should be considered a multiprocessing module issue...

JaredStemper added a commit to JaredStemper/DumpsterDiver that referenced this issue Apr 25, 2023
Following workaround suggested in securing#43 to avoid concurrency issues.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant