Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read attributes inaccessible via multiprocessing pool #1877

Open
dkoslicki opened this issue Aug 19, 2018 · 0 comments
Open

Read attributes inaccessible via multiprocessing pool #1877

dkoslicki opened this issue Aug 19, 2018 · 0 comments

Comments

@dkoslicki
Copy link

For the Read class, it appears that the sequence attribute is inaccessible when passed to a parallel pool. As a MWE, consider:

import multiprocessing
import khmer

def my_func(record):
	return len(record.sequence)

parser = khmer.ReadParser('test_file.fq')

# read in a few records
read_list = []
i = 0
for record in parser.iter_reads():
	read_list.append(record)
	if i >= 10:
		break
	i += 1

pool = multiprocessing.Pool(processes=2)

res = map(my_func, read_list)  # works
res2 = pool.map(my_func, read_list)  # doesn't work: AttributeError: 'Read' object has no attribute 'sequence'.

print(res)
print(res2)
pool.close()

This results in:

Traceback (most recent call last):
  File "MinimumWorkingError.py", line 22, in <module>
    res2 = pool.map(my_func, read_list)  # doesn't work: AttributeError: 'Read' object has no attribute 'sequence'.
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
AttributeError: 'Read' object has no attribute 'sequence'.

Same thing happens in python2 or python3 (khmer.__version__ 2.1.1). Unsure, but #645 may be relevant.

Motivation is I'd like to stream in a sequence file and immediately pass it off to pool workers for downstream analysis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant