Pickling error on creating generators

[leminhlong-pixta]

Thank you for making this great tool. Currently I am trying to use your code in my own project.

I have a question regarding multiprocessing. For some reason, train and test generators do not work on Jupyter Notebook:

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-25-d9d404f23400> in <module>()
      3 test_datagen.set_pipeline([T.random_transform, T.random_crop, T.preprocess_input])
      4 test_generator = test_datagen.flow(X_test, y_test_cat, batch_size=64, seed=11, pool=pool)
----> 5 for x in test_generator:
      6     print (x)

/home/a161115/dev/food-101-keras/tools/image_gen_extended.pyc in next(self)
    727             index_array, current_index, current_batch_size = next(self.index_generator)
    728         # The transformation of images is not under thread lock so it can be done in parallel
--> 729         result = self.pool.map(process_image_worker, ((self.image_data_generator.process, self.X[j], self.rngs[i%self.batch_size]) for i, j in enumerate(index_array)))
    730         batch_x = np.array(result)
    731 

/usr/lib/python2.7/multiprocessing/pool.pyc in map(self, func, iterable, chunksize)
    249         '''
    250         assert self._state == RUN
--> 251         return self.map_async(func, iterable, chunksize).get()
    252 
    253     def imap(self, func, iterable, chunksize=1):

/usr/lib/python2.7/multiprocessing/pool.pyc in get(self, timeout)
    556             return self._value
    557         else:
--> 558             raise self._value
    559 
    560     def _set(self, i, obj):

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

You noted that the kernel needs to be restarted everytime code is stopped manually. Is there currently a solution for this problem?

[ayox]

@leminhlong-pixta have you found a solution for it ?