Create branch that uses flow_from_directory to avoid preloading all images into RAM

[stratospark]

This will hopefully make this project more feasible to run on a wider range of systems. I have extended the multiprocessing ImageDataGenerator to also load and processes images from disk in parallel: https://github.com/stratospark/keras-multiprocess-image-data-generator. It would be interesting to see how long it takes to train the classifier on a a system with less RAM than the total size of the food-101 dataset.

[olgabrinza]

Hello, @stratospark !
I have an issue using flow_from_directory option to create a generator. When i'm tying to fit my model with the fit_generator(like this) model.fit_generator( train_generator, steps_per_epoch=samples_per_epoch, epochs=50, validation_data=validation_generator, callbacks=None, validation_steps=validation_steps)

i have theTypeError: process() missing 1 required positional argument : 'rng'

The fool error looks likee:

`TypeError Traceback (most recent call last)
in ()
5 validation_data=validation_generator,
6 callbacks=None,
----> 7 validation_steps=validation_steps)

/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
89 warnings.warn('Update your ' + object_name + 90 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper

/usr/local/lib/python3.5/dist-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
2190 batch_index = 0
2191 while steps_done < steps_per_epoch:
-> 2192 generator_output = next(output_generator)
2193
2194 if not hasattr(generator_output, 'len'):

/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py in get(self)
791 success, value = self.queue.get()
792 if not success:
--> 793 six.reraise(value.class, value, value.traceback)

~/.local/lib/python3.5/site-packages/six.py in reraise(tp, value, tb)
691 if value.traceback is not tb:
692 raise value.with_traceback(tb)
--> 693 raise value
694 finally:
695 value = None

/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py in _data_generator_task(self)
656 # => Serialize calls to
657 # infinite iterator/generator's next() function
--> 658 generator_output = next(self._generator)
659 self.queue.put((True, generator_output))
660 else:

~/photo_data/tools/image_gen_extended.py in next(self, *args, **kwargs)
664
665 def next(self, *args, **kwargs):
--> 666 return self.next(*args, **kwargs)
667
668 def process_image_worker(tup):

~/photo_data/tools/image_gen_extended.py in next(self)
903 if x.ndim == 2:
904 x = np.expand_dims(x, axis=0)
--> 905 x = self.image_data_generator.process(x)
906 if i == 0:
907 batch_x = np.zeros((current_batch_size,) + x.shape)

TypeError: process() missing 1 required positional argument: 'rng'`

Thank you in advance for your help!

[fvisconti]

Hello, @stratospark !
I have an issue using flow_from_directory option to create a generator. When i'm tying to fit my model with the fit_generator(like this) model.fit_generator( train_generator, steps_per_epoch=samples_per_epoch, epochs=50, validation_data=validation_generator, callbacks=None, validation_steps=validation_steps)

i have theTypeError: process() missing 1 required positional argument : 'rng'

Hi, I had the same issue.
rng in image_gen_extended.py is just a random number generator: if you just set globally:

rng = np.random

everything should work.