flexmap allocate huge amount of physical memory

[vitstradal]

PDL::IO::FlexRaw::flexmap claims to work with mmap'ed file so can handle data bigger than physical memory. However, during initialization before assigning mmaped area, it creates empty zeroes piddle witch final dimensions, and it allocates (and then immediately deallocate) size of data. In code it is here.

This is my naive fix: here.

Following reproducer takes parameter memory size in MiB, and tryes two dimensional piddle of such size. If you put there number bigger than your free physical memory, it will fail on Out of memory or by OOM killer and did not reach done message.

#!/usr/bin/env perl                                                             
use strict;                                                                     
use warnings;                                                                   
use feature 'say';                                                              
use PDL::IO::FlexRaw;                                                           
my $size = shift(@ARGV) or die "usage: $0 MEM_SIZE_IN_MiB\n";                   
my $cols = 16;                                                                  
my $rows = $size * 1024 * 1024 / 8 / $cols;                                     
say "Creating piddle ${rows}x$cols size about ${size}MiB ";                     
my $mapflex = mapflex(                                                          
        'data.dat',                                                             
        [ { Type => 'double', NDims => 2, Dims => [$cols,$rows] } ],            
        { Creat => 1, Trunc   => 1, ReadOnly => 0, }                            
);                                                                              
say 'done';                                                                     

[mohawk2]

Thanks for the report, which I confirm shows a real problem! There's a slight issue with your immediate fix, which makes the tests fail, which is revealed by changing the last line of your script to:

say "done: size=$size mapflex.info=", $mapflex->info;

which then prints:

Creating piddle 40960000x16 size about 5000MiB 
done: size=5000 mapflex.info=PDL: Double D [42]

A great technique for testing just one bit of the vast repo, that I use all the time including here, is:

make core && (cd IO/FlexRaw/; make test)

[vitstradal]

Sorry for my bad fix, it has more problems than it solves. I assumed flexmap can be used read-and-write, but it turned out that it is read-only.

[mohawk2]

Your assumption was right (it's mapflex btw)! It's a minor gremlin, which I'm just fixing up now.

[vitstradal]

Wow. If you can make mapflex writable, it would help me a lot!
Honestly I do not know exactly how to fix it without breaking tests. They are yelling Trying to touch data of an untouchable (mmapped?) pdl (t/flexraw.t line 93).

[mohawk2]

Sorry, are you saying that mapflex doesn't make ndarrays that can be modified in a persistent way (or at all)? If so, could you put here some code you think should work? It will help me copy-paste into new tests :-)

[mohawk2]

To try to be clear: mapflex very much does work to write data if you mutate the ndarray that's mapped. The tests say in comments that this behaviour doesn't work, but that isn't right; the tests are run and work. Please try it, albeit with small data only until I get this fixed.

I'm working towards a slightly different model for mapflex, where several ndarrays are pointing into different parts of the same Perl SV; it's not pre-allocating data, but FlexRaw does an alignment check and that seems to be trying to copy the whole data block into RAM, which obviously crashes. Not sure why yet. Still working on it.

In the meantime, mapfraw seems to return instantly, even with huge data, and work great. If your application allows for it (only one ndarray per file), can you try it?

[mohawk2]

In fact, I've just realised why mapflex was behaving as it does: to check for memory-alignment problems, it accesses the first element and signal-handles for SIGBUS. However, using efficient programming techniques (!), to access the first element, it uses $pdl->flat->at(0).

The issue arises because flat is an alias for clump(-1), and clump (at least until I figure out how to make it not need to do this, which is conceptually tricky), has to physicalise (i.e. allocate) its output. That means allocating the same size again of memory. No big deal for a few hundred KB, but quite a problem for a few GiB if you don't mean to do it.

[mohawk2]

The above commit fixes this problem as far as your script seems to show, while still passing the tests.

@vitstradal Please could you try it, and if you agree, could you close the issue? Please make sure you're happy that ndarrays can also write.

[vitstradal]

Thank you very much @mohawk2, it working very nice. Thank you for quick fix very much.

Sorry about the write/readonly note. I create wrong impression from one of my many experiments, and got confused. Every think is OK.

[mohawk2]

That's quite alright! As you might tell from the various commits it took to fix this, and the obscure things that were involved in causing it, it was a complicated situation, so it's understandable to be confused - I was :-)