Using multiple selections for a variable in one read

[hr87]

Is it possible to use multiple different selections for a single variable in one read? I have data that is stored 3D in space, plus 2 extra dimensions $(x, y, z, m, g)$, but I need to read it in as a 1D array in space $(v, m, g)$. One way to do this would be to loop over all spatial $v$, set the selection and call Get.

However, I am not sure, if it is possible to have multiple selections, or if the last selection call overwrites all selections for previous Get.

Another way would be to copy the variable, set different selections. Is that a possibility?

[pnorbert]

Yes, you can do

   var.SetSelection(s1);
   engine.Get(var, ptr1);
   var.SetSelection(s2);
   engine.Get(var, ptr2);
   ...
   engine.PerformGets();

However, the reader is not smart enough to combine work on multiple selections and it will read the necessary contiguous (probably the entire) dataset to carry out one selection again and again.

If performance is a concern, and if this is a big 5D array written by many processes, I would read the entire data block by block and make the transformation in memory manually.

[hr87]

Ok, I am trying to get the data with this Get version: https://adios2.readthedocs.io/en/v2.9.2/api_full/api_full.html#_CPPv4I0ENK6adios26Engine3GetEv8VariableI1TEPP1T and then re-organize when copying it over. However, I get seg faults when attempting this

        // get variable from adios                                                                                                                                                                                                                                                                                                                                                          
        auto variable = this->adios_io_.InquireVariable( "intensities" );                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                                            
        // check if found                                                                                                                                                                                                                                                                                                                                                                   
        if ( !variable ) {                                                                                                                                                                                                                                                                                                                                                                  
            // throw error                                                                                                                                                                                                                                                                                                                                                                  
            throw std::runtime_error(                                                                                                                                                                                                                                                                                                                                                       
                "Variable intensities not found in file!" );                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                                                                            
        } // end if variable                                                                                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                                                                                            
        // define memory space                                                                                                                                                                                                                                                                                                                                                              
        variable.SetMemorySpace( this->io_memory_space_ );                                                                                                                                                                                                                                                                                                                                  
        // set step selection for variable, read only 1 step                                                                                                                                                                                                                                                                                                                                
        variable.SetStepSelection( { this->step_, 1 } );                                                                                                                                                                                                                                                                                                                                    
        // set shape of data to read                                                                                                                                                                                                                                                                                                                                                        
        variable.SetSelection( { start_idx, count } );                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                            
        // pointer to data                                                                                                                                                                                                                                                                                                                                                                  
        const real_t *data = nullptr;                                                                                                                                                                                                                                                                                                                                                       
        // get data                                                                                                                                                                                                                                                                                                                                                                         
        this->adios_engine_.Get( variable, &data );                                                                                                                                                                                                                                                                                                                                         
        // perform reads 
        //this->adios_engine_.PerformGets();                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
        // check pointer                                                                                                                                                                                                                                                                                                                                                                    
        assert( data );                                                                                                                                                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                                                                                                                            
        // read in data from pointer                                                                                                                                                                                                                                                                                                                                                        
        _intensities.deserialize( _mesh, data ); 

where deserialize pretty much calculates an index and tries to copy the value over

/// calculate index in adios memory
const auto idx = ...
// copy over value
value(v, m, g) = data[idx]

I opened the file in random access mode. I get a seg fault for idx = 0, so it is not a problem with the index calculation. Any idea what I am doing wrong?

[hr87]

Here is the error from gdb

Thread 1 "catch-tests" received signal SIGSEGV, Segmentation fault.
(gdb) l
332
333             // check pointer
334             assert( data );
335
336             // read in data from pointer
337             _intensities.deserialize( _mesh, data );
338
339         } // end void read_intensities
340
341     protected:
(gdb) p data
$3 = (const real_t *) 0x3ff0000000000000
(gdb) p data[0]
Cannot access memory at address 0x3ff0000000000000

and here is the debug in the function deserialize

(gdb) l
611                 } // end for d
612
613                 // check index
614                 assert( idx >= 0 && idx <= static_cast<integer_t>( this->values_.size() ) );
615                 // write value to memory block
616                 value = _span[idx];
617             } );
618
619         } // end void serialize
620
(gdb) p _span
$4 = (const lreal_t *&) @0x7fffffff1cb8: 0x3ff0000000000000
(gdb) p idx
$5 = 0
(gdb) p _span[idx]
Cannot access memory at address 0x3ff0000000000000
(gdb) 

[eisenhauer]

Hi,
So, trying to sort what might be going on. First, this is C++ above, yes? Or Matlab? Might not matter, but clarification would probably be good. Regardless, I think there's some confusion. First lets talk about this bit of code:

        const real_t *data = nullptr;                                                                                                                                                                                                                                                                                                                                                       
        // get data                                                                                                                                                                                                                                                                                                                                                                         
        this->adios_engine_.Get( variable, &data ); 
        // perform reads 
        //this->adios_engine_.PerformGets(); 

Two points here. The first is that the Get() interface into which you pass a raw pointer does not allocate data for you. Instead, it assumes that you have allocated enough space to hold whatever you are reading. In this case, you're passing in the address of data, which is a stack-allocated variable, so when this Get() does happen, you're going to overwrite the stack and bad things will happen. If you want ADIOS to allocate memory for what you're reading, you should declare data as a "std:vector<real_t> data" so that ADIOS can resize that appropriately. Second, you've commended out the call to PerformGets(). By default the ADIOS Get() function operates in Deferred mode, so it doesn't do anything immediately but waits until PerformGets() (or EndStep() in streaming mode). So, you'll need the call to PerfromGets().

(It looks like the documentation you point too says that "the value is immediately available", but that should only be true if the value that is subject to the Get() is a single value, not an array. I don't know you're data, but it sure looks like you're trying to read an array there. That doc is also plastered with notes about "Preliminary, experimental API, may change soon. " that probably shouldn't be there. Once we get this sorted out, we can sort the documentation issues.)

[hr87]

Yes, this is C++, sorry for the confusion.

I pass an address to pointer, which is set to nullptr. From the documentation, it sounds like that pointer will be changed to point to ADIOS internal memory, similar to the Put methods that return a span. The value of the pointer changes. It reads like that ADIOS manages the memory, since it is internal buffers.

The documentation is not clear if that is only for a scalar or a range of data. Since the reverse Put operations can be used for a range of data, I assumed this one can be too.

I tried with and without PerformPuts, got the same pointer back, and same problem.

PS: I got it to work with pre-allocated memory and a different Get version.

[eisenhauer]

Unfortunately memory management between Put() and Get() are necessarily pretty dissimilar. There's nothing like a Span on the Get() side, mostly because you may request a selection that has to be built from multiple write blocks and never existed as a contiguous unit anywhere. So there's nothing on the read side where we give you pointers to ADIOS-internal memory. You must either provide us with pre-allocated memory, or give us an appropriately-typed std::vector (that you own) and ADIOS will set its size appropriately.

I'll be honest, without digging deeper, i'm a little puzzled as to how this is compiling. You're passing in a void** to Get(), and we have created methods where the second parameter to Get() is T* where T is a set of atomic types. Somehow C++ must be promoting void* to something, maybe a string? Maybe uintptr_t which can be converted to uint64_t, which would be valid, but not what you want. But somehow it's compiling, and then somehow the Get() is doing something, which it shouldn't be doing without PerformGets() unless variable maps to a single-value datum and not an array. I don't have the whole code, so I don't know if or how that might be happening, but I have to suspect something like that.

While I'm not entirely sure of how it's doing what it's doing, I am sure that it is not going to do what you intend because that's not the way ADIOS manages memory on the reader side. For an array, you must either pass in a std::vector (easier because ADIOS can resize it to whatever is needed), or determine the size of the buffer you need (perhaps using the variable.SelectionSize() method), allocate the memory and pass in a pointer to that memory.

[hr87]

I am passing double**, and the documentation has a method, that accepts that (the one linked above).

Anyways, thanks for the help. Appreciate it.

[eisenhauer]

Oh, and glad you got it working pre-allocated... If there are no more questions around this, you can close when you're ready.