Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extended values example to show some current bugs with string processing #4050

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pnorbert
Copy link
Contributor

LocalString is a string LocalValue written by every process. It shows up as 1D array of strings that can be read back easily into a vector in C++.
However, it shows problems with the two possible block-by-block access, through metadata (AllStepsBlocksInfo), or through SetBlockSelection. One of them is always retrieving the first block.

GlobalStringEveryoneWrites is another string variable, declared as global value, yet written by every process. These can only be accessed by block-by-block methods, and again, one of them is wrong. Which one is wrong, depends on using BP4 or BP5.

@pnorbert
Copy link
Contributor Author

pnorbert commented Feb 27, 2024

With BP5. bpls works as expected:

adios@LAP131864:~/ADIOS2/build.debug$ ./bin/bpls -l values.bp -d LocalString
  string    LocalString                 {4}
    (0)    "cdefghijk" "lmno" "mnopq" "gh"

adios@LAP131864:~/ADIOS2/build.debug$ ./bin/bpls -l values.bp -d LocalString -D
  string    LocalString                 {4}
        step 0:
          block 0: [0:0] = "(null)" / "(null)"
    (0)    "cdefghijk"
          block 1: [1:1] = "(null)" / "(null)"
    (0)    "lmno"
          block 2: [2:2] = "(null)" / "(null)"
    (0)    "mnopq"
          block 3: [3:3] = "(null)" / "(null)"
    (0)    "gh"
adios@LAP131864:~/ADIOS2/build.debug$ ./bin/bpls -l values.bp -d GlobalStringEveryoneWrites
  string    GlobalStringEveryoneWrites  scalar = "cdefghijk"
"cdefghijk"

adios@LAP131864:~/ADIOS2/build.debug$ ./bin/bpls -l values.bp -d GlobalStringEveryoneWrites -D
  string    GlobalStringEveryoneWrites  scalar = "cdefghijk"
        step 0: 4 instances available
               "cdefghijk" "lmno" "mnopq" "gh"

Relevant output of adios2_basics_valuesRead (and adios2_basics_valuesReadRandom).
There are 2 issues, reading each block of a string LocalValue using AllStepsBlocksInfo[step][block].Value, and
reading each block of a string GlobalValue using SetBlockSelection.

LocalString:
    Read as 1D array into std::vector<std::string>:
        block 0 = cdefghijk
        block 1 = lmno
        block 2 = mnopq
        block 3 = gh
    Accessed block-by-block from metadata directly:
        block 0 =
        block 1 =
        block 2 =
        block 3 =
    Read block-by-block using SetBlockSelection:
        block 0 = cdefghijk
        block 1 = lmno
        block 2 = mnopq
        block 3 = gh
GlobalStringEveryoneWrites:
    Accessed block-by-block from metadata directly:
        block 0 = cdefghijk
        block 1 = lmno
        block 2 = mnopq
        block 3 = gh
    Read block-by-block using SetBlockSelection:
        block 0 = cdefghijk
        block 1 = cdefghijk
        block 2 = cdefghijk
        block 3 = cdefghijk

@pnorbert
Copy link
Contributor Author

With BP4, bpls output is identical to BP5.
There is only 1 issue with reading all blocks of a global value using SetBlockSelection.
Relevant output of adios2_basics_valuesRead (and adios2_basics_valuesReadRandom):

LocalString:
    Read as 1D array into std::vector<std::string>:
        block 0 = cdefghijk
        block 1 = lmno
        block 2 = mnopq
        block 3 = gh
    Accessed block-by-block from metadata directly:
        block 0 = cdefghijk
        block 1 = lmno
        block 2 = mnopq
        block 3 = gh
    Read block-by-block using SetBlockSelection:
        block 0 = cdefghijk
        block 1 = lmno
        block 2 = mnopq
        block 3 = gh
GlobalStringEveryoneWrites:
    Accessed block-by-block from metadata directly:
        block 0 = cdefghijk
        block 1 = lmno
        block 2 = mnopq
        block 3 = gh
    Read block-by-block using SetBlockSelection:
        block 0 = cdefghijk
        block 1 = cdefghijk
        block 2 = cdefghijk
        block 3 = cdefghijk

@pnorbert
Copy link
Contributor Author

@vicentebolea Could we have adios2::core target available during the adios build too? External codes have to use adios2::core when trying to use adios2::helper functions. During the adios build, adios2_core is available as a target but then that is not available in an external build.

@vicentebolea
Copy link
Collaborator

@pnorbert do you mean if we can export the adios2::core target, so projects doing find_package(adios2) can have that target ?

@pnorbert
Copy link
Contributor Author

The opposite. Projects can use adios2::core once they find_package(adios2). They cannot use adios2_core as target.

However, when building adios itself, the examples have no access to adios2::core yet, but they can use adios2_core for some reason.

More on that, I see cmake files having
if(NOT TARGET adios2::core)
and
if(NOT TARGET adios2_core)
in various places
e.g.

examples/basics/queryWorker/CMakeLists.txt:if(NOT TARGET adios2_core)
cmake/adios2-config-common.cmake.in:if(NOT TARGET adios2::core)

I don't know what the latter is trying to achieve.

@pnorbert
Copy link
Contributor Author

Actually, it would be simpler if the adios2::cxx11 adios2::cxx11_mpi targets would already add libadios2_core.so to the linker line.

We have been installing all the headers to include/adios2 for a while to give access to plugins (and users) to use all the internal adios functions. But then we need to go through this extra step to be able to link the code.

@vicentebolea
Copy link
Collaborator

Actually, it would be simpler if the adios2::cxx11 adios2::cxx11_mpi targets would already add libadios2_core.so to the linker line.

This already is the case adios2::core is its private dependency.

@vicentebolea
Copy link
Collaborator

More on that, I see cmake files having
if(NOT TARGET adios2::core)
and
if(NOT TARGET adios2_core)
in various places
e.g.

This was needed for having the tests being buildable in our build tree and also being able to build as standalone projects. (So users can refer to these cmakelists.txt when trying adios)

@vicentebolea
Copy link
Collaborator

@vicentebolea Could we have adios2::core target available during the adios build too? External codes have to use adios2::core when trying to use adios2::helper functions. During the adios build, adios2_core is available as a target but then that is not available in an external build.

Now I understand the question, Well we could but this is a common pattern in cmake projects, it makes a clear divison of internal and external targets, this will be an expensive change. Why do you need this?

…y every proces. It shows up as 1D array of strings that can be read back easily into a vector<string> in C++. It also shows problems with the two possible block-by-block access, through metadata (AllStepsBlocksInfo), or through SetBlockSelection. One of them is always retrieving the first block. GlobalStringEveryoneWrites is another string variable, declared as global value, yet written by every process. These can only be accessed by block-by-block methods, and again, one of them is wrong. Which one is wrong, depends on using BP4 or BP5.

use adios2::core in external build mode, use adios2_core in internal build mode to get access to adios2::helper functions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants