Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split parameter (and other) files into parts to facilitate embedding large files into shared library #85

Open
ellio167 opened this issue Nov 4, 2022 · 0 comments

Comments

@ellio167
Copy link
Member

ellio167 commented Nov 4, 2022

Currently parameter (metadata, etc.) files are processed through xxd to create C source code and then compiled into a binary .o file for linking into an item's shared library.

This fails for large files due to excessive compiler memory usage.
--- See below link ---
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html
https://thephd.dev/_vendor/future_cxx/papers/d1967.html
https://github.com/ThePhD/embed
--- xxxx ---
The above proposed standard is evolving and being accepted, but it is nowhere near ready for availability and wide use in the wild.

See this message on openkim-dev:
https://groups.google.com/d/msg/openkim-dev/aTjMuUxRNJY/VrqiPfFcCAAJ
where we discuss dropping xxd in favor of a native cmake implementation. In the end we implemented both. xxd is (much) faster and used if it exists. If not, a cmake implementation is used. -- This is independent of the time needed to compile the output source code.

--- some not-so-portable options for embedding files into binaries ---
https://unix.stackexchange.com/questions/494844/embedding-binary-data-into-an-executable-using-llvm-tools
--- xxx ---

To continue with this approach the kim-item shared-library format will need to be updated to understand that files are embedded in a series of chunks and the cmake processing of kim-items will need to be updated to split the files and handle all the data pointers appropriately.

This will be a significant effort, and we will end up with rather large shared library files (which might become an issue sooner than later--I don't know). My preference was to abandon embedded files as described in #59. However, #59 is a rather bigger (more involved) change than split.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant