Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce levelization in libxrpl with CMake #5111

Open
wants to merge 15 commits into
base: develop
Choose a base branch
from

Conversation

thejohnfreeman
Copy link
Collaborator

@thejohnfreeman thejohnfreeman commented Aug 28, 2024

Adds two CMake functions:

  • add_module(library subdirectory): Declares an OBJECT "library" (a CMake abstraction for a collection of object files) with sources from the given subdirectory of the given library, representing a module. Isolates the module's headers by creating a subdirectory in the build directory, e.g. .build/tmp123, that contains just a symlink, e.g. .build/tmp123/basics, to the module's header directory, e.g. include/xrpl/basics, in the source directory, and putting .build/tmp123 (but not include/xrpl) on the include path of the module sources. This prevents the module sources from including headers not explicitly linked to the module in CMake with target_link_libraries.
  • target_link_modules(library scope modules...): Links the library target to each of the module targets, and removes their sources from its source list (so they are not compiled and linked twice).

Uses these functions to separate and explicitly link modules in libxrpl:

  • Level 01: beast
  • Level 02: basics
  • Level 03: json, crypto
  • Level 04: protocol
  • Level 05: resource, server

Define each "module" of libxrpl as a separate OBJECT library in CMake.
Link each one only to modules of a lower level.
Then link all of them into libxrpl itself.
Copy link

codecov bot commented Aug 28, 2024

Codecov Report

Attention: Patch coverage is 90.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (9d58f11) to head (ad16bbd).
Report is 3 commits behind head on develop.

Files with missing lines Patch % Lines
include/xrpl/basics/SHAMapHash.h 0.0% 2 Missing ⚠️
src/xrpld/app/tx/detail/Payment.cpp 88.9% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           develop   #5111     +/-   ##
=========================================
- Coverage     77.9%   77.9%   -0.0%     
=========================================
  Files          782     781      -1     
  Lines        66616   66618      +2     
  Branches      8161    8127     -34     
=========================================
- Hits         51902   51896      -6     
- Misses       14714   14722      +8     
Files with missing lines Coverage Δ
include/xrpl/basics/Number.h 100.0% <ø> (ø)
include/xrpl/basics/base_uint.h 96.8% <100.0%> (+0.1%) ⬆️
include/xrpl/basics/partitioned_unordered_map.h 99.2% <100.0%> (+<0.1%) ⬆️
include/xrpl/protocol/AmountConversions.h 87.7% <ø> (ø)
include/xrpl/protocol/FeeUnits.h 90.4% <ø> (ø)
include/xrpl/protocol/Fees.h 100.0% <ø> (ø)
include/xrpl/protocol/IOUAmount.h 100.0% <ø> (ø)
include/xrpl/protocol/LedgerHeader.h 100.0% <ø> (ø)
include/xrpl/protocol/MPTAmount.h 100.0% <100.0%> (ø)
include/xrpl/protocol/PayChan.h 100.0% <ø> (ø)
... and 36 more

... and 7 files with indirect coverage changes

Impacted file tree graph

@ximinez
Copy link
Collaborator

ximinez commented Aug 29, 2024

Not a review yet, but this is such a cool idea.

cmake/isolate_headers.cmake Outdated Show resolved Hide resolved
Comment on lines 125 to 126
PRIVATE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/src>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is probably not needed ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, upon some testing, this whole target_include_directories seem to be not needed.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not needed now that all of the subdirectories in libxrpl are modules. It was necessary as I migrated them one-by-one, and it could be necessary in the future if someone adds a non-module subdirectory. Could be nice if it "just works" without tampering with the CMake. What do you think?

Copy link
Collaborator

@Bronek Bronek Sep 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps comment out and put a comment above, to explain why this is not necessary and in what condition it might become necessary again ?

@Bronek
Copy link
Collaborator

Bronek commented Sep 20, 2024

I like how you moved extract to appropriate locations, but it seems that extract is function-wise the same as hash.

Given we already specialize both boost::hash (e.g. in IPEndpoint.h) and std::hash (e.g. in Book.h) I think that cleaning this up belongs in a different PR (also converting extract to hash), so maybe just take a note for future refactoring.

@Bronek
Copy link
Collaborator

Bronek commented Sep 23, 2024

I do wonder whether Builds/levelization is becoming superfluous with this change. Why should we keep it, if the ordering of modules (and header isolation) enforce it for us ?

Copy link
Collaborator

@Bronek Bronek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall very nice change. Some comments in the middle, unsure if they require changes in this PR or not.

@thejohnfreeman
Copy link
Collaborator Author

Builds/levelization is still used to diagnose cycles in xrpld. I want to remove it after we've eliminated all of the cycles.

@Bronek
Copy link
Collaborator

Bronek commented Sep 24, 2024

Builds/levelization is still used to diagnose cycles in xrpld. I want to remove it after we've eliminated all of the cycles.

Makes sense. All the more reason to make an effort moving parts of xrpld into libxrpl, but that's separate discussion and more than one PR.

@Bronek Bronek self-requested a review September 24, 2024 16:23
Copy link
Collaborator

@Bronek Bronek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement, thanks !

@ximinez
Copy link
Collaborator

ximinez commented Oct 24, 2024

Bad news right out the gate on Windows builds:

$ cmake build/cmake/msvc19.ON
-- Using Conan toolchain: C:/Dev/rippled/review1/build/conan.msvc19/build/generators/conan_toolchain.cmake
-- Conan toolchain: C++ Standard 20 with extensions OFF
-- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.19045.
[...]
-- Conan: Component target declared 'xxHash::xxhash'
-- Conan: Including build module from 'C:/.conan/4b4fe1f3/1/lib/cmake/protobuf/protobuf-generate.cmake'
-- Conan: Including build module from 'C:/.conan/4b4fe1f3/1/lib/cmake/protobuf/protobuf-module.cmake'
-- Conan: Including build module from 'C:/.conan/4b4fe1f3/1/lib/cmake/protobuf/protobuf-options.cmake'
CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.beast/xrpl/beast':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:69 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.basics/xrpl/basics':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:76 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.json/xrpl/json':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:80 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.crypto/xrpl/crypto':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:83 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.protocol/xrpl/protocol':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:87 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.resource/xrpl/resource':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:94 (add_module)
  CMakeLists.txt:123 (include)


CMake Error at cmake/isolate_headers.cmake:44 (file):
  file failed to create symbolic link
  'C:/Dev/rippled/review1/build/cmake/msvc19.ON/modules/xrpl.libxrpl.server/xrpl/server':
  A required privilege is not held by the client.

Call Stack (most recent call first):
  cmake/add_module.cmake:25 (isolate_headers)
  cmake/RippledCore.cmake:97 (add_module)
  CMakeLists.txt:123 (include)


-- xrpl.libxrpl.basics with 14 sources took 14 sources from xrpl.libxrpl
-- xrpl.libxrpl.beast with 15 sources took 15 sources from xrpl.libxrpl
-- xrpl.libxrpl.crypto with 3 sources took 3 sources from xrpl.libxrpl
-- xrpl.libxrpl.json with 9 sources took 9 sources from xrpl.libxrpl
-- xrpl.libxrpl.protocol with 51 sources took 51 sources from xrpl.libxrpl
-- xrpl.libxrpl.resource with 4 sources took 4 sources from xrpl.libxrpl
-- xrpl.libxrpl.server with 2 sources took 2 sources from xrpl.libxrpl
-- Configuring incomplete, errors occurred!

@thejohnfreeman
Copy link
Collaborator Author

@ximinez ok, try again.

@ximinez
Copy link
Collaborator

ximinez commented Oct 30, 2024

It's building successfully now. Neat workaround. Super annoying that this is an issue, though.

I remember there were some file(CREATE_LINK calls elsewhere that were simply disabled under Windows. Would it be worth coming back to those to use this same technique?

Otherwise, I'll come back to the rest of this review as soon as I get some other priorities knocked out.

Copy link
Collaborator

@godexsoft godexsoft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good 👍 Just one question - not a blocker by any means.

# add_module(parent a)
# add_module(parent b)
# target_link_libraries(project.libparent.b PUBLIC project.libparent.a)
function(add_module parent name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be even nicer if you could optionally pass the libraries to link the target with right here too? Is it possible to add the PUBLIC lib1 lib2 through args somehow?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find the current call pattern nice enough. It reads like conventional CMake to me. If someone wants to add the function you're talking about, I think that's fine, but I'd rather keep this function just doing the one thing.

@ximinez
Copy link
Collaborator

ximinez commented Nov 15, 2024

@thejohnfreeman I see this has two approvals. Since I've verified that it builds under Windows, it's up to you if you want to wait for mine, too, or go ahead and add the "Passed" label and provide a commit message. I'll preemptively mark it as "Blocked" now to be sure we wait until after the 2.3.0 release before merging.

@thejohnfreeman
Copy link
Collaborator Author

@ximinez I would still appreciate your review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants