[FEA] set ${pkg}_DIR for project dependencies to assist with building dependencies in a common build dir

[trxcllnt]

In node-rapids we're taking advantage of a centralized source and build cache for all our C++ dependencies. We have multiple CMakeLists.txt project files that depend on certain expensive-to-build RAPIDS libraries like libcudf, and we don't want to build those libraries multiple times.

CPM has a utility for telling it to centralize where it downloads dependency sources via CPM_SOURCE_CACHE, but it has no corresponding setting for centralizing dependency build dirs i.e. CPM_BINARY_CACHE.

We can trick CPM (everything else that uses FetchContent) into centralizing build dirs via the FETCHCONTENT_BASE_DIR variable.

However, we later run into issues when a built project's export targets go to look up its own dependencies via FindPackage(). The dependencies aren't installed, and they're not in a location FindPackage() knows to search.

And since FetchContent doesn't create directory names FindPackage recognizes as locations a package could be (i.e. thrust vs. thrust-build), we can't simply do list(APPEND CMAKE_PREFIX_PATH ${FETCHCONTENT_BASE_DIR}).

Our workaround for this is to set ${pkg}_DIR for a project's dependencies before calling CPMFindPackage(). We have this little CMake function to help us:

function(_set_package_dir_if_exists pkg dir)
  if (EXISTS "${CPM_BINARY_CACHE}/${dir}-build")
      set(${pkg}_DIR "${CPM_BINARY_CACHE}/${dir}-build" PARENT_SCOPE)
  endif()
endfunction()

[robertmaynard]

The primary issue holding this back is that this could break consuming projects when a cache is shared between multiple consumers.

For example consider a cache where dependency generates a shared library. Consumer A builds and links to version 1, Consumer B of the cache comes around and builds a new SHA1 version of dependency but with the same version number.
Running A will most likely now break :(

I think that for this to work safely it needs to be explicit on the consuming side to opt into the cache. The dependencies should generate / encode the location but shouldn't set it unless some requested flag is set by the consumer.

@trxcllnt Did I capture the issues correctly?

[trxcllnt]

Consumer B of the cache comes around and builds a new SHA1 version of dependency but with the same version number.

Should note for completeness -- this would happen even if the version numbers are different.

To support this, CPM (or maybe CMake's FetchContent?) would need to include the git hash in the path passed to ExternalProject_Add(BINARY_DIR). If two projects declare a dependency on separate hashes/versions of a project, they should be built into separate locations.

For example, here's the CMakeLists.txt automatically generated when CPMFindPackage(arrow) downloads and builds Arrow:

include(ExternalProject)
ExternalProject_Add(arrow-populate
                     "UPDATE_DISCONNECTED" "False" "GIT_REPOSITORY" "https://github.com/apache/arrow.git" "GIT_TAG" "apache-arrow-1.0.1" "GIT_SHALLOW" "TRUE"
                    SOURCE_DIR          "<root>/.cache/cpm/arrow/8e5cd63fd71d7269acd5d779a6767e97ede977f5"
                    BINARY_DIR          "<root>/.cache/build/Release/arrow-build"
                    CONFIGURE_COMMAND   ""
                    BUILD_COMMAND       ""
                    INSTALL_COMMAND     ""
                    TEST_COMMAND        ""
                    USES_TERMINAL_DOWNLOAD  YES
                    USES_TERMINAL_UPDATE    YES
)

In the above case, BINARY_DIR should probably include the git hash as the SOURCE_DIR does.