Add robot-log-visualizer package, its dependencies and rob_sup_pure_python_ycm_ep_helper CMake function to add pure Python packages

[traversaro]

This PR adds the robot-log-visualizer package and its dependencies to the robotology-superbuild, if the option ROBOTOLOGY_USES_PYTHON and ROBOTOLOGY_ENABLE_DYNAMICS_FULL_DEPS are enabled.

As the robot-log-visualizer has many dependencies, for apt this dependencies were not added to the apt.txt that is installed by ./script/install_apt_dependencies.sh. Instead, we provided a new apt-python.txt that is installed by ./script/install_apt_python_dependencies.sh. Similarly, for what regard conda the dependencies are not the one listed in the main conda documentation, but we instead updated the documentation on the dependencies that are required by ROBOTOLOGY_USES_PYTHON in https://github.com/robotology/robotology-superbuild/blob/master/doc/cmake-options.md#python .

Furthermore, as robot-log-visualizer is a pure Python package, the PR also adds a rob_sup_pure_python_ycm_ep_helper CMake helper function to simply the process or writing Build<package>.cmake scripts for pure Python packages. As an example, this is the Buildobot-log-visualizer.cmake script:

include(RobSupPurePythonYCMEPHelper)

find_or_build_package(icub-models QUIET)
find_or_build_package(pyqtconsole QUIET)
find_or_build_package(meshcat-python QUIET)

rob_sup_pure_python_ycm_ep_helper(robot-log-visualizer
                                  REPOSITORY ami-iit/robot-log-visualizer.git
                                  DEPENDS meshcat-python
                                          pyqtconsole
                                          icub-models
                                  TAG main
                                  COMPONENT dynamics
                                  FOLDER src)

[traversaro]

I were not sure were those commands were running, so I did for test:

              CONFIGURE_COMMAND echo "CONFIGURE_COMMAND is executed in " && pwd
              BUILD_COMMAND echo "BUILD_COMMAND is executed in " && pwd
              INSTALL_COMMAND echo "INSTALL_COMMAND is executed in " && pwd)

and the result is:

-- Build files have been written to: /home/traversaro/robotology-superbuild/buildapt
[  0%] Skipping update step for 'robot-log-visualizer'
[  0%] Performing configure step for 'robot-log-visualizer'
CONFIGURE_COMMAND is executed in
/home/traversaro/robotology-superbuild/buildapt/src/robot-log-visualizer
[  0%] Performing build step for 'robot-log-visualizer'
BUILD_COMMAND is executed in
/home/traversaro/robotology-superbuild/buildapt/src/robot-log-visualizer
[  0%] Performing install step for 'robot-log-visualizer'
INSTALL_COMMAND is executed in
/home/traversaro/robotology-superbuild/buildapt/src/robot-log-visualizer
[ 33%] Completed 'robot-log-visualizer'
[100%] Built target robot-log-visualizer

We just need to understand how to get the source directory to pass to pip.

[traversaro]

We just need to understand how to get the source directory to pass to pip.

From https://github.com/robotology/ycm/blob/3971712471a684b486b808b255b8b3187e098202/build-modules/BuildGtkDatabox.cmake#L18 I guess it is <SOURCE_DIR>, see also https://cmake.org/cmake/help/latest/module/ExternalProject.html#explicit-step-management .

[traversaro]

Two problems:

pip install options and installation location

So, there are two variants that could work for us, but none that works out of the box.

Use of --target option

The one that we have now is:

INSTALL_COMMAND ${Python3_EXECUTABLE} -m pip install --no-deps --target=${YCM_EP_INSTALL_DIR}/${ROBSUB_PYTHON_INSTALL_DIR}

That (on Ubuntu/apt) installs files in:

• <install_prefix>/lib/python3.8/robot-log-visualizer
• <install_prefix>/lib/python3.8/bin

With this the Python package would be found without modifying the setup.sh, but the executable no.

Use of --prefix option

The alternative is:

INSTALL_COMMAND ${Python3_EXECUTABLE} -m pip install --no-deps --prefix=${YCM_EP_INSTALL_DIR}

That install files in (on Ubuntu/apt):

• <install_prefix>/lib/python3.8/robot-log-visualizer
• <install_prefix>/bin

With this variant, the executable could be find without modification of setup.sh, but the python no as the superbuild existing logic installs python packages in <install_prefix>/lib/python3.

I found no combination of options that permit to install both the executable and the python library exactly were we usually put them in the superbuild.
I need to think about it, but we can just choose an option and modify the setup.sh accordingly.

Dependencies

Ideally, I would prefer to install all the possible dependencies with apt and the remaining one with the superbuild, so we can just use the existing ProjectTags mechanism to lock them. @GiulioRomualdi do you have any idea on which dependencies are not available on apt. I guess the only missing one is meshcat-python, but I would need to check. The alternative is that we implement some locking mechanism also using pip, but that seems complex.

After a first check, the one that are either not available on Ubuntu 20.04 or not sufficiently recent are:

• meshcat-python (not available in Ubuntu)
• u-msgpack-python (available, but version in Ubuntu 20.04 is not recent enough for meshcat https://github.com/rdeits/meshcat-python/blob/c3a9ceaa2b82ba1146b174d901a63269a9b5432f/setup.py#L22)
• pyngrok (not available in Ubuntu)

Probably it is reasonable to just package them as well in the superbuild (perhaps with a simplified macro so that their Build<>.cmake file are super-simple) .

[traversaro]

apt deps: sudo apt install python3-pyqt5 python3-pyqt5.qtmultimedia python3-matplotlib python3-pyqt5.qtwebengine python3-pyqtconsole python3-qtpy python3-h5py python3-u-msgpack python3-tornado python3-zmq python3-ipython
non-package deps on apt: u-msgpack-python meshcat-python pyngrok pyqtconsole

[traversaro]

I decided to go for the --target option and modify the setup scripts to add to the PATH the required directory, it seems to be that this is the best option.

[traversaro]

• u-msgpack-python (available, but version in Ubuntu 20.04 is not recent enough for meshcat https://github.com/rdeits/meshcat-python/blob/c3a9ceaa2b82ba1146b174d901a63269a9b5432f/setup.py#L22)

The version installed by apt on ubuntu 20.04 seems ok.

• pyngrok (not available in Ubuntu)

Apparently this is not required?

[traversaro]

robot-log-visualizer works fine on both Linux (apt):

and Windows (conda):

There is an icon problem, but apparently that is a known problem that is currently being worked on (@GiulioRomualdi I do not think there is an issue, in case there is feel free to drop a link). Beside that, the PR is working. The only remain thing to check is to try to build some conda packages, I will trigger a job for doing that.

[traversaro]

@GiulioRomualdi I did the modifications for the conda packages, should we had the numix icons as a dependency?

[traversaro]

Conda packages generation worked fine, see https://github.com/robotology/robotology-superbuild/runs/6023302790 . I think the PR is ready for review. @GiulioRomualdi should we add the numix dependendencies? Which packages in particular?

[traversaro]

All conda jobs are failing with error:

2022-04-14T14:15:28.9231130Z [192/376] Performing configure step for 'pyqtconsole'
2022-04-14T14:15:28.9331620Z FAILED: src/pyqtconsole/CMakeFiles/YCMStamp/pyqtconsole-configure /Users/runner/work/robotology-superbuild/robotology-superbuild/build/src/pyqtconsole/CMakeFiles/YCMStamp/pyqtconsole-configure 
2022-04-14T14:15:28.9434040Z cd /Users/runner/work/robotology-superbuild/robotology-superbuild/build/src/pyqtconsole && /Users/runner/miniconda3/envs/test/bin/cmake -E env PKG_CONFIG_PATH=/Users/runner/work/robotology-superbuild/robotology-superbuild/build/install/lib/pkgconfig: && /Users/runner/miniconda3/envs/test/bin/cmake -E touch /Users/runner/work/robotology-superbuild/robotology-superbuild/build/src/pyqtconsole/CMakeFiles/YCMStamp/pyqtconsole-configure
2022-04-14T14:15:28.9534560Z cmake -E env: no command given

Probably some regression related to recent CMake version (or connected to robotology/ycm-cmake-modules#50) that creates problems if one passes "" as CONFIGURE_COMMAND or BUILD_COMMAND of ycm_ep_helper ?

As a workaround probably we can just specify a command that just exits with success an all operating systems, such as cmake --version .

[traversaro]

As a workaround probably we can just specify a command that just exits with success an all operating systems, such as cmake --version .

The workaround worked fine.

[traversaro]

Conda packages generation worked fine, see https://github.com/robotology/robotology-superbuild/runs/6023302790 . I think the PR is ready for review. @GiulioRomualdi should we add the numix dependendencies? Which packages in particular?

@GiulioRomualdi friendly ping

[GiulioRomualdi]

For the time being the visualizer do not depend on numix so we can avoid to add the dependency.

[traversaro]

For the time being the visualizer do not depend on numix so we can avoid to add the dependency.

Ack, I think that we can merge then. Not sure about the icon situation, but we can always debug once it has been merged.

[traversaro]

@GiulioRomualdi agreed to merge in person.