Help with custom realtime ScenarIO backend

[HorvathDawson]

I am starting to work on a scenario realtime back-end for my custom robot. Before I drive to deep into it with minimal cpp knowledge is there any knowledge transfer that would be useful for me?

I only need it to support force controller mode. I already have my hardware control loop that does PID control (SO I think set_force_target should translate one to one for my force target on hardware? assuming PID constants are matched?).

How would you recommend I format the project? Should I fork gym-ignition/scenarIO and work off of that? How would I make this integrate well into gym-ignition?

Thank you for any advice :)
Dawson

Also thank you so much for the amazing work you ave done on this project !!

[diegoferigo]

In general you should be able to do everything from downstream, no need to operate directly upstream. Here below a possible task breakdown:

1. Develop a new real-time runtime that calls the environment at a desired rate and enforces the agent period to be followed. You can start from the placeholder realtime_runtime.py, copy it in your project, and play a bit with time.sleep() and similar.
2. Pick a language to implement the new ScenarIO backend:
   • If you can control your robot from Python, you can create new classes inheriting from scenario.bindings.core.World|Model|Joint|Link that call your Python module to get and set real robot data.
   • If you can control your robot from C++, you can create new classes inheriting from scenario::core::World|Model|Joint|Link that call your C++ APIs to get and set real robot data. Then, you should also create the bindings to expose this backend to Python, you can copy what we do upstream with CMake for the Gazebo backend.

Note that, while the ScenarIO APIs are quite a lot, you can implement only the methods that you need, leaving the others either as dummy or not implemented.

Once your backend is ready, if your task is developed in such a way that has no *.to_gazebo calls, you can replace the entrypoint (e.g. this one for the demo environments) specifying the real-time runtime instead of the gazebo runtime. The task should work also on the new real-time backend without changing any logic.

If you have randomizers that work for the gazebo back-end, most probably you have to develop new ones of the real-time back-end.

[HorvathDawson]

Reply to old thread

This approach forces you to think of a proper entrypoint from where to get all the ScenarIO objects. You have two choices:

I was planning (am currently) to use the world class as my entry point. Is there any reason I would not want to do that? How would using the world as a entry point work with the swig bindings? Would there be issues with that?

As you can see here, I was passing to this class all the conf files needed to create a Model from which afterwards I could communicate with the real robot.

So far the way I have been building the backend is much less general. Since I only plan to interface with a specific robot that wont be changing I haven't put any effort into using a config file etc to define the robot. Instead the Implementation details will be exposed from our Robot SDK.

This is the result of a properinstall_basic_package_files configuration of ScenarIO Core. As written above, you don't need do to it unless you want to distribute your C++ code.

When you say distribute do you mean with pypi? I was hoping to eventually build our package as a pip installable package. However, that is not a concern at the moment so I will ignore this for now and take a look at it again in the future. Makes the CMake much simpler which I enjoy haha,

Reading these line, however, I noticed that you import the ScenarIO targets from the Python folder. This is ok, although if you are a developer that use C++ APIs, I would warmly recommend to switch to the Developer installation, that installs the entire C++ project outside the Python folder. Since Python is not an initial concern, it would provide a cleaner setup. In a Developer installation, if you install in the default system folder (you need root permissions), CMake is able to find the targets without the need to specify CMAKE_PREFIX_PATH

I have a few questions about this. I have been using the python install for two reasons; Firstly I hope to make this package work as its own pip install that integrates into the python install of ignition gym. Second, I tried to install the developer version of ignition gym and ran into a bunch of issues especially with the IDynTree package. I spent some time trying to debug this but ultimately didn't want to spend to long forcing it to work. I could try to set it up again on a new user.

Quick note: if you remain in the C++ domain, I think everything is already in place here upstream. Instead, for extending the new backend with Python bindings, I suspect we need to start distributing somehow the core.i SWIG file. I have to think how to do it properly, initially it can be copied and pasted in downstream projects if needed.

Yes I have noticed that I will need the core.i file. Currently I have it copied into my downstream repo. It would be nice to have it exposed upstream though.

Questions on Bindings

I am at the point where I want to start trying to install my backend as a plugin to the existing bindings. I am slightly unsure what I need to change from the original CMake code you wrote for the original scenarIO project bindings.

1. Would I be able to to replace all of this With add_subdirectory(my_subpackage)?
2. Is there anything special I would have to do for CMake to find ScenarioSwig here when I replace gazebo with my downstream package?
3. Is there anything I would need to do for this to work with a virtual environment? I would rather not have to test installing the bindings on my main environment. Even more general did you find an easy way to test installing bindings?
4. Is there anything other than the .i interface file and its corresponding CMake files that you think I would need to make to build the bindings? It doesn't seem like I need to make any .init file or other file for the bindings.

[HorvathDawson]

I am getting a error with the including of core.i. This is what I have currenty,

It doesnt seem like you do anything special to have swig find the components in the build tree so I am not sure what the issue is. My code is in this branch if you do want to take a look.

[HorvathDawson]

I was able to get swig to find the headers by linking the libraries like this,

target_link_libraries(${scenario_swig_name} PUBLIC ScenarioCore)

target_link_libraries(${scenario_swig_name} PUBLIC ScenarioMonopod)

However now I am getting some weird static cast issue with the class pointers,

invalid static_cast from type ‘SwigValueWrapper<std::vector<std::shared_ptrscenario::monopod::Model, std::allocator<std::shared_ptrscenario::monopod::Model > > >’ to type ‘const std::vector<std::shared_ptrscenario::core::Model >&’

no match for ‘operator=’ (operand types are ‘SwigValueWrapper<std::vector<std::shared_ptrscenario::monopod::Model, std::allocator<std::shared_ptrscenario::monopod::Model > > >’ and ‘std::vector<std::shared_ptrscenario::core::Model >’)

no type named ‘type’ in ‘struct std::enable_if<false, void>’

[diegoferigo]

Reply to old thread

This approach forces you to think of a proper entrypoint from where to get all the ScenarIO objects. You have two choices:

I was planning (am currently) to use the world class as my entry point. Is there any reason I would not want to do that? How would using the world as a entry point work with the swig bindings? Would there be issues with that?

This looks good, I just wanted to point out two different alternatives. In the Gazebo ScenarIO backend, all the World / Model / Link / Classes are stateless, meaning that different objects created from the same simulation behave in the same way. This approach requires another class (GazeboSimulator) used to initialize these objects. I don't think this is a necessary feature in your case, I think you can proceed with World as entrypoint.

As you can see here, I was passing to this class all the conf files needed to create a Model from which afterwards I could communicate with the real robot.

So far the way I have been building the backend is much less general. Since I only plan to interface with a specific robot that wont be changing I haven't put any effort into using a config file etc to define the robot. Instead the Implementation details will be exposed from our Robot SDK.

Ok, this is up to you. Also on real robot, often a conf file is a good idea. Often there are multiple parameters accepted that otherwise should have to be hardcoded in the sources, but it could not be your case.

Reading these line, however, I noticed that you import the ScenarIO targets from the Python folder. This is ok, although if you are a developer that use C++ APIs, I would warmly recommend to switch to the Developer installation, that installs the entire C++ project outside the Python folder. Since Python is not an initial concern, it would provide a cleaner setup. In a Developer installation, if you install in the default system folder (you need root permissions), CMake is able to find the targets without the need to specify CMAKE_PREFIX_PATH

I have a few questions about this. I have been using the python install for two reasons; Firstly I hope to make this package work as its own pip install that integrates into the python install of ignition gym. Second, I tried to install the developer version of ignition gym and ran into a bunch of issues especially with the IDynTree package. I spent some time trying to debug this but ultimately didn't want to spend to long forcing it to work. I could try to set it up again on a new user.

The two setups are mostly the same (User and Developer). What's different is only where the headers and libraries of ScenarIO are installed. In order to let CMake find them, if they are not installed in one of the default folders, you have to configure manually your environment exporting CMAKE_PREFIX_PATH. You do this in your CMake, when instead it should be done by hand outside since the location could vary from system to system.

I was able to get swig to find the headers by linking the libraries like this,

target_link_libraries(${scenario_swig_name} PUBLIC ScenarioCore)

target_link_libraries(${scenario_swig_name} PUBLIC ScenarioMonopod)

Yes, you need to link ScenarioCore against your bindings, it will add to your compiler command line the include path of the missing headers and link against the component libraries.

However now I am getting some weird static cast issue with the class pointers,

invalid static_cast from type ‘SwigValueWrapper<std::vector<std::shared_ptrscenario::monopod::Model, std::allocator<std::shared_ptrscenario::monopod::Model > > >’ to type ‘const std::vector<std::shared_ptrscenario::core::Model >&’

no match for ‘operator=’ (operand types are ‘SwigValueWrapper<std::vector<std::shared_ptrscenario::monopod::Model, std::allocator<std::shared_ptrscenario::monopod::Model > > >’ and ‘std::vector<std::shared_ptrscenario::core::Model >’)

no type named ‘type’ in ‘struct std::enable_if<false, void>’

Oh wow, SWIG problems are always funny -.- I have no idea what's going on here, at least for what concern the SwigValueWrapper. For the std::enable_if, are you sure you are not using this in a public header (included in your .i file)? Swig does not support modern C++ features.

[HorvathDawson]

I finally fixed the shared_ptr bug after way more hours than I would like to admit.

I now have bindings for my cpp backend that compile downstream :)

I plan to package these bindings as site-lib python package. I am a little unsure how I should do this. The issue is that it seems the only way to do this is to modify the existing scenario site-lib package to include my backend. this means I would have to install my bindings files, cpp headers, cmake files for my backend, and update the parent ScenarioConfig.cmake file to include my backends target. I would also have to update the init to import my swig bindings.

I tested this by manually moving my files and it seemed to work. However, I am not sure how I would do this all in cmake. Also this does not feel like it would be very good practice modifying the existing Scenario package in the sitelib directly like this.

I thought about making a "scenario_rt" package that is installed. The issue is I would have to modify my task classes to instead import this package. This is a big issue since I dont want the realtime backend to be coupled to the environments for cloud training.

I would like this to be as simple as possible to install. Do you have any ideas?