Proposal: Python API

[JulianOrteil]

@mvp I'm a user of uhubctl for a couple of projects I work on that require USB device integration with this project managing device states. I have to commend you and the other contributors for the work done here. Phenomenal.

Anyways, all of these projects use Python as the core language as it can cover really every aspect I need. That being said, using uhubctl can sometimes be a challenge for a couple of reasons:

1. Non-native support from Python. To use uhubctl, I have to open a dedicated subprocess to use your CLI and then use some pretty advanced regex to parse the output. As a side-effect, this is a security hole as well.
2. Lacking JSON output option #124 results in pretty harsh parsing requirements on us users.

These issues are pretty obvious, especially 1, so I'd like to propose a Python wrapper library for uhubctl. I want to fully disclose I have little experience with C and creating Python bindings for C (most of my C-like experience is Arduino), but considering the projects that support Python through C bindings (OpenCV, NumPy, TensorFlow, etc.), doing this is well within the realm of possibility.

I'll have to experiment with this to find the optimal approach, but I thought I'd get your opinion on it.

Hope to hear back,
Jules

[mvp]

There are few ways to address this:

1. Parse uhubctl output in python wrapper. I don't think it's necessarily difficult or security sensitive. I believe it is possible to do it reliably.
2. Implement -j (--json) option to support json output, then have python wrapper to call uhubctl appropriately. Parsing will be slightly easier, but will use json module.
3. Reimplement uhubctl completely in python. Will require pyusb - not a simple requirement because pyusb is not installed by default. Also, some constructs used by uhubctl are not trivial (and maybe impossible) to implement in python.

I believe first option is much easier overall. That said, I would not mind implementing -j, which will need changing command line parsing and output quite extensively (the only reason it was not done just yet).

[JulianOrteil]

Apologies, I did not put that security comment in the correct spot. Opening and running subprocesses through Python are a security threat, not the parsing of output.

I agree that 3 is likely not possible--I wasn't even considering the use of pyusb because I wanted to avoid having to change much of your current source code. Like how pre-commit has a mypy-mirror project that wraps around mypy to make it compatible with pre-commit.

I'd also like to see how feasible it'd be to call uhubctl natively. This would require some changes to your source code, but nothing notably difficult (likely returning output instead of printing it as you are now).

[JulianOrteil]

After some experimentation on my end, I have determined that running uhubctl natively from python is possible, but impossible with the current design of the program. I was able to successfully compile uhubctl with Python wrappers and got it to run via main() from Python a few times; however, I occasionally ran into a few issues like the notoriously helpful "segmentation fault (core dumped)".

If we decide to follow this approach, I may need some help with some C-related details to do it right. Regardless, this approach is the most intensive as it would likely require exposing custom functions in uhubctl to make it compatible as an API, not just a standalone program. What these would look like or their requirements are unknowns but can be determined as development progresses.

Overall, I still favor this approach (albeit to a lesser degree lol) as it seems to be the most "holistic". I dislike using subprocesses, especially in a situation like this, as I view them to be "hacky". However, I have no issue using them if the alternative is a truckload of work.

Jules

[mvp]

I understand that you don't like to spawn external process.
That said, using python wrapper seems like really bad idea in this particular case, because it would want to pull full libusb library in.

In terms of portability, spawning external process is safest: you can write python script which will not pull any module dependencies (except for import subprocess, which is standard anyway), and will only require that uhubctl executable is present on the PATH - thats all you need. And if uhubctl is not available in PATH, you can complain back to the user asking them to install missing tool. Any other approach is much more fragile and demanding.

Note that all projects in this list: https://github.com/mvp/uhubctl#notable-projects-using-uhubctl are using uhubctl as external process, and it seems to work for everyone just fine.

Also, even if we implement -j, that will make parsing output quite a bit easier, but it will not remove a need for external process.

[JulianOrteil]

Sure, why don't we move forward with running this through subprocesses then. Are you leaning towards implementing -j then, or do you want me to move forward with creating a custom parser?

[mvp]

Implementing -j will take time and considerable effort. In meantime, I recommend making custom parser, I can help you to write python code for such parser. I guess it should be python class which supports all functionality that uhubctl offers - various filters (-p, -l, -L, -n, -s), options (-f, -e, -N, -r, -R, -w, -e, -d) and actions (-a).

[JulianOrteil]

That would be most beneficial, yes. You know the output of this project more than anyone.

Because I don't see this library benefitting from maintaining any object states, I believe structuring this similar to numpy will likely be the best approach (mostly global functions implementing standalone code).

So, for example:

import uhubctl

# Examples for commonly-implemented functions
hub_count = uhubctl.get_hub_count()
compatible_hubs = uhubctl.get_compatible_hubs()

# Example of the main implementation function
uhubctl.exec(action=..., location=..., level=..., vendor=..., ports=...)
...