support for UDP ublox update protoco

[tridge]

As suggested by @semuadmin in PR #119 this is a discussion on the UPD UBX sub-protocol.
The motivation for this addition for me is supporting ArduPilot users (see https://ardupilot.org ) to update their uBlox firmware. The uBlox module is frequently embedded deep in the aircraft and we would like to avoid the user having to take it out and change cabling. This is especially the case when the uBlox receiver is part of a DroneCAN peripheral and thus the UART is not exposed at all.
For pyubx2 I'm just contributing back a few changes that were needed to allow UDP to work. The actual fw updater is currently in my junkcode repo as a staging point before it is incorporated into ArduPilot MAVLink GCS and the DroneCAN gui tool. For MAVLink and DroneCAN the UBX protocol is tunnelled over MAVLink and DroneCAN tunnelling sub-protocols.
The key reason we can't use uCenter is uCenter can only connect to remote devices over TCP, and TCP doesn't carry the needed baudrate changes. By re-implementing the protocol ourselves we can add the required tunnelling wrappers to carry both the bytes and the baudrate changes.
Note that I am not under NDA to u-blox in any way and don't have access to any secret information (eg leaked specs). I noted that u-blox says they can give details on the update protocol under NDA, but I didn't ask them for it as that would likely prevent an open implementation.
The history of working out the protocol is that another ArduPilot developer worked out the basics of the protocol by watching wire traffic when uCenter does an update, then I completed that work yesterday to work out the final pieces. It is really a quite simple protocol. I captured the bytes and baudrate changes using a modified version of ArduPilot which can do port forwarding, I just added logging of the bytes along with the baud changes. The actual capture I used is here:
http://uav.tridgell.net/tmp/ublox-fup4.dat
If you'd prefer not to have UPD in pyubx2 that is fine. I can just do my own UBX parser and generator for this project, but I thought it may be of interest.

[tridge]

the key changes I have made are:

• added UPD messages. This is definitely not complete, just sufficient for my simple update script
• fixed a problem with parsing replies where the uBlox module does not fill in the full reply. This happens with the UPD-WRITE protocol where the request has 512 bytes of data and the reply comes in two parts, first an ACK-ACK that the module has received the request then a UPD-WRITE with no data in the A512 array but with the same address as the request. That UPD-WRITE with empty data killed the pyubx2 parser
• added timeout handling. The current UBXReader code assumes that if it has read the header then it can read the rest of the body of the message in a single read() call. That is an invalid assumption on any form of network socket.
• the key = None change is needed so the exception doesn't try to print {key} without the variable, leading to nested exception
• the protfilter check is because UBXReader was trying to use readline on a mavlink socket which is invalid. If we have said we only want UBX then we really shouldn't be trying to parse NMEA, it should treat NMEA as noise

[tridge]

status of the implementation at https://github.com/tridge/junkcode/blob/master/u-blox-fw-update/ubx_update.py is

• update over direct serial works well, tested only on ZED-F9P so far
• update over MAVLink tunnelling works as well, also tested only on ZED-F9P

I'll do some testing on other models soon. I have a drawer full of various uBlox modules to test. I suspect using the chiperase may need a relatively recent receiver (otherwise why would they make it optional in uCenter?) so may need to use the slower sector erase for older modules

[semuadmin]

Hi @tridge

Thanks for taking the time to share this with the community – much appreciated.

Drone pilot configuration is a popular use case for these Python GNSS utilities, and I'd be more than happy to enhance pyubx2's utility in this area. This is not the first time functionality like this has been requested (see earlier feature requests and discussions here and in PyGPSClient). There are, however, a number of potential issues with the changes you're proposing:

1. First and foremost, u-blox's UBX firmware update procedure is - as you've alluded - not in the public domain, and there would clearly be potential copyright infringement issues in incorporating code which had in effect been reverse-engineered from a private codebase. Whether u-blox would actually get legal is a moot point, but I'd certainly want to ensure I was on firm ground here. I’m also not party to an NDA and my (repeated) efforts to broach this with the vendor have so far been rebuffed, but I am happy to try again.
2. Unlike the other supported UBX update commands, which are easily reversible, firmware updates carry the risk of bricking the end user's device if not properly implemented. Although the BSD-2 license expressly excludes warranty of any kind, I'd want to ensure this has been tested on a broad range of UBX devices (not just ArduPilot devices). However, this is - as you say - largely an end user application issue; simply having the relevant UPD* payload definitions in pyubx2 is fine.
3. Your code changes effectively address two challenges. One is implementing the relevant payload definitions (the changes you’ve made are fine, caveats above notwithstanding); the other is accommodating a tunnelled MAVlink TCP data stream. pyubx2 is already capable of working with any data stream that supports a read(n) -> bytes method (of which readline() is essentially a special case). The stream itself is defined externally, though pyubx2 implements a lightweight SocketStream wrapper class which allows it to consume TCP streams natively (as long as they inherit the Python Socket class). Rather than amend the core code to handle specific stream types, you can simply define these externally - it may in fact be easier and more flexible to implement a similar wrapper class for MAVlink TCP connections.
4. Re. protocol filtering, I’m not sure your changes would work. Even if you’re not intending to consume other message protocols (NMEA or RTCM), they must still be removed from the incoming buffer in order to get to the next UBX message. The most efficient way to do this is to parse the message and discard the output – otherwise you’re effectively doing repeated single byte reads until the next b”\xb5\x62” protocol header. You can also get spurious errors if other protocols happen to contain embedded UBX-like byte sequences (this is actually why pyubx2 incorporates the pynmeagps and pyrtcm libraries). You could, of course, temporarily disable these protocols prior to performing the update (it would be a simple matter to include the necessary CFG* commands into your ubx_update.py script).

Finally, a couple of pedantic procedural points:

1. The Contributor guidelines ask that you use the prescribed PR template and sign your commits. This helps ensure changes can be reviewed by a volunteer team more efficiently and guarantees the integrity and provenance of any updates - which ultimately helps the contributor as well as the repo. Git signing is supported natively on Linux and is relatively simple to set up on MacOS and Windows using GPG Suite or GPG4Win respectively.
2. The guidelines also require that any changes are 'All your own work'. You mention that the UBX firmware update protocol is actually the work of another developer, but you don't explicitly acknowledge them in your PR. It would be only courteous to give credit where credit's due.

Hope this makes sense. Any queries or comments, don’t hesitate to get back to me.

[tridge]

@semuadmin quite a few things to cover here.
First off, I would recommend using the term "network protocol analysis" not "reverse engineering" for this type of "work out the protocol from the watching the network". It helps differentiate between attaching a debugger to u-center (which may violate some license agreement) and watching bytes within ArduPilot that are sent by u-center (which won't). I'm not sure if you know my background, I'm the original author of Samba, an implementation of the SMB/CIFS network protocol and a lot of that implementation came from network protocol analysis in the same way. I spent a lot of time learning the finer legal details of this type of protocol analysis as part of that work. Basic summary is that it is perfectly legal in US, Europe and Australia (and likely nearly all other jurisdictions) to work out a protocol by watching the wire traffic.
I know you may feel uncomfortable about it anyway, and if so then I'll just publish the work separately, no problem at all. I'll likely use my own UBX parser in that case. I wrote 'pyUblox' back in 2012 when I was implementing a simple RTK system based on RXM-RAW. I haven't updated it in a long time though, and it's still py2 specific, which is why I reached for pyubx2. See https://github.com/tridge/pyUblox if you are curious. I think pyubx2 is better structured then my pyUblox ever was, although I was quite pleased with the UBloxDescriptor() approach to describing UBX allows for handling variants as u-blox change packet formats between versions of the protocol. See https://github.com/tridge/pyUblox/blob/master/ublox/ublox.py#L342
On (2) the risk of bricking isn't as high as you might think. The receivers implement a ROM firmware (bootloader) which is what you boot to when you do an update. If you corrupt the update then it just doesn't boot the main firmware and stays in this bootloader. It knows it is corrupt as it has 2 32 bit checksums. When it refuses to boot due to a corrupt fw then you can just update again. Note that it certainly could confuse the user as the bootloader will talk UBX at 9600 baud and the user may not know they have to connect at that baud rate to recover. That is a matter of documentation I think. I plan to finish implementing an auto-bauder to avoid this issue in the fw update script. See the beginnings of that in the autobaud function in the script.
On (3), I presume you are referring to the timeout code? I would say that the existing _read_bytes() function is really just buggy as it returns EOFError when more bytes could be coming. I don't see how a wrapper handles that without being able to specify a timeout. You don't want to wait forever but the amount of time you want to wait does depend on the context. Looking at read() in SocketStream it just implements an infinite wait, which I think really isn't what is wanted in most cases, I think you do need some type of timeout. Still, I could handle this with my own wrapper if need be.
On (4) I thought it was more logical to treat protocols you don't want as noise. Assuming pyubx2 implements correct resync and CRC checks then it should produce the same results, but calling the NMEA and RTCM3 parsers means you can be vulnerable to exceptions if you get unknown NMEA sentences or RTCM3 types. It also has an infinite timeout if you get a bit of linenoise that looks like NMEA and then it sits waiting for an end of line forever due to the readline. if the user has specifically said they don't want NMEA or RTCM3 then getting errors due to NMEA and RTCM3 parsing would just be annoying.
On the procedural issues happy to reformat the PR and sign commits if you think you do want to accept it
The other ArduPilot dev who did some work on the UPD protocol was Sid @bugobliterator https://github.com/bugobliterator/ - I didn't actually use his code, just learned some things from what he did. He implemented in C++ inside the ArduPilot UBLOX driver, whereas I preferred to do it in python as a general purpose utility class. I'll ping Sid and see if he wants credit.
Thanks for taking the time to look at this!

[semuadmin]

@tridge

Out of interest, how do ArduPilot users do firmware updates now? If it's not possible to remove the GNSS core and update via UART, does the vendor not provide an alternative mechanism?

[tridge]

Out of interest, how do ArduPilot users do firmware updates now? If it's not possible to remove the GNSS core and update via UART, does the vendor not provide an alternative mechanism?

unfortunately most users just never update. Some use the existing pass-thru mechanisms we have in ArduPilot to attach u-center, we implement baud-rate sniffing in our USB driver and can mirror that to a 2nd UART with the SERIAL_PASS* parameters. That only works for receivers which are serial-attached to the flight controller though which is what motivated this work. A lot of people now use DroneCAN GPS modules which only attach on CAN. Some vendors add a USB port to their GPS modules eg this one:
https://holybro.com/collections/gps-rtk-systems/products/dronecan-h-rtk-f9p-helical
and that USB port allows for u-center update, but many vendors/products don't have that, for example this is CAN only:
https://holybro.com/collections/dronecan-h-rtk/products/dronecan-h-rtk-f9p-rover
I hit this last week with my new plane:

it has 2 F9P GPS modules which can't easily be updated and currently have 1.13 firmware on them, which I know is dangerous (that firmware has a very nasty bug when you lose RTK lock where the position and altitude can jump a long way, I've seen logs from drone light show drones that have a huge height change on RTK loss.
The builder of that aircraft are going to send me a special adapter cable to allow me to update, but I'll likely end up just using my new fw updater over DroneCAN (got to use the new toys!)

[semuadmin]

I'm not sure if you know my background, I'm the original author of Samba, an implementation of the SMB/CIFS network protocol.

I did not realise that - respect!

and a lot of that implementation came from network protocol analysis in the same way. I spent a lot of time learning the finer legal details of this type of protocol analysis as part of that work. Basic summary is that it is perfectly legal in US, Europe and Australia (and likely nearly all other jurisdictions) to work out a protocol by watching the wire traffic

Fair comment - I'm probably being unduly cautious (FYI I'm currently working on a SPARTN protocol parser and I have had some pushback in certain areas precisely because I used WIreshark to deduce some implementation details which are poorly documented - this has hampered testing efforts). As I say, simply having the payload definitions in pyubx2 shouldn't ruffle too many feathers, and the end user implementation is down to others.

On (3), I presume you are referring to the timeout code? I would say that the existing _read_bytes() function is really just buggy as it returns EOFError when more bytes could be coming. I don't see how a wrapper handles that without being able to specify a timeout.

That would be the intention - to implement any timeout or EOF handling in the wrapper. I'm conscious my SocketStream class is probably a little too lightweight (thought it seems to work fine for the vast majority of use cases - I do explicitly mention the need for proper external timeout and EOF handling in the header comments). I need to review your changes in more detail to understand any wider implications, but more than happy to take advice from the author of Samba :-)

Be a shame to have to go down another UBX parser route here (and not just for personal vanity reasons ;-) ) - let me review formally next week and I'll get back to you - I normally try to keep weekends clear.

That is a sweet UAV - what sort of endurance do you get, if you don't mind me asking?

PS - be interested to see how your auto-bauder works out - I did originally have something very similar in PyGPSClient but it proved more trouble than it was worth at the time.

[tridge]

That is a sweet UAV - what sort of endurance do you get, if you don't mind me asking?

no problem, I'm suffering from the usual new-plane urge to show it off :-)
It can fly for between 5 and 8 hours depending on the airspeed you fly at.
I think the auto bauder should work OK, but still need to do more work on it. We have an auto-bauder for UBX and other protocols in ArduPilot too, but that is a bit more complex - it injects UBX and NMEA packets to try to force the module to the desired baud rate. The one in my python script will not try to change the baudrate, just detect from a set of likely rates.

[tridge]

btw, you might be interested in this:
https://download.samba.org/pub/tridge/misc/french_cafe.txt
I wrote it as part of my prep as a witness in the EU vs Microsoft anti-trust case (I was a witness on the EU commission side) to explain how Samba was developed (Samba was a major component of the case).
The idea was to explain network protocol analysis in a way that a lay person could understand. It seemed to work well with the judges in that case. Microsoft lost badly, and had to pay a few billion in fines, plus they were ordered to give us detailed protocol specs on a wide range of their protocols. So ironically after we won that case we no longer needed to use those techniques. That actually made Samba much less interesting to work on! Protocol decoding is fun :-)

[semuadmin]

FYI I've had the following feedback from u-blox's EU agent:

"Unfortunately, u-blox would not be amenable to these restricted UDP* payload definitions being incorporated into the open-source pyubx2 library. These payload definitions are explicitly restricted for non-public customer use (i.e. using them to configure the module in your product) only.
Adding these messages to an open-source library would violate the terms of use."

So, unfortunately, I'm not able to include these restricted UPD* payload definitions at this time.

[tridge]

@semuadmin no problem, and I understand your position

[semuadmin]

@semuadmin no problem, and I understand your position

@tridge I'd be happy to assist with your project as best I can - do you have a direct contact address?