Issue with EOFError Handling in _parse_nmea Method

[ariansharifi]

Hello,

I hope this message finds you well. I've been using your pyubx2 library, and I'm really impressed with its functionality and design. Recently, I encountered a potential issue when feeding a file with NMEA content into the library, which I wanted to bring to your attention.

Issue Description

In the _parse_nmea method source code here, the library checks for the end of an NMEA message to see if it terminates with a CRLF (\x0D\x0A):

# read the rest of the NMEA message from the buffer
byten = self._stream.readline()  # NMEA protocol is CRLF-terminated
if byten[-2:] != b"\x0d\x0a":
    raise EOFError()

Potential Concern

The above check raises an EOFError if the last two bytes of the message are not CRLF. However, this can be problematic, particularly when reading files with NMEA content. In my case, the method incorrectly raised an EOFError, leading to an early exit from the file processing.

Suggestion

It may be beneficial to revisit the EOF determination logic better to remove it, since EOF file check already done here:

I greatly appreciate the effort you've put into developing this library and hope this feedback is constructive. If there's any additional information I can provide or if I can assist in resolving this issue, please feel free to reach out.

Thank you for your time and contributions to the open-source community.

Best regards,

[semuadmin]

Hi @ariansharifi

The NMEA protocol specifies that sentences should end with CRLF (b"\x0d\x0a") - this is what GNSS receivers output and this is what the NMEA parser checks for. This is intentional behaviour. pyubx2.ubxreader is designed to parse mixed-protocol binary streams which can include a combination of NMEA, UBX and RTCM messages, so the parser has to cater for a variety of different EOM (end of message) and EOF (end of file) conditions.

The relevant NMEA EOM check is here - the code segment you link to to above only applies to UBX or RTCM binary message types, which do not use a CRLF terminator.

In what way is this problematic?

NB: some generic text editors (e.g. Notepad) can corrupt NMEA data logs by stripping out the CR or LF character. It's possible this may be the cause of your issue. I recommend viewing or editing NMEA data files with a proper hex editor that preserves the CRLF line ending. This issue is explicitly mentioned in the Troubleshooting section of README.md.

[ariansharifi]

this is a example test.log file, which all messages are nmea data

import pyubx2

with open("test.log", "rb") as log_file:
    ubt = pyubx2.UBXReader(log_file)
    while True:
        raw, parsed = ubt.read()
        print(parsed)
        if (not parsed) and (not raw):
            print("EXIT FILE")
            break

it will not print parsed data it just exit the file immideitly but when I removing those lines it will work as excpected and parse all file
in this case I think it will be preoblematic

[semuadmin]

As I suspected, the test.log file you've provided has become corrupted somehow - the CRLF (0x0D0A) terminators have been replaced by LF (0x0A):

Compare this with an uncorrupted NMEA log taken from a u-blox F9P device using PyGPSClient's datalog facility:

How was your test.log file produced?

NB: I can update NMEA message processing within pyubx2.ubxreader to cater for such data streams - it's a relatively trivial change - but strictly this should not be necessary. A well-formed NMEA data stream should always have CRLF message terminators.

[semuadmin]

FYI here's an example code snippet that WILL parse your test.log file as it stands OK, using a standard readline() loop:

from pynmeagps import NMEAReader

with open("test.log", "rb") as infile:
    i = 0
    while True:
        raw = infile.readline()
        if not raw:
            break
        msg = NMEAReader.parse(raw)
        print(msg)
        i += 1

print(f"{i} NMEA messages read from input file")

...
<NMEA(GPGGA, time=15:42:48.730000, lat=51.5895333333, NS=N, lon=-0.4811666667, EW=W, quality=1, numSV=12, HDOP=1.0, alt=0.0, altUnit=M, sep=0.0, sepUnit=M, diffAge=, diffStation=)>
<NMEA(GPGSA, opMode=A, navMode=3, svid_01=1, svid_02=2, svid_03=3, svid_04=4, svid_05=5, svid_06=6, svid_07=7, svid_08=8, svid_09=9, svid_10=10, svid_11=11, svid_12=12, PDOP=1.0, HDOP=1.0, VDOP=1.0)>
<NMEA(GPRMC, time=15:42:48.730000, status=A, lat=51.5895333333, NS=N, lon=-0.4811666667, EW=W, spd=2263.8, cog=298.1, date=2023-11-28, mv=0.0, mvEW=W)>
8583 NMEA messages read from input file

However, in parsing this data I notice that this appears to be a very old version of the NMEA protocol (< NMEA 2.3 - the latest version is 4.11) - e.g. the GPRMC messages lack the posData and navPosition attributes that were added in versions 2.3 and 4.10 respectively. Out of interest, what device did these messages come from?

[ariansharifi]

Hi again,
thank you for decent answer I got it from nmeagen.org,noticed the issue, I've got what to do tnx

I also have one more thing to add.
I've been playing around with updating ublox firmware – a bit of a puzzle since a lot of the process isn't well documented. What I did was capture the messages going back and forth between ublox devices and uCenter, and then I used your library to parse them. It was quite a dive, but I managed to uncover some undocumented messages that are key to updating the firmware.

I thought this might be something you or others using your library could be interested in. It's not every day you get to decode a bit of a mystery, right? If you think this info could be useful, I'm more than happy to share what I've found. (if someone else haven't found/published them yet whici I'm not aware of :D )

Your library has been a great help in this little adventure. Let me know if this sounds interesting to you and how I might share these discoveries with the community.

[semuadmin]

Hi again @ariansharifi

You are not the first to suggest this. There is unfortunately a bit of a long story here...

The UBX © protocol is a proprietary protocol, copyrighted by u-blox™. The GNSS device interface specifications are published by u-blox on their web site and are effectively in the public domain (e.g. https://www.u-blox.com/sites/default/files/u-blox-M9-SPG-4.04_InterfaceDescription_UBX-21022436.pdf). It is these documents that pyubx2 is based on.

However, some aspects of the UBX protocol are NOT in the public domain and u-blox only makes this restricted information available to developers under NDA (Non-Disclosure Agreement). This NDA-protected material cannot (legally) be incorporated into an open-source library like pyubx2 (though you may come across some other repos on GitHub that have attempted to 'reverse engineer' the restricted protocols using e.g. packet sniffing, which is considered by them to be 'fair game').

The restricted information includes, for example, UBX payloads relating to:

• firmware upgrades (UPD message types)
• internal u-blox testing and diagnostic procedures (including DBG, SEC, TRK and TUN message types)
• private configuration database keys for Generation 9+ devices. (CFG-VAL* message types)

I am myself an NDA signatory and have in the past asked u-blox if they would be prepared to allow me to include some or all of these restricted payload definitions in pyubx2, but the answer has always been 'NO'. Frankly, the maintenance of this library is entirely voluntary and unpaid on my part (I receive absolutely no support from u-blox) so I'm not really motivated to challenge this position.

So the current situation is that pyubx2 will happily parse these undocumented message types from a UBX data stream but it cannot decode them into their constituent data fields. It will, instead, simply display the raw undecoded payload bytes with a 'NOMINAL' tag. So if, for example, you record a UBX data stream in u-center with the 'include configuration' and 'debug' options enabled, that data stream will include a bunch of undocumented message types. Here's an illustration:

<UBX(DBG-0c31-NOMINAL, data_01=b'\x01', data_02=b'\x03', data_03=b'\x00', data_04=b'\x04', data_05=b'\x02', data_06=b'\x01', data_07=b'\x00', data_08=b'\x00', data_09=b'\x0a', data_10=b'\x9a', data_11=b'\x04', data_12=b'\x20', data_13=b'\x0a', data_14=b'\x08', data_15=b'\x05', data_16=b'\x20', data_17=b'\x10', data_18=b'\x60', data_19=b'\x04', data_20=b'\x20', data_21=b'\xb8', data_22=b'\x11', data_23=b'\x0b', data_24=b'\x15', data_25=b'\x7e', data_26=b'\x11', data_27=b'\xcb', data_28=b'\x6b', data_29=b'\xa8', data_30=b'\x11', data_31=b'\x0b', data_32=b'\x18', data_33=b'\x08', data_34=b'\x11', data_35=b'\x0b', ...)>
<UBX(DBG-0c34-NOMINAL, data_01=b'\x03', data_02=b'\x01', data_03=b'\x12', data_04=b'\x05', data_05=b'\x33', data_06=b'\x12', data_07=b'\x01', data_08=b'\x01', data_09=b'\xa9', data_10=b'\x08', data_11=b'\x02', data_12=b'\x02', data_13=b'\xe9', data_14=b'\x7a', data_15=b'\x00', data_16=b'\x00', data_17=b'\x00', data_18=b'\x00', data_19=b'\x00', data_20=b'\x00', data_21=b'\x3d', ...)>
<UBX(DBG-0c10-NOMINAL, data_01=b'\x20', data_02=b'\x00', data_03=b'\x07', data_04=b'\x00', data_05=b'\x24', data_06=b'\xee', data_07=b'\x10', ...)>

In the illustration above, the messages are actually internal debug messages but their content is known only to u-blox engineers (and NDA signatories).

Hope this helps.

FYI you might be interested to read this older Discussions topic on the same subject: #120

[ariansharifi]

ok got it, so we don't dig into message that what's are they

but I can give my app that can update a firmware file to ublox reciver using your lib :D we don't know what those messages are exactly, but we can do our own firmware update with those ( if you can publish it in the example folder without any legal problem)

[semuadmin]

but I can give my app that can update a firmware file to ublox reciver using your lib :D we don't know what those messages are exactly, but we can do our own firmware update with those ( if you can publish it in the example folder without any legal problem)

So - yes - it is technically possible to update u-blox firmware via UBX UPD messages, but it wouldn't be a trivial undertaking.

I have sight of the necessary UPD firmware upgrade protocols (under NDA) and have a private fork of pyubx2 that already implements most of the necessary functionality, but for reasons I've already outlined I'm not at liberty to put that in the public domain.

The current public version of pyubx2 does NOT implement the necessary UPD message types and the update process itself is quite complex - it's not a case of simply sending a UBX firmware payload to the receiver.

The other issue (as I've mentioned in earlier Discussion threads) is that adding a firmware upgrade facility to pyubx2 introduces a very real risk of users bricking their devices (either through bugs in the code or end-user abuse). So while the BSD-2 license explicitly excludes any kind of warranty, I couldn't in all conscience introduce such a facility without rigorous testing across a range of u-blox devices, and as an unpaid volunteer project I simply don't have the resources to do that.

Obviously you're at liberty to do whatever you want with the pyubx2 library, but I would advise caution in this area.

Happy to continue this Discussion thread but suggest moving it to a separate topic.