getting tons of CRC and Serial Timeout all of a sudden

[Mehrdadg]

Hi All
My Genmon SW running on RPI3 with purchased Genmon HAT has been working flawlessly for the last 6+ months however, over the last couple of days, I am seeing tons of errors happening and don't know the root cause so any help is greatly appreciated.
Below is a snapshot of the serial monitor page:
Communication Stats :
Packet Count : M: 38058, S: 35950
CRC Errors : 42
CRC Percent Errors : 0.11%
Timeout Errors : 2048
Timeout Percent Errors : 5.38%
Modbus Exceptions : 0
Validation Errors : 0
Sync Errors : 0
Invalid Data : 0
Discarded Bytes : 18
Comm Restarts : 1
Packets Per Second : 6.93
Average Transaction Time : 0.0312 sec
Modbus Transport : Serial
Serial Data Rate : 9600

running the latest FW revision: V1.19.03
I have restarted the genmon app, restarted the RPI and still the same issue

thanks in advance,

[jgyates]

I would check your cabling first. Connections that are marginal can be effected by heat / cool conditions and also vibrations. Since you have both timeout and CRC errors, to me, this is the most likely problem. The connections between the RS-232 converter and the pi as well as the connections between the RS-232 and the controller should be checked.

[Mehrdadg]

You were spot on as one of the screw terminals on the serial line was a little loose. I tightened it and so far no CRC or timeout errors.

thanks!

[Mehrdadg]

I hate to admit I might have rushed to victory too quickly:(
I did have a somewhat loose connection on the Tx line but that did not resolve my issue entirely...

I still this issue happening but what is very strange is that the only time I am getting "Not Receiving Data" with many timeouts and CRC errors happen to happen daily at exactly 4AM or 5AM and sometimes around 9A or ~10PM

I know this sounds very strange but I am thinking maybe something wrong with my RPI3 that is doing something and starving the CPU to process the serial interface.

I have also checked for any CRON jobs happening at exactly 4 or 5AM but no luck!

I have updated the RPI to its latest version and still having issue. I am now thinking maybe time to reinstall the RPI OS and start fresh.

Any recommendation is greatly appreciated.

[jgyates]

I would check all of the wired connections on the pi. Also check that the pi is not experiencing any power issues. On the Monitor page look at the following and make sure all are "OK". If any are not then you may have power issues.

   Pi CPU Frequency Throttling : OK
    Pi ARM Frequency Cap : OK
    Pi Undervoltage : OK

Intermittent issues can be difficult to track down. Systematically replacing pieces of the cable or hardware may help drill down. In my experience intermittent comm issues are more likely a connection issue than a bad pi, but a bad pi is possible too.

[Mehrdadg]

Hi
I checked all of the wiring and all connections looked great. I also viewed all the signals to/from the RPI on scope and all levels checked out ok in/out of the serial translator so I am confident all is good.

below is the snapshot of the monitor page and you can see nothing abnormal being reported on CPU Freq, UV etc:

Communication Stats :
Packet Count : M: 5337435, S: 5331579
CRC Errors : 182
CRC Percent Errors : 0.00%
Timeout Errors : 5607
Timeout Percent Errors : 0.11%
Modbus Exceptions : 0
Validation Errors : 0
Sync Errors : 0
Invalid Data : 0
Discarded Bytes : 66
Comm Restarts : 4
Packets Per Second : 54.30
Average Transaction Time : 0.0312 sec
Modbus Transport : Serial
Serial Data Rate : 9600

Platform Stats :
    CPU Temperature : 129.56 F
    Pi Model : Raspberry Pi 3 Model B Rev 1.2
    Pi CPU Frequency Throttling : OK
    Pi ARM Frequency Cap : OK
    Pi Undervoltage : OK
    CPU Utilization : 9.63%
    OS Name : Raspbian GNU/Linux
    OS Version : 11 (bullseye)
    System Uptime : 2 days, 6:35:22
    Network Interface Used : eth0
    System Time : Tuesday June 4, 2024 21:20:53

what is really strange is the exact times/repeatable times when I get the notifications of the communication errors!

[jgyates]

are you using mobile link at the same time as genmon?

If the errors are occurring at the same time every day, it could be that there is some housekeeping task that is occurring on the controller at that time that is interfering with serial comms. I have never heard of this happening before, just a thought. If it that repeatable in terms of time the replacing the OS may be a thing to try.

[Mehrdadg]

I am not using mobile link at all and nothing is connected to the controller other than genmon. Task this weekend is to replace the os.

Indeed a strange problem:)

Thanks and will report back after os replacement and running for day or so.

[jgyates]

Thanks for the feedback and status. I updated the script to include the callmebot add on in the backup.

[Mehrdadg]

hi
I changed the RPI, installed a fresh RPI OS, re-installed the Genmon SW but unfortunately none of the fixed the time out issues.
I still get timeout errors right at 4AM (100% of the time) and sometimes during the day and it usually lasts about ~20 minutes and then it clears itself. Worth mentioning that CRC errors are gone and I think that had to do with loose connections and since I tighten the screws, it cleared this issue.

I have triple checked the wiring and all looks great. I even put scope on the TX and RX lines and the signal levels and transitions look ok as well. Note that when I am getting timeout errors, the generator is not running.

So honestly I am at a loss and now suspecting something has gone wrong with my controller:( or the genmon hat. What is really puzzling if there is a signal integrity issues, then why on earth this is happening 4AM every day - truly a strange issue I am experiencing all of a sudden.

One thing worth mentioning is that my generator is due for "Service A" maintenance since June 1 and oddly enough my issues started around the ~time frame.

[jgyates]

If this an evolution 2.0 controller? There are some bugs / unusual behavior that I have seen from the Evo 2.0 firmware, but I have not seen this specific behavior before. Here are some suggestions:

1. Change the time on the controller (but don't sync the controller time to the pi via genmon) and see if it still occurs at 4AM. This should let you know if the issues is with the controller or genmon / pi based on the time the timeouts occur.
2. Alternatively, you can reset the service A alarm and see if that corrects the issue. if that corrects the issue then you can probably assume it is a controller issue.

[Mehrdadg]

Yes it is evolution 2 controller. Will try your suggestions but unfortunately not until the weekend.

One question... how can I reset the service A alarm? Can this be accomplished remotely using Genmon or I need to physically do it via the controller? Any instructions to carry it out via Genmon or controller itself is greatly appreciated.
Thanks