3.0.4 - Error in Watch Loop Crash

[alfwro13]

I am running GivTCP 3.0.4. My inverter is Gen 1 HY5.0 with 9.5 kWh battery. I am on Octopus Agile tariff so I have automation setup to charge at night when there are cheap slots available.
Last night however GivTCP crashed resulting in an empty battery this morning. GivTCP log file was full of Error in Watch Loop (100K+ lines)
At various times the software tried to restart without success. Restarting the addon fixed the problem.
The inverter was online all night - I have a HA ping sensor setup and that is showing as Connected throughout the night.
Here is the relevant bits from the log file:

192.168.99.42024-12-08 21:51:28,800 - startup     -  [ERROR   ] - Self Run loop process stuck. Killing and restarting...
2024-12-08 21:51:28,801 - startup     -  [INFO    ] - Restarting Invertor read loop every 10s
2024-12-08 21:51:29,101 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-08 21:51:29,101 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-08 21:51:29,103 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-08 22:00:30,299 - GE - mqtt        -  [INFO    ] - Requesting Control Action: setBatteryPauseMode - {'state': 'Disabled'}
2024-12-08 22:00:51,396 - GE - write       -  [INFO    ] - Setting Battery Pause Mode to Disabled was a success
2024-12-08 22:17:28,813 - startup     -  [ERROR   ] - Self Run loop process stuck. Killing and restarting...
2024-12-08 22:17:28,813 - startup     -  [INFO    ] - Restarting Invertor read loop every 10s
2024-12-08 22:17:29,119 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-08 22:17:29,119 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-08 22:17:29,121 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-08 22:18:19,176 - GE - read        -  [ERROR   ] - Error in inital detect/refresh: ('TimeoutError', 'read.py', 77)
2024-12-08 22:18:19,176 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-08 22:18:19,176 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-08 22:18:20,189 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-08 22:18:28,814 - startup     -  [ERROR   ] - Self Run loop process stuck. Killing and restarting...
2024-12-08 22:18:28,814 - startup     -  [INFO    ] - Restarting Invertor read loop every 10s
2024-12-08 22:18:29,120 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-08 22:18:29,121 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-08 22:18:30,174 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-08 22:19:00,888 - GE - mqtt        -  [INFO    ] - Requesting Control Action: rebootAddon - {}
2024-12-08 22:19:00,969 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-08 22:19:00,973 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-08 22:19:00,976 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-08 22:19:01,984 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-08 22:19:02,003 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-08 22:19:02,006 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-08 22:19:03,070 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)



2024-12-09 01:59:27,652 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 01:59:27,656 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 01:59:27,660 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 01:59:28,671 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 01:59:28,674 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 01:59:29,014 - startup     -  [ERROR   ] - Self Run loop process stuck. Killing and restarting...
2024-12-09 01:59:29,014 - startup     -  [INFO    ] - Restarting Invertor read loop every 10s
2024-12-09 01:59:29,325 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-09 01:59:29,325 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-09 01:59:29,329 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-09 02:00:06,712 - GE - read        -  [ERROR   ] - Error in inital detect/refresh: ('TimeoutError', 'read.py', 77)
2024-12-09 02:00:06,712 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-09 02:00:06,712 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-09 02:00:07,781 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-09 02:00:29,015 - startup     -  [ERROR   ] - Self Run loop process stuck. Killing and restarting...
2024-12-09 02:00:29,016 - startup     -  [INFO    ] - Restarting Invertor read loop every 10s
2024-12-09 02:00:29,326 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-09 02:00:29,326 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-09 02:00:31,329 - GE - read        -  [ERROR   ] - Unable to connect to inverter on: 192.168.99.4
2024-12-09 02:00:31,329 - GE - read        -  [ERROR   ] - Error in self_run. Re-running watch_plant: ('UnboundLocalError', 'read.py', 1874)
2024-12-09 02:00:33,332 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-09 02:00:33,332 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-09 02:00:33,339 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-09 02:00:55,237 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:55,247 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:55,250 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:56,287 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:56,604 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:56,614 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:57,630 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:57,632 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:58,978 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:00:58,985 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:00,987 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('CommunicationError', 'read.py', 112)
2024-12-09 02:01:02,049 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:02,059 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:02,065 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:04,067 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('CommunicationError', 'read.py', 112)
2024-12-09 02:01:04,070 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:04,076 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:05,118 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:07,121 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('CommunicationError', 'read.py', 112)
2024-12-09 02:01:07,127 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:08,191 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 02:01:08,218 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)



2024-12-09 04:29:25,609 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:25,613 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:26,624 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:26,627 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:27,648 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:27,652 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:27,658 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:27,661 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:28,673 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:29:29,149 - startup     -  [ERROR   ] - Self Run loop process stuck. Killing and restarting...
2024-12-09 04:29:29,149 - startup     -  [INFO    ] - Restarting Invertor read loop every 10s
2024-12-09 04:29:29,441 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-09 04:29:29,442 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-09 04:29:29,443 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-09 04:30:27,845 - GE - read        -  [INFO    ] - Saving current energy stats at start of day rate tariff (Dynamic)
2024-12-09 04:30:27,867 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:30:27,871 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:30:27,886 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:30:27,925 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:30:28,959 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:30:28,964 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:30:28,970 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:30:30,005 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)



#Here automation was trying to enable grid charging:

2024-12-09 04:31:42,670 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:42,914 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:42,916 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:43,965 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:43,967 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:43,969 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:45,183 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:46,023 - GE - mqtt        -  [INFO    ] - Requesting Control Action: enableChargeSchedule - {'state': 'enable'}
2024-12-09 04:31:46,024 - GE - mqtt        -  [INFO    ] - Requesting Control Action: setChargeRate - {'chargeRate': '2600'}
2024-12-09 04:31:46,026 - GE - mqtt        -  [INFO    ] - Requesting Control Action: setBatteryPauseMode - {'state': 'Disabled'}
2024-12-09 04:31:46,027 - GE - mqtt        -  [INFO    ] - Requesting Control Action: setBatteryMode - {'mode': 'Eco'}
2024-12-09 04:31:46,209 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:46,212 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:46,215 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:47,232 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:47,235 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:47,237 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 04:31:47,242 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)


2024-12-09 05:54:25,641 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:54:26,653 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:54:26,655 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:54:26,658 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:54:27,679 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:54:27,682 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:54:27,688 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:54:27,691 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:54:29,216 - startup     -  [ERROR   ] - Self Run loop process stuck. Killing and restarting...
2024-12-09 05:54:29,217 - startup     -  [INFO    ] - Restarting Invertor read loop every 10s
2024-12-09 05:54:29,514 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-09 05:54:29,514 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-09 05:54:29,519 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-09 05:55:00,189 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:55:00,193 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:55:00,197 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:55:01,216 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:55:01,363 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:55:01,370 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:55:01,374 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 05:55:01,393 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)


2024-12-09 07:00:28,390 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 07:00:28,402 - GE - read        -  [ERROR   ] - Error in Watch Loop: ('TypeError', 'read.py', 126)
2024-12-09 07:00:29,271 - startup     -  [ERROR   ] - Self Run loop process stuck. Killing and restarting...
2024-12-09 07:00:29,271 - startup     -  [INFO    ] - Restarting Invertor read loop every 10s
2024-12-09 07:00:29,600 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-09 07:00:29,600 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-09 07:00:30,623 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-09 07:00:40,864 - startup     -  [INFO    ] - ==================== STARTING GivTCP==========================
2024-12-09 07:00:40,875 - startup     -  [INFO    ] - Network Found: 192.168.99.254/24
2024-12-09 07:00:42,876 - startup     -  [INFO    ] - Searching for Inverters
2024-12-09 07:00:42,876 - startup     -  [INFO    ] - Scanning network for GivEnergy Devices...
2024-12-09 07:00:45,087 - startup     -  [INFO    ] - INV- Scanning network (1):192.168.99.254/24
2024-12-09 07:00:50,566 - startup     -  [INFO    ] - INV- Scanning network (2):192.168.99.254/24
2024-12-09 07:01:00,680 - startup     -  [INFO    ] - No inverters found...
2024-12-09 07:01:00,680 - startup     -  [ERROR   ] - =============================================================
2024-12-09 07:01:00,680 - startup     -  [ERROR   ] - ====               NO INVERTERS FOUND                    ====
2024-12-09 07:01:00,680 - startup     -  [ERROR   ] - ====      add manually using a file editor to modify     ====
2024-12-09 07:01:00,681 - startup     -  [ERROR   ] - ====          /config/GivTCP/allsettings.json            ====
2024-12-09 07:01:00,681 - startup     -  [ERROR   ] - =============================================================
2024-12-09 07:01:00,683 - startup     -  [INFO    ] - Setting up invertor: 1
2024-12-09 07:01:00,717 - startup     -  [INFO    ] - ==============================================================
2024-12-09 07:01:00,717 - startup     -  [INFO    ] - ====             Web Gui Config is at                     ====
2024-12-09 07:01:00,717 - startup     -  [INFO    ] - ====     http://192.168.99.66:8099/config.html             ====
2024-12-09 07:01:00,717 - startup     -  [INFO    ] - ==============================================================
2024-12-09 07:01:00,718 - startup     -  [INFO    ] - Running Invertor 1 (SERIAL_NUMBER) read loop every 10/120s
[2024-12-09 07:01:00 +0000] [90] [INFO] Starting gunicorn 23.0.0
[2024-12-09 07:01:00 +0000] [90] [INFO] Listening at: http://0.0.0.0:6350 (90)
[2024-12-09 07:01:00 +0000] [90] [INFO] Using worker: sync
[2024-12-09 07:01:00 +0000] [106] [INFO] Booting worker with pid: 106
[2024-12-09 07:01:00 +0000] [105] [INFO] Starting gunicorn 23.0.0
[2024-12-09 07:01:00 +0000] [105] [INFO] Listening at: http://0.0.0.0:6345 (105)
[2024-12-09 07:01:00 +0000] [105] [INFO] Using worker: sync
[2024-12-09 07:01:00 +0000] [107] [INFO] Booting worker with pid: 107
[2024-12-09 07:01:00 +0000] [108] [INFO] Booting worker with pid: 108
[2024-12-09 07:01:00 +0000] [109] [INFO] Booting worker with pid: 109
2024-12-09 07:01:01,159 - GE - read        -  [INFO    ] - Starting watch_plant loop...
2024-12-09 07:01:01,160 - GE - GivLUT      -  [CRITICAL] - Opening Modbus Connection to 192.168.99.4
2024-12-09 07:01:02,180 - GE - read        -  [CRITICAL] - Detecting inverter characteristics...
2024-12-09 07:01:22,136 - GE - read        -  [INFO    ] - Publishing Home Assistant Discovery messages
2024-12-09 07:02:42,031 - GE - mqtt        -  [INFO    ] - Requesting Control Action: enableChargeSchedule - {'state': 'enable'}
2024-12-09 07:02:42,035 - GE - mqtt        -  [INFO    ] - Requesting Control Action: setChargeRate - {'chargeRate': '2600'}
2024-12-09 07:02:42,036 - GE - mqtt        -  [INFO    ] - Requesting Control Action: setBatteryPauseMode - {'state': 'Disabled'}
2024-12-09 07:02:42,037 - GE - mqtt        -  [INFO    ] - Requesting Control Action: setBatteryMode - {'mode': 'Eco'}
2024-12-09 07:02:43,318 - GE - write       -  [INFO    ] - Setting Charge Schedule to enable was a success
2024-12-09 07:02:43,785 - GE - write       -  [INFO    ] - Setting battery charge rate 2600 was a success
2024-12-09 07:02:44,255 - GE - write       -  [INFO    ] - Setting Battery Pause Mode to Disabled was a success
2024-12-09 07:02:45,231 - GE - write       -  [INFO    ] - Setting Eco mode was a success

[gcoan]

@alfwro13 you might find this automation I wrote useful https://springfall2008.github.io/batpred/output-data/#givtcp-activity-monitor

It checks for a number of different things that might indicate if GivTCP (or MQTT) is not running such as battery temperature, date last updated, etc, and restarts givtcp if it detects a problem