frigate:0.8.0-beta3 - clips stop working after some time

[walthowd]

Hi all -- I'm trying to track down an issue in frigate:0.8.0-beta3 where after some hours/days clip generation seems to stop working.

I can see that object analysis is still working, and the /camera/person/best.jpg file is updates -- the HA sensor sensor.camera_person is also updated, but when the issue is happening the object count seems high. I typically see 9, 10 or 15 as the sensor value.

Also the /camera live view point doesn't show a person bounding box when the issue is occurring. Restarting frigate seems to bring clips back to working.

I'm running now with full debug logging and will open an issue or try to track the code down myself (Blake is working way to hard at this IMHO) but wanted to post the discussion here in case anyone else has anything to share.

Thanks!

[walthowd]

So far, seeing DEBUG inserts into frigate.db, but nothing actually makes it into the database.

2021-01-09T17:49:37.405926393Z peewee                         DEBUG   : ('INSERT INTO "event" ("id", "label", "camera", "start_time", "end_time", "top_score", "false_positive", "zones", "thumbnail") VALUES (?, ?, ?, ?, ?, ?, ?, json(?), ?)', ['1610214554.776564-yd4cb7', 'person', 'doorbell', 1610214554.776564, 1610214560.065895, 0.71484375, False, '[]', 'BINARY_SS'])

No other errors or debug info present.

[walthowd]

Now on beta-0.80-RC1 -- Clips detection works until this is logged by docker, then after that no more clips are generated:

2021-01-19T21:35:22.637442842Z frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process
2021-01-19T21:35:22.637594130Z root                           INFO    : Waiting for detection process to exit gracefully...
2021-01-19T21:35:52.665805958Z root                           INFO    : Detection process didnt exit. Force killing...
2021-01-19T21:35:52.712351150Z detector.coral                 INFO    : Starting detection process: 6630
2021-01-19T21:35:55.485986725Z frigate.edgetpu                INFO    : Attempting to load TPU as usb
2021-01-19T21:35:55.489139272Z frigate.edgetpu                INFO    : TPU found

[walthowd]

Also maybe a camera crashing, then Frigate goes into a loop with No cache clips for <cameraname>. Waiting. . . and doesn't generate any clips for any other camera either.

[walthowd]

Opened a issue:
#578

Added some additional debugging:
walthowd@df978ad

[walthowd]

Looks like we finally figured it out. Root cause is FFMPEG not existing when losing a RTMP stream. Adding rw_timeout flag fixes that.

  garage:
    ffmpeg:
      inputs:
        - path: "rtmp://192.168.1.128/bcs/channel0_sub.bcs?channel=0&stream=0&user=view&password=camera-password"
          roles:
            - detect
        - path: "rtmp://192.168.1.128/bcs/channel0_main.bcs?channel=0&stream=0&user=view&password=camera-password"
          roles:
            - clips
            - record
          global_args:
            - -rw_timeout
            - '5000000'
            - -hide_banner
            - -loglevel
            - info
    fps: 4
    height: 480
    width: 640
    clips:
      enabled: true
    record:
      enabled: true
      retain_days: 30
    objects:
      track:
        - person
        - car
        - truck