Detection appears to be stuck... and so am I

[CWuestefeld]

As a new user, I'm trying to get frigate running. I've got the container running, it can talk to the camera. But not much more than that is working. In particular, I'm getting the below sequence of log messages over and over again.

frigate  | 2023-07-06 16:52:00.200973117  [2023-07-06 16:52:00] root                           INFO    : Waiting for detection process to exit gracefully...
frigate  | 2023-07-06 16:52:30.222706887  [2023-07-06 16:52:30] root                           INFO    : Detection process didnt exit. Force killing...
frigate  | 2023-07-06 16:52:30.238071013  [2023-07-06 16:52:30] root                           INFO    : Detection process has exited...
frigate  | 2023-07-06 16:52:30.257719816  [2023-07-06 16:52:30] detector.coral                 INFO    : Starting detection process: 1217
frigate  | 2023-07-06 16:52:32.918747264  [2023-07-06 16:52:30] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
frigate  | 2023-07-06 16:52:32.930638761  [2023-07-06 16:52:32] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
frigate  | 2023-07-06 16:53:10.250649287  [2023-07-06 16:53:10] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...

In testing, I have seen the system to detect me; apparently it's 82% sure that I'm a person. I'm not sure how reliable this is, but the fact that this happened at all seems to suggest that it's successfully communicating with the Coral.

The hardware this is running on is an old orphaned desktop, so an old Intel x86 64 bit architecture, and apparently old enough that its onboard GPU is of no use. But that doesn't seem to be the problem, as far as I can tell. I'm running Ubuntu 22 LTS, which I'd think would be pretty vanilla.

I found this conversation: #1163 but apparently that was in a VM, while I'm on bare metal. But it talks about doing some tests on the Coral device. Unfortunately the Coral test script requires python < 3.10 - but my linux distro has 3.10, so I can't run it.

Config is reproduced below.

Any suggestion where to go from here?

mqtt:
  enabled: False  #  host: mqtt.server.com #Please replace with a valid mqtt broker
detectors:
  # Required: name of the detector
  coral:
    # Required: type of the detector
    # Valid values are 'edgetpu' (requires device property below) and 'cpu'.
    type: edgetpu
    # Optional: device name as defined here: https://coral.ai/docs/edgetpu/multiple-edgetpu/#using-the-tensorflow-lite-python-api
    device: usb
# Optional: model modifications
model:
  # Optional: path to the model (default: automatic based on detector)
  path: /edgetpu_model.tflite
  # Optional: path to the labelmap (default: shown below)
  labelmap_path: /labelmap.txt
  # Required: Object detection model input width (default: shown below)
  width: 320
  # Required: Object detection model input height (default: shown below)
  height: 320
cameras:
  front:
    ffmpeg:
      hwaccel_args: " "
      inputs:
        - path: rtsp://192.168.0.124:554/?user=admin&password=&channel=1&stream=0.sdp #replace with your RTSP camera url
          roles:
            - detect
            - rtmp
    detect:
      width: 2592
      height: 1944
  
# Optional: Detect configuration
# NOTE: Can be overridden at the camera level
detect:
  # Optional: width of the frame for the input with the detect role (default: shown below)
  width: 800
  # Optional: height of the frame for the input with the detect role (default: shown below)
  height: 600
  # Optional: desired fps for your camera for the input with the detect role (default: shown below)
  # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
  fps: 5
  # Optional: enables detection for the camera (default: True)
  # This value can be set via MQTT and will be updated in startup based on retained value
  enabled: True
  # Optional: Number of frames without a detection before frigate considers an object to be gone. (default: 5x the frame rate)
  max_disappeared: 25
  # Optional: Configuration for stationary object tracking
  stationary:
    # Optional: Frequency for confirming stationary objects (default: shown below)
    # When set to 0, object detection will not confirm stationary objects until movement is detected.
    # If set to 10, object detection will run to confirm the object still exists on every 10th frame.
    interval: 0
    # Optional: Number of frames without a position change for an object to be considered stationary (default: 10x the frame rate or 10s)
    threshold: 50
    # Optional: Define a maximum number of frames for tracking a stationary object (default: not set, track forever)
    # This can help with false positives for objects that should only be stationary for a limited amount of time.
    # It can also be used to disable stationary object tracking. For example, you may want to set a value for person, but leave
    # car at the default.
    # WARNING: Setting these values overrides default behavior and disables stationary object tracking.
    #          There are very few situations where you would want it disabled. It is NOT recommended to
    #          copy these values from the example config into your config unless you know they are needed.
    max_frames:
      # Optional: Default for all object types (default: not set, track forever)
      default: 3000
      # Optional: Object specific values
      objects:
        person: 1000
# Optional: Object configuration
# NOTE: Can be overridden at the camera level
objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
# Optional: Record configuration
# NOTE: Can be overridden at the camera level
record:
  # Optional: Enable recording (default: shown below)
  # WARNING: If recording is disabled in the config, turning it on via
  #          the UI or MQTT later will have no effect.
  # WARNING: Frigate does not currently support limiting recordings based
  #          on available disk space automatically. If using recordings,
  #          you must specify retention settings for a number of days that
  #          will fit within the available disk space of your drive or Frigate
  #          will crash.
  enabled: True
  # Optional: Number of minutes to wait between cleanup runs (default: shown below)
  # This can be used to reduce the frequency of deleting recording segments from disk if you want to minimize i/o
  expire_interval: 60
  # Optional: Retention settings for recording
  retain:
    # Optional: Number of days to retain recordings regardless of events (default: shown below)
    # NOTE: This should be set to 0 and retention should be defined in events section below
    #       if you only want to retain recordings of events.
    days: 0
    # Optional: Mode for retention. Available options are: all, motion, and active_objects
    #   all - save all recording segments regardless of activity
    #   motion - save all recordings segments with any detected motion
    #   active_objects - save all recording segments with active/moving objects
    # NOTE: this mode only applies when the days setting above is greater than 0
    mode: all
  # Optional: Event recording settings
  events:
    # Optional: Maximum length of time to retain video during long events. (default: shown below)
    # NOTE: If an object is being tracked for longer than this amount of time, the retained recordings
    #       will be the last x seconds of the event unless retain->days under record is > 0.
#    max_seconds: 300
    # Optional: Number of seconds before the event to include (default: shown below)
    pre_capture: 2
    # Optional: Number of seconds after the event to include (default: shown below)
    post_capture: 2
    # Optional: Objects to save recordings for. (default: all tracked objects)
    objects:
      - person
    # Optional: Restrict recordings to objects that entered any of the listed zones (default: no required zones)
    required_zones: []
    # Optional: Retention settings for recordings of events
    retain:
      # Required: Default retention days (default: shown below)
      default: 10
      # Optional: Mode for retention. (default: shown below)
      #   all - save all recording segments for events regardless of activity
      #   motion - save all recordings segments for events with any detected motion
      #   active_objects - save all recording segments for event with active/moving objects
      #
      # NOTE: If the retain mode for the camera is more restrictive than the mode configured
      #       here, the segments will already be gone by the time this mode is applied.
      #       For example, if the camera retain mode is "motion", the segments without motion are
      #       never stored, so setting the mode to "all" here won't bring them back.
      mode: motion
      # Optional: Per object retention days
      objects:
        person: 2

[NickM-27]

There are a number of issues I see here:

1. it is discouraged to copy and pate the reference config as there are a number of things there that are not the default and not ideal. I would suggest starting over and just going through the getting started guide https://docs.frigate.video/guides/getting_started
2. Check the cable on the coral, many users have had this problem because the included cable was bad; try a different one
3. try different ports on the device to see if it makes a difference

[CWuestefeld]

OK, I've pared down the config to what I think is the bare minimum, shown below. I replaced the USB cable with a brand new one. And I plugged that into a different USB port (on the back rather than a front port).

None of this has changed the situation.

mqtt:
  enabled: False  #  host: mqtt.server.com #Please replace with a valid mqtt broker
detectors:
  # Required: name of the detector
  coral:
    # Required: type of the detector
    # Valid values are 'edgetpu' (requires device property below) and 'cpu'.
    type: edgetpu
    # Optional: device name as defined here: https://coral.ai/docs/edgetpu/multiple-edgetpu/#using-the-tensorflow-lite-python-api
    device: usb
cameras:
  front:
    ffmpeg:
      hwaccel_args: " "
      inputs:
        - path: rtsp://192.168.0.124:554/?user=admin&password=&channel=1&stream=0.sdp #replace with your RTSP camera url
          roles:
            - detect
            - rtmp
    detect:
      width: 2592
      height: 1944
  

[NickM-27]

in that case the most likely culprit is that your motherboard is failing to provide enough power to the coral, corals require a considerable amount of power and SBCs as well as some PCs fail to provide that power. A powered USB hub would be the next thing to try

[CWuestefeld]

I had a spare powered hub sitting around, so I tried this. But it makes the problem worse, the Coral isn't recognized at all, I just get errors. I'm not sure what that problem might be. The ports are blue, leading me to believe they're USB3. And even if they're not, Coral is still USB2 compatible at reduced performance, right? So at the moment I don't have a definitive answer here.

[NickM-27]

Coral is still USB2 compatible at reduced performance, right?

yes, if you have a non-blue USB 2 port might be worth trying that since the reduced performance would also mean reduced power usage.

Other than that, you might try running it as you have been but check the syslogs to see if there are any usb errors.

[Saduff]

Having the same issue. I've had the Coral connected to a red USB port for a while, but found out it's actually USB 2.0. Inference speed was around 37 ms. Detection has been getting stuck a couple times a day at random times. I now switched it to a blue USB port and confirmed it's connected as USB 3.0. Inference speed is now 10 ms, but now detection gets stuck constantly over and over, so it doesn't work at all.

Power issue? Should I get a USB 3 hub?

Update: it looks like the original cable included in the Coral package is supposed to be white. Mine is from eBay and the included cable is black. The cable also doesn't look USB 3 (no blue color). I ordered a new USB 3.2 10 Gbps cable. I'll report the results when it arrives.

[NickM-27]

what is your OS / container setup? it does sound like you've tried everything hardware wise

[Saduff]

Hardware

It's an old PC.

• MB: ASUS P9X79 LE
• CPU: Intel i7-3930K

Software

Arch Linux (kernel 6.7.5) on bare metal, Frigate official docker image.

I did some more testing. Here's what I found.

When the Coral is connected to the MB's USB3 port, I get this over and over in the kernel log whenever Frigate's detection process restarts:

reset SuperSpeed USB device number 4 using xhci_hcd
LPM exit latency is zeroed, disabling LPM.

However, when the Coral is connected to the PCIe card with USB-C 3.2 ports, I get the log above, but also this:

xhci_hcd 0000:02:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 1
xhci_hcd 0000:02:00.0: Looking for event-dma 00000000ba0032e0 trb-start 00000000ba002fe0 trb-end 0000000000000000 seg-start 00000000ba002000 seg-end 00000000ba002ff0

This is the same as here: google-coral/edgetpu#615

Many people think virtualization (e.g. Proxmox) may be to blame, but there are some like myself who have the same problem and don't use VMs. This comment sums it up well: #2607 (comment)

When I plug it into a USB2 port, I only get this once in the kernel log, no message about LPM:

reset high-speed USB device number 4 using ehci-pci

One more thing. From the Frigate docs:

1. When the device is first plugged in and has not initialized it will appear as 1a6e:089a Global Unichip Corp. when running lsusb or checking the hardware page in HA OS.
2. Once initialized, the device will appear as 18d1:9302 Google Inc. when running lsusb or checking the hardware page in HA OS.

When I run Frigate for the first time after plugging in the Coral, it crashes with the following error:

ValueError: Failed to load delegate from libedgetpu.so.1.0

Then I can see with lsusb that the Coral has initialized (Google Inc.) and Frigate starts up normally after being restarted by docker.

But is this normal and how it's supposed to be that it crashes the first time? Or should the Coral be initialized by the host beforehand? In that case, how? AFAIK, the Coral won't initialize until running inference.

I also tried turning detection off for all but one camera, but it made no difference.

[Saduff]

I also tested a powered USB3 hub now and it's somewhat working. Detection gets stuck every 1-11 minutes now, totally random. Could the Frigate watchdog be too restrictive? Like what if the Coral usually provides a response in 10 ms, but sometimes spikes to 20-30 ms. Would the watchdog think it's stuck then? I haven't looked at the code, so not sure how it works.

[greyman12]

I have been having this same issue but the frequency has been every 3 days or so. So far I have tried 2 hubs and I have replaced the cable. This is running bare metal Debian 12 on an intel nuc but I have tested on another pc with the same result.

[Saduff]

I gave up on the USB Coral and got the mPCIe version. So far so good, inference speed is around 7 ms which is 5 times faster than USB2 Coral. Haven't run it long enough yet to determine if it's stable unlike the USB version, but no issues so far.

[greyman12]

I am so close to doing the same, I just wonder what is happening. As I said before I see it about every 3 days, there is nothing in the logs prior to frigates watchdog process basically force resetting the usb device. If there was some further debug that could be turned on that might give more of a clue. I have seen this issue reported previously but never really an answer other change the cable or add a hub.

[greyman12]

I haven’t tried that as yet and don’t really want to take the hit on the detection speed, so just putting up with it resetting every couple of days and continuing to research.

[anktababa]

Hi folks,
Just found this thread, as it seems i have the same issue, every 2-3minutes.
Just installed the coral usb on my NUC gen 11. (Frigate container inside a VM running on proxmox .)
I'll try to replace the usb cable and hoping that it will the issue. Do you know if a coral might need to be updated ?

2024-04-05 10:23:55.098986177  [2024-04-05 10:23:55] root                           INFO    : Detection process didnt exit. Force killing...
2024-04-05 10:23:55.109830923  [2024-04-05 10:23:55] root                           INFO    : Detection process has exited...
2024-04-05 10:23:55.166534234  [2024-04-05 10:23:55] detector.coral                 INFO    : Starting detection process: 1266
2024-04-05 10:23:58.646159724  [2024-04-05 10:23:55] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-04-05 10:23:58.655164143  [2024-04-05 10:23:58] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-04-05 10:25:05.121230005  [2024-04-05 10:25:05] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting 

[Saduff]

I tried different cables and a powered USB hub, but nothing helped, so I switched to the mPCIe Coral and don't have any issues anymore.

[anktababa]

In case it can help other people.
After trying everything regarding the USB/Cable i decided to go away from a VM and USB passthrough (tried the entire port as recommended but also the spicy, same result) and use LXC container directly on proxmox.
Since that move it seems that my TPU is not disconnecting anymore. Stable since ;

s6-rc: info: service legacy-services successfully started
2024-04-05 15:40:45.908563381  [2024-04-05 15:40:43] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-04-05 15:40:45.914347600  [2024-04-05 15:40:45] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
root@FrigateLXC:~/frigate# 

While it was disconnecting everything few minutes, now it is stable for 4hours so it seems to be root cause, now clue why but at least it's fix the issue.

[Saduff]

In my case, I was running the Frigate container on bare metal, so I had a different issue. 4 hours is not enough to be sure. For me with USB2, the disconnects happened roughly a couple times per day, so there were periods where it would run fine for like 12 hours straight until it disconnects. But when Coral was connected to USB3, then I had the same experience as you (i.e. completely unusable).

But hopefully the issue is fixed for you. For me, the mPCIe Coral has been rock solid, not a single disconnect ever.

[greyman12]

As @Saduff said 4 hours is not enough, I have tried so many things the longest period of time between a reset I saw was 6 days. I have given up also an ordered the mPCIe version.

[anktababa]

hi folks,
You were right ... unfortunately for me

2024-04-05 15:40:45.908563381  [2024-04-05 15:40:43] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-04-05 15:40:45.914347600  [2024-04-05 15:40:45] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-04-11 12:30:09.645662378  [2024-04-11 12:30:09] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-04-11 12:30:12.305626610  [2024-04-11 12:30:12] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-04-11 15:51:26.031821316  [2024-04-11 15:51:26] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-04-11 15:51:28.773951691  [2024-04-11 15:51:28] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-04-11 22:32:09.079028255  [2024-04-11 22:32:06] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-04-11 22:32:09.083442916  [2024-04-11 22:32:09] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

No idea why after one week and why many times same day and not today but the conlusion ; not stable.

[flip1900]

does anyone have updates on this issue?
i have frigate running on a intel nuc 11th gen with coral connected to usb 3.0
running frigate on docker with ubuntu as my os.
also randomly having this "detection appears to be stuck" issue, every few hours or so