[Detector Support]: Frigate+ 2024.3 detection oddity

[csreynolds]

Describe the problem you are having

Consistantly since trying 2024.3, several detections seem to have been mapped incorrectly.

Most notable, 'licence_plate' gets detected as 'cat' and 'ups' gets detected as 'bear'. Another oddity is 'squirrel' gets detected as 'bird' pretty regularly but also sometimes 'umbrella', so not sure about this one.

I thought this was due to low training #s but wasn't sure until suggestions in frigate+ were available. The suggestions are spot on every time.

I've been using the tensorrt docker image with a P4000, a P2000 and 2x minipcii coral bridge things from here. Not sure multiple corals is related to the problem.

Happy to put things in debug for logs if needed, just need to know how.

Version

0.15.0-869fa26

Frigate config file

mqtt:
  host: mqtt
database:
  path: /db/frigate.db

detectors:
  coral_pci0:
    type: edgetpu
    device: pci:0
  coral_pci1:
    type: edgetpu
    device: pci:1
  coral_pci2:
    type: edgetpu
    device: pci:2
  coral_pci3:
    type: edgetpu
    device: pci:3
    model:
      path: plus://[PLUSCODE]

semantic_search:
  enabled: true
  model_size: large
  reindex: false

birdseye:
  enabled: true
  restream: false
  mode: continuous
  quality: 5

detect:
  stationary:
    interval: 5000
    threshold: 45

objects:
  track:
    - bbq_grill
    - bear
    - bicycle
    - bird
    - boat
    - car
    - cat
    - cow
    - deer
    - dhl
    - dog
    - face
    - fedex
    - fox
    - goat
    - horse
    - license_plate
    - motorcycle
    - package
    - person
    - rabbit
    - raccoon
    - robot_lawnmower
    - squirrel
    - umbrella
    - ups
    - usps
    - waste_bin

audio:
  enabled: true
  listen:
    - bark
    - fire_alarm
    - scream
    - speech
    - yell
    - bang
    - glass
    - gunshot
    - explosion
    - boom
    - siren
    - door
    - doorbell
    - lawn_mower
    - police_car
    - ambulance
    - fire_engine
    - emergency_vehicle
    - honk

record:
  enabled: true
  retain:
    days: 14
  alerts:
    retain:
      days: 14
    pre_capture: 10
    post_capture: 10
  detections:
    retain:
      days: 14
    pre_capture: 10
    post_capture: 10
snapshots:
  enabled: true
  retain:
    default: 14
    objects:
      person: 30
      car: 30

go2rtc:
  streams:
    Doorbell:
      - rtspx://192.168.7.1:7441/yGZZwZ0jUMzda7h1
    Doorbell_package:
      - rtspx://192.168.7.1:7441/XyS63nB06PWEtgJd
  webrtc:
    candidates:
      - 192.168.7.201:8555
      - stun:8555

cameras:
  Garage:
    enabled: true
    birdseye:
      mode: objects
    objects:
      track:
        - person
        - cat
        - dog
        - bird
        - face
    ffmpeg:
      input_args: preset-rtsp-restream
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
        - path: rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - detect
            - audio
    motion:
      mask: 0.816,0.043,0.816,0.073,0.969,0.071,0.969,0.043
  FrontDoor:
    enabled: true
    motion:
      threshold: 40
      contour_area: 30
      improve_contrast: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Doorbell
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-ubiquiti
    objects: {}
  FrontDoor_PackageCam:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/Doorbell_package
          roles:
            - record
            - detect
            - audio
  FrontDeck:
    enabled: true
    objects:
      track:
        - car
        - person
        - face
        - cat
        - dog
        - bird
    ffmpeg:
      input_args: preset-rtsp-restream
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
        - path: rtsp://viewonly:viewonly12@192.168.7.182:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - detect
            - audio
    motion:
      mask: 0.813,0.072,0.972,0.072,0.971,0.041,0.813,0.04
  BackYard:
    enabled: true
    ffmpeg:
      input_args: preset-rtsp-restream
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
        - path: rtsp://viewonly:viewonly12@192.168.7.183:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - detect
            - audio
    motion:
      mask: 0.816,0.041,0.817,0.071,0.97,0.074,0.969,0.042
  Driveway:
    enabled: true
    ffmpeg:
      input_args: preset-rtsp-restream
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
        - path: rtsp://viewonly:viewonly12@192.168.7.184:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - detect
            - audio
    motion:
      mask: 0.014,0.943,0.015,0.974,0.165,0.973,0.165,0.945
      threshold: 40
      contour_area: 30
      improve_contrast: true
version: 0.15-0

docker-compose file or Docker CLI command

services:
   frigate:
     container_name: frigate
     # add groups from host for render, plugdev, video
     group_add:
       - "998" # render
       - "39"  # video
     shm_size: "16384mb"
     image: ghcr.io/blakeblackshear/frigate:869fa26-tensorrt
     deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              #              device_ids: ['0']
              capabilities: [gpu]
              count: all
     devices:
       - "/dev/apex_0:/dev/apex_0"
       - "/dev/apex_1:/dev/apex_1"
       - "/dev/apex_2:/dev/apex_2"
       - "/dev/apex_3:/dev/apex_3"
       - "/dev/dri/:/dev/dri/"
     environment:
       - PGID=${PGID}
       - PUID=${PUID}
       - TZ=${TZ}
       - NVIDIA_DRIVER_CAPABILITIES=all
       - NVIDIA_VISIBLE_DEVICES=all
       - PLUS_API_KEY=${FRIGATE_PLUS_API_KEY}
       - YOLO_MODELS=${FRIGATE_YOLO_MODELS}
       - TRT_MODEL_PREP_DEVICE=0
       - change=true
     logging:
       driver: json-file
       options:
         max-file: ${DOCKERLOGGING_MAXFILE}
         max-size: ${DOCKERLOGGING_MAXSIZE}
     privileged: true
     ports:
        - '5000:5000/tcp'
        - '5001:5001/tcp'
        - '8554:8554'
        - '8971:8971'
     volumes:
       - /etc/localtime:/etc/localtime:ro
       - /dev/bus/usb:/dev/bus/usb
       - ${DOCKERCONFDIR}/frigate/config:/config
       - ${DOCKERCONFDIR}/frigate/db/:/db/ 
       - /bigstorage/frigatemedia:/media/frigate

   mqtt:
     container_name: mqtt
     image: eclipse-mosquitto:1.6
     ports:
       - "1883:1883"

Relevant Frigate log output

No errors relevant in log, Can provide detector debug if I can figure out how to show that.

Install method

Docker Compose

Object Detector

Coral

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[csreynolds]

In my searches initially, I thought this might be related to #15421 but I am not using the yolo model.

Maybe has something to do with that though. Not sure of the inner workings here.

[NickM-27]

You should be applying the model globally not only on the last detector. This is a config issue.

[csreynolds]

Ah this is a case of carrying my config from 0.12 through everything until now.

I had simply pasted the coral detectors in place where the old tensorrt section was when I got them and the indent lived on.

I moved the 'model:' section above the detectors and there have been no licence_plate cats in the last 10min.

Thanks for the extra set of eyes Nick! Sorry about the bother.

Working config snippet:

model:
  path: plus://[PLUS_MODEL_ID]

detectors:
  coral_pci0:
    type: edgetpu
    device: pci:0
  coral_pci1:
    type: edgetpu
    device: pci:1
  coral_pci2:
    type: edgetpu
    device: pci:2
  coral_pci3:
    type: edgetpu
    device: pci:3
#    model:
#      path: /edgetpu_model.tflite

#  tensorrt:
#    type: tensorrt
#    device: 0 #This is the default, select the first GPU
#    model:
#      path: /config/model_cache/tensorrt/yolov7-640.trt
#      input_tensor: nchw
#      input_pixel_format: rgb
#      width: 640
#      height: 640