[Detector Support]:

[jupytertemi]

Describe the problem you are having

We are trying to use a custom model (built on the Azure Custom Vision service) on our frigate service, as we have found that DeepStack does not quite meet the accuracy levels we expect.
We have trained our models with specific courier company logos and named faces for facial recognition.

We saved the .tflite and label.txt files within the same location as our frigate folders (we run Linux Debian Bookworm OS).
However, when we update the config.yaml file with the detector settings for the custom model, it breaks frigate. The specific error message and config.yml entries are as shown below.

In summary, the error expects a FLOAT32, which is the same format we exported our Azure-trained model as, but it appears that frigate sees the model as a UINT8, which it isn’t.
We have inspected the .tflite file to validate it is indeed in FLOAT32.
We will be eternally grateful if you could offer any guidance or support, as we have exhausted every avenue of research or approach to solving this problem, with no luck at all.

Error Message

ValueError: Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: Placeholder

Config.yml entry for custom model

detectors:

custom_model:

type: cpu

model:

  path: /home/admin/frigate/custom_model/model.tflite

  width: 416

  height: 416

  input_tensor: nhwc

  input_pixel_format: rgb

  labelmap_path: /home/admin/frigate/custom_model/labelmap.txt

objects:

track:

- person

- auspost

- cecilia

- "courier please"

- fiona

- kathy

- startrack

- david

- toll

model:

labelmap:

0: car

1: person

2: car

3: person

4: person

5: car

6: person

7: car

Version

Frigate v3.9

Frigate config file

# yaml-language-server: schema=http://ccab4aaf-frigate-fa:5000/api/config/schema.json

mqtt:
  host: 192.168.0.63
  user: admin  # Replace with your MQTT username
  password: [PASSWORD]  # Replace with your MQTT password
  topic_prefix: frigate

audio: # <- enable audio events for all camera
  enabled: True

cameras:
  GarageCamera:
    ffmpeg:
      hwaccel_args: preset-rpi-64-h264
      inputs:
        - path: rtsp://username:password@192.168.0.117:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record 
            - audio         
          input_args: -analyzeduration 1000M -probesize 1000M
        - path: rtsp://username:password@192.168.0.117:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
            - rtmp
          input_args: -analyzeduration 1000M -probesize 1000M
    detect:
      width: 640
      height: 360
      fps: 5

    record:
      enabled: True
      retain:
        days: 7
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      retain:
        default: 10
  
detectors:
  custom_model:
    type: cpu
    model:
      path: /home/admin/frigate/custom_model/model.tflite
      width: 416
      height: 416
      input_tensor: nhwc
      input_pixel_format: rgb
      labelmap_path: /home/admin/frigate/custom_model/labelmap.txt

objects:
  track:
    - person
    - auspost
    - cecilia
    - "courier please"
    - fiola
    - katherine
    - startrack
    - temi
    - toll

model:
  labelmap:
    0: car
    1: person
    2: car
    3: person
    4: person
    5: car
    6: person
    7: car

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-standard-arm64
    volumes:
      - /home/admin/frigate/config:/config
      - /home/admin/frigate/storage:/media/frigate
      - /home/admin/frigate/custom_model:/home/admin/frigate/custom_model:ro
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 2000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds

Relevant log output

2024-06-09 20:01:22.279261040  [2024-06-09 20:01:22] frigate.detectors              WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
2024-06-09 20:01:22.280505780  Process detector:custom_model:
2024-06-09 20:01:22.281484651  Traceback (most recent call last):
2024-06-09 20:01:22.281542318    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-06-09 20:01:22.281544429      self.run()
2024-06-09 20:01:22.281546132    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-06-09 20:01:22.281547373      self._target(*self._args, **self._kwargs)
2024-06-09 20:01:22.281568040    File "/opt/frigate/frigate/object_detection.py", line 125, in run_detector
2024-06-09 20:01:22.281569855      detections = object_detector.detect_raw(input_frame)
2024-06-09 20:01:22.281571336    File "/opt/frigate/frigate/object_detection.py", line 75, in detect_raw
2024-06-09 20:01:22.281572725      return self.detect_api.detect_raw(tensor_input=tensor_input)
2024-06-09 20:01:22.281574262    File "/opt/frigate/frigate/detectors/plugins/cpu_tfl.py", line 41, in detect_raw
2024-06-09 20:01:22.281575818      self.interpreter.set_tensor(self.tensor_input_details[0]["index"], tensor_input)
2024-06-09 20:01:22.281583669    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 572, in set_tensor
2024-06-09 20:01:22.281584966      self._interpreter.SetTensor(tensor_index, value)
2024-06-09 20:01:22.281587521  ValueError: Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: Placeholder

Operating system

Debian

Install method

Docker Compose

Coral version

CPU (no coral)

Any other information that may be helpful

No response

[NickM-27]

there may be some issue with floats in that part, @blakeblackshear may be able to answer this better than I can

but I want to note that even if the model was accepted you would not see object detections. This is because your labelmap and object -> track list do not match

[jupytertemi]

Thanks for replying, Nick.
Can you explain further? I am a little lost on how the labelmap works. The model I built exported with a label txt file (not a label map). I had to create one myself, but I admit I built the labelmap without a full udnerstanding of how it works.

i'd appreciate any guidance.

Thank you.

[blakeblackshear]

The error message is saying that your model is expecting float32, but Frigate is passing a uint8 input.

Your model doesn't have the inputs (and probably outputs) that are expected by Frigate.

I would start by developing a simple python script that runs your model on a test image. Then compare that to what Frigate's source code does.

You will either need to modify Frigate with a custom detector or modify your model to align with the expected inputs and outputs.

[serapme]

have the same issue ,the model is converted from pytorch model