[Support]: No MQTT event for top scoring snapshot?

[sshaikh]

Describe the problem you are having

I am processing MQTT update events in HA. One of the conditions I set is to only be interested in an update if the score gets improved upon. I establish this by checking if after/score is equal to the top score.

However I have a trace of MQTT events that don't seem to contain the event that generated the top score so this filtering skips all of the updates.

Version

0.14.1-f4f3cfa

What browser(s) are you using?

firefox

Frigate config file

na

Relevant Frigate log output

na

Relevant go2rtc log output

na

FFprobe output from your camera

na

Frigate stats

No response

Install method

Docker Compose

docker-compose file or Docker CLI command

na

Object Detector

Coral

Network connection

Wired

Camera make and model

na

Screenshots of the Frigate UI's System metrics pages

No response

Any other information that may be helpful

❯ mqttui log "frigate/events" -b mqtt://mqtt -u user --password xxx | grep doorbell > dump.txt
❯ grep uatsm dump.txt
20:30:44.161 QoS:AtMostOnce  frigate/events                                     Payload(1296): {"before":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158643.737997,"snapshot":null,"label":"person","sub_label":null,"top_score":0,"false_positive":true,"start_time":1735158643.737997,"end_time":null,"score":0.74609375,"box":[331,216,370,288],"area":2808,"ratio":0.5416666666666666,"region":[250,92,570,412],"stationary":false,"motionless_count":0,"position_changes":0,"current_zones":[],"entered_zones":[],"has_clip":false,"has_snapshot":false,"attributes":{},"current_attributes":[]},"after":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158643.916881,"snapshot":{"frame_time":1735158643.916881,"box":[332,216,372,290],"area":2960,"region":[186,92,506,412],"score":0.734375,"attributes":[]},"label":"person","sub_label":null,"top_score":0.734375,"false_positive":false,"start_time":1735158643.737997,"end_time":null,"score":0.734375,"box":[332,216,372,290],"area":2960,"ratio":0.5405405405405406,"region":[186,92,506,412],"stationary":false,"motionless_count":1,"position_changes":0,"current_zones":[],"entered_zones":[],"has_clip":false,"has_snapshot":false,"attributes":{},"current_attributes":[]},"type":"new"}
20:30:44.907 QoS:AtMostOnce  frigate/events                                     Payload(1450): {"before":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158643.916881,"snapshot":{"frame_time":1735158643.916881,"box":[332,216,372,290],"area":2960,"region":[186,92,506,412],"score":0.734375,"attributes":[]},"label":"person","sub_label":null,"top_score":0.734375,"false_positive":false,"start_time":1735158643.737997,"end_time":null,"score":0.734375,"box":[332,216,372,290],"area":2960,"ratio":0.5405405405405406,"region":[186,92,506,412],"stationary":false,"motionless_count":1,"position_changes":0,"current_zones":[],"entered_zones":[],"has_clip":false,"has_snapshot":false,"attributes":{},"current_attributes":[]},"after":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158644.827651,"snapshot":{"frame_time":1735158644.827651,"box":[340,213,381,312],"area":4059,"region":[192,94,512,414],"score":0.76953125,"attributes":[]},"label":"person","sub_label":null,"top_score":0.74609375,"false_positive":false,"start_time":1735158643.737997,"end_time":null,"score":0.76953125,"box":[340,213,381,312],"area":4059,"ratio":0.41414141414141414,"region":[192,94,512,414],"stationary":false,"motionless_count":0,"position_changes":1,"current_zones":[],"entered_zones":[],"has_clip":false,"has_snapshot":false,"attributes":{},"current_attributes":[]},"type":"update"}
20:30:45.230 QoS:AtMostOnce  frigate/events                                     Payload(1487): {"before":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158644.827651,"snapshot":{"frame_time":1735158644.827651,"box":[340,213,381,312],"area":4059,"region":[192,94,512,414],"score":0.76953125,"attributes":[]},"label":"person","sub_label":null,"top_score":0.74609375,"false_positive":false,"start_time":1735158643.737997,"end_time":null,"score":0.76953125,"box":[340,213,381,312],"area":4059,"ratio":0.41414141414141414,"region":[192,94,512,414],"stationary":false,"motionless_count":0,"position_changes":1,"current_zones":[],"entered_zones":[],"has_clip":false,"has_snapshot":false,"attributes":{},"current_attributes":[]},"after":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158645.016502,"snapshot":{"frame_time":1735158645.016502,"box":[360,203,401,365],"area":6642,"region":[198,109,518,429],"score":0.74609375,"attributes":[]},"label":"person","sub_label":null,"top_score":0.75390625,"false_positive":false,"start_time":1735158643.737997,"end_time":null,"score":0.74609375,"box":[360,203,401,365],"area":6642,"ratio":0.25308641975308643,"region":[198,109,518,429],"stationary":false,"motionless_count":1,"position_changes":1,"current_zones":["driveway-zone"],"entered_zones":["driveway-zone"],"has_clip":true,"has_snapshot":true,"attributes":{},"current_attributes":[]},"type":"update"}
20:30:50.302 QoS:AtMostOnce  frigate/events                                     Payload(1507): {"before":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158645.016502,"snapshot":{"frame_time":1735158645.016502,"box":[360,203,401,365],"area":6642,"region":[198,109,518,429],"score":0.74609375,"attributes":[]},"label":"person","sub_label":null,"top_score":0.75390625,"false_positive":false,"start_time":1735158643.737997,"end_time":null,"score":0.74609375,"box":[360,203,401,365],"area":6642,"ratio":0.25308641975308643,"region":[198,109,518,429],"stationary":false,"motionless_count":1,"position_changes":1,"current_zones":["driveway-zone"],"entered_zones":["driveway-zone"],"has_clip":true,"has_snapshot":true,"attributes":{},"current_attributes":[]},"after":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158648.873943,"snapshot":{"frame_time":1735158647.026484,"box":[494,180,638,477],"area":42768,"region":[0,0,684,684],"score":0.83984375,"attributes":[]},"label":"person","sub_label":null,"top_score":0.822265625,"false_positive":false,"start_time":1735158643.737997,"end_time":null,"score":0.5,"box":[537,119,639,473],"area":36108,"ratio":0.288135593220339,"region":[276,116,640,480],"stationary":false,"motionless_count":1,"position_changes":1,"current_zones":["driveway-zone"],"entered_zones":["driveway-zone"],"has_clip":true,"has_snapshot":true,"attributes":{},"current_attributes":[]},"type":"update"}
20:30:54.412 QoS:AtMostOnce  frigate/events                                     Payload(1507): {"before":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158648.873943,"snapshot":{"frame_time":1735158647.026484,"box":[494,180,638,477],"area":42768,"region":[0,0,684,684],"score":0.83984375,"attributes":[]},"label":"person","sub_label":null,"top_score":0.822265625,"false_positive":false,"start_time":1735158643.737997,"end_time":null,"score":0.5,"box":[537,119,639,473],"area":36108,"ratio":0.288135593220339,"region":[276,116,640,480],"stationary":false,"motionless_count":1,"position_changes":1,"current_zones":["driveway-zone"],"entered_zones":["driveway-zone"],"has_clip":true,"has_snapshot":true,"attributes":{},"current_attributes":[]},"after":{"id":"1735158643.737997-uatsm0","camera":"doorbell","frame_time":1735158648.873943,"snapshot":{"frame_time":1735158647.026484,"box":[494,180,638,477],"area":42768,"region":[0,0,684,684],"score":0.83984375,"attributes":[]},"label":"person","sub_label":null,"top_score":0.822265625,"false_positive":false,"start_time":1735158643.737997,"end_time":1735158654.269696,"score":0.5,"box":[537,119,639,473],"area":36108,"ratio":0.288135593220339,"region":[276,116,640,480],"stationary":false,"motionless_count":1,"position_changes":1,"current_zones":["driveway-zone"],"entered_zones":["driveway-zone"],"has_clip":true,"has_snapshot":true,"attributes":{},"current_attributes":[]},"type":"end"}

[sshaikh]

Perhaps beyond the scope of this question but my intention is this:

1. The first time a person enters the driveway_zone, send the snapshot.
2. On further updates, only send an update if the snapshot (ie detection) score has improved.

Since these are being processed in HA, I only have the MQTT state for both determinations. The scoring has stumped me (I'm pretty sure this worked as expected before the recent MQTT changes).

[NickM-27]

An update is published to /events when the object has a better snapshot saved

[NickM-27]

It depends on the config. If a user has a model with face as a detectable object then it chooses the best face. Same with license_plate / car.

Otherwise it's based on the score and size of the bounding box.

[sshaikh]

Perfect. So the idea is to just keep taking (guaranteed better) updates as they get sent and not dwell on the details? That sounds reasonable.

The issue is that with a consumer like HA there isn't much (any?) state kept between mqtt events so it's difficult to update an already sent notification. But I'll have a think about that now.

Do all objects have at most 1 new and end event, and zero or more updates in between?

[sshaikh]

Also, is there a reason why event 1's before score isn't the top score until event 2?

And why event 2's after score isn't ever seen as a top score?

[NickM-27]

Every object that is not a false positive is guaranteed to have a single new and end event, and likely at least one update.

[NickM-27]

Score is the score at that specific moment, top_score is the best median score for the object.