0.15.0 Beta 3

[blakeblackshear]

Beta Documentation: https://deploy-preview-13787--frigate-docs.netlify.app/

Images

• ghcr.io/blakeblackshear/frigate:0.15.0-beta3
• ghcr.io/blakeblackshear/frigate:0.15.0-beta3-standard-arm64
• ghcr.io/blakeblackshear/frigate:0.15.0-beta3-tensorrt
• ghcr.io/blakeblackshear/frigate:0.15.0-beta3-tensorrt-jp4
• ghcr.io/blakeblackshear/frigate:0.15.0-beta3-tensorrt-jp5
• ghcr.io/blakeblackshear/frigate:0.15.0-beta3-rk
• ghcr.io/blakeblackshear/frigate:0.15.0-beta3-rocm
• ghcr.io/blakeblackshear/frigate:0.15.0-beta3-h8l

Changes since Beta 2

• SHM refactor and bugfixes
• Add ability to bulk delete tracked objects in Explore
• Add sorting to Explore
• Fix bug where audio events would not show in Explore
• Fix bug that caused a crash when creating external events
• Refactor cleanup to consider review severity
• Other minor UI fixes and tweaks

Major Changes for 0.15.0

Breaking Changes

There are several breaking changes in this release, Frigate will attempt to update the configuration automatically. In some cases manual changes may be required. It is always recommended to back up your current config and database before upgrading:

1. Simply copy your current config file to a new location
2. Stop Frigate and make a copy of the frigate.db file

• Rewritten SHM system
  • The SHM system has been rewritten to be more stable. This means that a large % of shared memory (SHM) will be used by default but the total usage will not fluctuate, and Frigate will not fail from SHM overflowing. However, this also means that having the correct SHM size for your cameras, detector, and hardware is more important to having a smooth experience with Frigate. A new calculation is done on startup, so if your shm_size is too low then a warning will be printed in the log stating that it needs to be increased.
• Recording configuration changed
  • The record config has been refactored to allow for direct control of how long alerts and detections are retained. These values will be automatically populated from your current config, but you may want to adjust the values that are set after updating. See the updated docs here and ensure your updated config retains the footage you want it to.
• ffmpeg updates
  • ffmpeg has been updated to version 7. Some Intel users may need to change what hwaccel preset they are using (preset-vaapi may need to now be preset-intel-qsv-h264 or preset-intel-qsv-h265) if camera feeds are not functioning correctly after upgrading. This may need to be adjusted on a per-camera basis. If a qsv preset is not working properly, you may still need to use a preset-vaapi or revert to the previous ffmpeg version as described below.
  • The new version provides many improvements, but the old version can still be run if needed by specifying path: "5.0" in your ffmpeg: config entry. For example:

    ffmpeg:
      path: "5.0"
    

  • ffmpeg presets have been updated. It is always recommended to use ffmpeg presets rather than manual arguments.
  • ffmpeg is no longer part of $PATH. In most cases this is handled automatically.
    • go2rtc exec streams will need to add the full path for ffmpeg which in most cases will be /usr/lib/ffmpeg/7.0/bin/ffmpeg.
    • If the go2rtc config has bin: ffmpeg defined, it needs to be removed.
• Submitting to Frigate+
  • The Frigate+ view in the UI has been replaced by the new Explore view. See the notes below.

Explore

The new Explore pane in Frigate 0.15 makes it easy to explore every object tracked by Frigate. It offers a variety of filters and supports keyword and phrase-based text search, searching for similar images, and searching through descriptive text generated by AI models.

The default Explore pane shows a summary of your most recent tracked objects organized by label. Clicking the small arrow icon at the end of the list will bring you to an infinitely scrolling grid view. The grid view can also be set as the default by changing the view type from the Settings button in the top right corner of the pane.

The Explore pane also serves as the new way to submit images to Frigate+. Filters can be applied to only display tracked objects with snapshots that have not been submitted to Frigate+. The left/right arrow keys on the keyboard allow quick navigation between tracked object snapshots when looking at the Tracked Object Details pane from the grid view.

AI/ML Search

Frigate 0.15 introduces two powerful search features: Semantic Search and GenAI Search. Semantic Search can be enabled on its own, while GenAI Search works in addition to Semantic Search.

Semantic Search

Semantic Search uses a CLIP model to generate embeddings (numerical representations of images) for the thumbnails of your tracked objects, enabling searches based on text descriptions or visual similarity. This is all done locally.

For instance, if Frigate detects and tracks a car, you can use similarity search to see other instances where Frigate detected and tracked that same car. You can also quickly search your tracked objects using an "image caption" approach. Searching for "red car driving on a residential street" or "person in a blue shirt walking on the sidewalk at dawn" or even "a person wearing a black t-shirt with the word 'SPORT' on it" will produce some stunning results.

Semantic Search works by running an AI model locally on your system. Small or underpowered systems like a Raspberry Pi will not run Semantic Search reliably or at all. A dedicated GPU and 16GB of RAM is recommended for best performance.

See the Semantic Search docs for system requirements, setup instructions, and usage tips.

Generative AI

GenAI Search employs generative AI models to create descriptive text for the thumbnails of your tracked objects, which are stored in the Frigate database to enhance future searches. Supported providers include Google Gemini, Ollama, and OpenAI, so you can choose whether you want to send data to the cloud or use a locally hosted provider.

See the GenAI docs for setup instructions and use case suggestions.

Improved Tools for Debugging

Review Item Details Pane

A new Review Item Details pane can be viewed by clicking / tapping on the gray chip on a review item in the Review pane. This shows more information about the review item as well as thumbnails or snapshots for individual objects (if enabled). The pane also provides links to share the review item, download it, submit images to Frigate+, view object lifecycles, and more.

Object Lifecycle Pane

The Recordings Timeline from Frigate 0.13 has been improved upon and returns to 0.15 as the Object Lifecycle, viewable in the Review Details pane as well as the new Explore page. The new pane shows the significant moments during the object's lifecycle: when it was first seen, when it entered a zone, became stationary, etc. It also provides information about the object's area and size ratio to assist in configuring Frigate to tune out false positives.

Native Notifications

Frigate now supports notifications using the WebPush protocol. This allows Frigate to deliver notifications to devices that have registered to receive notifications in the Frigate settings, delivering them in a timely and secure manner. Currently, notifications will be delivered for all review items marked as alerts. More options for native notifications will be supported in the future.

See the notifications docs.

New Object Detectors

ONNX

ONNX is an open model standard which allows for a single model format that can run on different types of GPUs. The default, tensorrt, and rocm Frigate build variants include GPU support for efficient object detection via ONNX models, simplifying configuration and support more models. There are no default included ONNX models for object detection.

AMD MiGraphX

Support has been added for AMD GPUs via ROCm and MiGraphX. Currently there is no default included model for this detector.

Hailo-8

Support has been added for the Hailo8 and Hailo-8L hardware for object detection on both arm64 and amd64 platforms.

Other UI Changes

• Rework the Storage metrics pane to be more clear how much storage each camera is using
• Camera stream info button has been added to the camera metrics page allowing easy access to see camera codec information
• Add button for downloading the full set of logs
• The timeline handlebar will now automatically scroll while viewing recordings
• Review items and exports can now be shared via your browser's sharing pane or via direct link url. A secure context is required for the browser/OS sharing pane to function
• Add the ability to rename camera groups
• Add middle mouse button support in the Review pane to open full recordings view in a new tab
• Add Nvidia GPU encode and decode stats to system metrics
• New desktop keyboard shortcuts. In addition to the numerous keyboard shortcuts available with 0.14, 0.15 adds several more. In Live view, the numeric keys (1-9) will move PTZ cameras to presets. In Review, Page Up and Page Down will scroll through review items
• Accessibility updates for users with screen readers

Other Backend Changes

• FFMPEG updated to 7.0
• OpenVINO updated to 2024.3
• Use FastAPI instead of Flask for the Frigate API
• Add service manager infrastructure
• Install onevpl tools and update drivers for Intel
• If record is disabled for a camera then that camera will not save frames to the preview during that time
• Fix stalled connections to certain onvif cameras
• Fix autotracking absolute zoom interpolation logic
• TensorRT models are no longer generated by default, the ENV var must be set

---

This discussion was created from the release 0.15.0 Beta 3.

[Damien2407]

Everything is good for me ! Ty for all men !!!!

[bagobones]

Had one old object in Explore view I was finally able to delete with the new delete option..

It was well past retention for alerts / detections / snapshots which all top out at 1mo in my config.

[jdiegmueller]

Found something:

If I browse to /review?id=<review_id>I get caught in a loop, the browser bounces back and forther between /review and /review?id=<review_id>. I provide this URL as a clickAction in notifications.

If I roll back to Docker image ghcr.io/blakeblackshear/frigate:0.15.0-beta2-tensorrt the issue goes away, and then comes back when I switch back to Docker image ghcr.io/blakeblackshear/frigate:0.15.0-beta3-tensorrt.

[bagobones]

I hit this issue from the /export view as well as there is a clock button in that view labeled "Review in History" if you click it it will try and link to the review item ID and start the loop.

[NickM-27]

fixed in #15465

[bagobones]

here is an odd one.. I have an event on the /review alerts screen that shows the deer icon but no matching deer in the event or in explor?

The waste bins also show in the event but not the detail and don't show in explore but it still seems to have some trouble detecting those so I have been working on more samples.

[NickM-27]

it doesn't have anything to do with score and area, that filter applies to every object regardless. This simply depends on your review and event configs

[bagobones]

Ok.. still from a user point of view it is strange I can't see them in "Review Item Details" on this Alert or individually /explore like I can for all other detections.

The only way I can THINK that could happen is that I have a narrow zone along the road filtered by people and car only, the deer false positive would have had to have happened in that zone. I have since removed that limit on that zone all the other zones had no object limits.

In my case my required zones limit both alerts and detections to JUST my yard and that is the ONLY place I expect to see things recorded or alerted on.

[hawkeye217]

To be clear, you won't see a tracked object in Explore or in a review item if it doesn't qualify to be an alert or a detection. You aren't seeing deer in the review item or Explore because you don't have Frigate configured to save snapshots or footage for deer.

[bagobones]

To be clear, you won't see a tracked object in Explore or in a review item if it doesn't qualify to be an alert or a detection. You aren't seeing deer in the review item or Explore because you don't have Frigate configured to save snapshots or footage for deer.

The only way I can think that could happen is if the false positive was outside my required areas, or exactly in a narrow band on the street where I have a zone set to only people and cars, I have required areas for BOTH Alerts and Detections..

I have plenty of deer events for my front yard with snapshots.

[hawkeye217]

That is likely to be the case, then.

I just wrote some code to improve the message in the review details pane and list the labels that were detected but did not create a tracked object.

[dmitriykhait]

saw this in the logs, everything seems to still work, but figured i post it here

2024-12-12 20:53:00.994767280 Exception in thread event_cleanup: 2024-12-12 20:53:00.994780570 Traceback (most recent call last): 2024-12-12 20:53:00.994782544 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner 2024-12-12 20:53:00.997032462 self.run() 2024-12-12 20:53:00.997038996 File "/opt/frigate/frigate/events/cleanup.py", line 334, in run 2024-12-12 20:53:00.997286923 events_with_expired_clips = self.expire_clips() 2024-12-12 20:53:00.997291886 File "/opt/frigate/frigate/events/cleanup.py", line 260, in expire_clips 2024-12-12 20:53:00.997354186 events_to_update.extend([event.id for event in batch]) 2024-12-12 20:53:00.997370872 TypeError: 'Event' object is not iterable

[hawkeye217]

This has already been fixed in the latest dev image: #15465

[dmitriykhait]

sweet, thanks

[igiannakas]

Anyone else getting this error with the latest beta? It doesn’t seem to affect anything at least.

[hawkeye217]

You must be using Proxmox with semantic search enabled.

Other Proxmox users have reported this. It doesn't seem to cause issues.

[igiannakas]

That’s correct yes.

[pdobrien3]

was anything changed in ffmpeg? I have 3 hikvision cameras, all config'd alike and one of them keeps crashing ffmpeg and can't pull a stream. works fine on Beta 2. Any hints on where to look?

front_camera: Unable to read frames from ffmpeg process. is flooding the logs.

ultimately get: Too many unprocessed recording segments in cache for front_camera. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...

[hawkeye217]

I can confirm this is a bug with SHM handling when the camera name ends with an underscore and a number. We'll fix this for the next beta.

[hawkeye217]

Fixed in #15615

[pdobrien3]

thanks. should this be fixed in the next dev version?

[NickM-27]

in the latest, yes

[pdobrien3]

Fixed on my end also. Thank you !!!

[tpjanssen]

@NickM-27 @hawkeye217 something strange... An event with this object lifecycle:

1. "Person Detected" at 22:08:06. Showing a person attempting to enter a door that is slightly open. Timestamp on the camera says 22:08:07.
2. "Person Entering Zone xxxx" at 22:08:06.. This is the same picture as in step 1, only with the zones overlaid on the picture.
3. "Person Left" shows just a portion of an arm closing the door. Timestamp on the camera says 22:08:09.

The strange thing is that the snapshot shown is actually an image taken earlier than the one in step 1. The snapshot was captured 1 second before (22:08:06) the image in step 1, with the door still closed. How is it possible that the snapshot was taken earlier than the image in step 1?

[NickM-27]

Likely some difference between the recording and detect processes

[tpjanssen]

Why is the recording process involved in this?

[NickM-27]

Because that's the only way to get a snapshot for an object at an arbitrary point in time (object lifecycle)

[hawkeye217]

The annotation_offset is a parameter that can help with some of these discrepancies. Adjustable via the config or via the gear icon in the Object Lifecycle pane.

[Calimerorulez]

I just swiched on Generative AI, using Gemini. For a very large amount of responses, Gemini complains about receiving only a still image, and not a sequence:

That's a still image, not a sequence, so analyzing a sequence of actions and predicting future behavior is impossible. We can only speculate on the cyclist's intent based on the single frame.

[Calimerorulez]

The objects are mostly cars and bicycles indeed :)

[hawkeye217]

You can probably avoid Gemini complaining if you changed the default prompt, then.

https://deploy-preview-13787--frigate-docs.netlify.app/configuration/genai/#custom-prompts

[Calimerorulez]

Does increasing the detection FPS make any sense?

[hawkeye217]

Not really. Images are saved during the object's lifecycle at "significant" moments, and increasing the FPS doesn't necessarily guarantee that you will have more of those moments.

If these cars and bicycles are just zooming by your camera, an additional image won't really provide much more insight to the AI anyways, at least in a way that would actually be useful.

I would just change the default prompt to something like: Analyze the image or sequence of images containing the {label}. Focus on the likely intent or behavior of the {label} based on its actions and movement, rather than describing its appearance or the surroundings. Consider what the {label} is doing, why, and what it might do next.

Have you read this tip on using Semantic Search and Generative AI? Many users have found that they have been able to selectively disable or completely disable GenAI after discovering the power of Semantic Search.

[bagobones]

The prompt can likely be corrected just by a little re-wording it from

"Analyze the sequence of images containing the {label}. Focus on the likely intent or behavior of the {label} based on its actions and movement, rather than describing its appearance or the surroundings. Consider what the {label} is doing, why, and what it might do next."

to

"Analyze the image or sequence of images containing the {label}. Focus on the likely intent or behavior of the {label} based on its actions and movement, rather than describing its appearance or the surroundings. Consider what the {label} is doing, why, and what it might do next."

That may resolve the LLMs issue with the inconsistency. Each LLM is a little different, I have been experimenting with the Ollama supported ones locally and many behave differently and require adjustments to the prompts.

[lowlyocean]

If the "/embed" endpoint of Ollama's API at some point introduces an "image" parameter, would this project accept contributions to allow Semantic Search to optionally use that rather than running ONNX runtime locally? Looks like this is being tracked as an issue here

[NickM-27]

It would likely have the same challenges as supporting multiple embedding models now which is that their shapes are often different and will require a lot of configuration and DB clearing / reindexing to change models

[pdawg17]

I am using openvino with iGPU on beta 3 and CPU usage sits at 30+% at all times. Is that expected? Intel GPU when idle sits at 2-3%.

[NickM-27]

It can be, assuming detection is running. It would be great to see screenshots of the system and camera metrics

[stegl]

someone else have much longer times to load and play streams on live? (sometimes doesnt even start)

[hawkeye217]

Nothing changed in that area for this beta. Check your camera settings: https://docs.frigate.video/configuration/live/#camera-settings-recommendations

[johnlento]

Anyone having issues with Ollama? I swapped my GPU's and then upgraded to beta 3 now posts to Ollama return 200 but I get this in the logs.

2024-12-15 17:16:45.231494832 [2024-12-15 17:16:45] httpx INFO : HTTP Request: POST http://ollama:11434/api/show "HTTP/1.1 200 OK"
2024-12-15 17:16:58.360819516 Exception in thread embeddings_maintainer:
2024-12-15 17:16:58.360823779 Traceback (most recent call last):
2024-12-15 17:16:58.360825991 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2024-12-15 17:16:58.360827796 self.run()
2024-12-15 17:16:58.360829866 File "/opt/frigate/frigate/embeddings/maintainer.py", line 74, in run
2024-12-15 17:16:58.360831809 self._process_event_metadata()
2024-12-15 17:16:58.360833977 File "/opt/frigate/frigate/embeddings/maintainer.py", line 255, in _process_event_metadata
2024-12-15 17:16:58.360835934 self.handle_regenerate_description(event_id, source)
2024-12-15 17:16:58.360838136 File "/opt/frigate/frigate/embeddings/maintainer.py", line 339, in handle_regenerate_description
2024-12-15 17:16:58.360840000 x1_rel, y1_rel, width_rel, height_rel = event.data["region"]
2024-12-15 17:16:58.360841562 KeyError: 'region'

[hawkeye217]

Does this happen with all of your tracked objects, or just with specific ones?

[hawkeye217]

I was able to reproduce this by trying to regenerate a description on a manual event. Fixed in #15525

[johnlento]

Well that was quick, thanks. Will it just be in the next beta?

[hawkeye217]

Yes, also in the next dev build once the PR is merged.

[chrislawso]

editing and troubleshooting machine simultaneously. ignore this for now as the network may be the issue blocking ports.

[hawkeye217]

Is the name of the model you downloaded to Ollama before starting up Frigate actually llava:7b?

[chrislawso]

Is the name of the model you downloaded to Ollama before starting up Frigate actually llava:7b?

I successfully got frigate:717493e-tensorrt connected with ollama with llava:7b.

I found in another post #14599 (comment) "that self.tracked_events[event_id]] is in-memory and not persisted, so after any restart will be empty, and so regenerating will just fall back to a single image from base64.b64decode(event.thumbnail)" so apparently best generations happen when self.tracked_events[event_id]] is in-memory. I will work on adjusting variables to address the logs level=WARN source=sched.go:137 msg="multimodal models don't support parallel requests yet".

[hawkeye217]

See the docs: https://deploy-preview-13787--frigate-docs.netlify.app/configuration/genai#ollama

[primevisionservices]

I'm using the beta 3 version, it's been saving snapshots longer than I determined in the camera settings, my camera configuration is like this,
and snapshots are being saved for more than 1 day, I didn't configure anything globally

  camera_27:
    ffmpeg:
      hwaccel_args: preset-vaapi
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
        - path: rtsp://127.0.0.1:8554/camera_27
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
    detect:
      fps: 3
    objects:
      track:
        - person
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: false
      retain:
        default: 1
    record:
      enabled: true
      retain:
        days: 1
        mode: all
      alerts:
        retain:
          days: 1
          mode: all
        pre_capture: 10
      detections:
        retain:
          days: 1
          mode: all
        pre_capture: 10

[NickM-27]

this is a known bug that has been fixed for the next beta

[EM-Tom]

I had to regress to 14 due to the SHM issues, the refactor in beta 3 seems to have nailed that for me now so thanks a lot.

Now I am getting fairly regular service restarts, maybe every 10 mins. On 14 stable this was rock solid no issues at all.

Detection appears to be stuck. Restarting detection process..
Detection process didn't exit. Force killing...

CPU is around 17%
Storage is local and no IO wait
Everything resources looking barely tickled under metrics

Any idea what might be going down?

Thanks

[NickM-27]

Nothing changed regarding the coral so no idea why that would be occurring.

[aliamd996]

Hi everyone
How can export yolo11 models to work with frigate?

[NickM-27]

Yolo11 is not currently supported in frigate