Live View defaulting to MSE instead of WebRTC

[aegjoyce]

Describe the problem you are having

Have just upgraded to 0.14rc but can't seem to get Live View to push WebRTC - it goes with MSE which has a 3 second delay. WebRTC worked perfectly fine on 0.13 on the iPhone and Mac I am using (Safari for both). The removal of the Live View options means I am unable to override the preselection of MSE over WebRTC.

Steps to reproduce

Upgraded to 0.14rc. See below for specific config. Port 8555 is forwarded on my router.

Version

0.14.0-7c39b17

Frigate config file

cameras:
  3D_Printer:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/3D_Printer
          input_args: preset-rtsp-restream
    record:
      enabled: true
      retain:
        days: 14
        mode: all
    detect:
      enabled: false
    live:
      stream_name: 3D_Printer

go2rtc:
  webrtc:
    candidates:
      - 192.168.1.28:8555
      - stun:8555
  streams:
    3D_Printer:
      - rtsp://192.168.1.35:34263/9fcf018353a4bea0
      - "ffmpeg:3D_Printer#audio=opus"

ffmpeg:
  hwaccel_args: preset-vaapi

detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt
version: 0.14

Relevant log output

2024-07-31 20:43:18.756981409  [INFO] Preparing Frigate...
2024-07-31 20:43:18.811529714  [INFO] Starting Frigate...
2024-07-31 20:43:21.038158053  [2024-07-31 20:43:21] frigate.app                    INFO    : Starting Frigate (0.14.0-7c39b17)
2024-07-31 20:43:21.038437152  [2024-07-31 20:43:21] frigate.util.config            INFO    : Checking if frigate config needs migration...
2024-07-31 20:43:21.079995717  [2024-07-31 20:43:21] frigate.util.config            INFO    : frigate config does not need migration...
2024-07-31 20:43:27.118070828  [2024-07-31 20:43:27] frigate.config                 WARNING : Customizing more than a detector model path is unsupported.
2024-07-31 20:43:27.136379144  [2024-07-31 20:43:27] peewee_migrate.logs            INFO    : Starting migrations
2024-07-31 20:43:27.137174686  [2024-07-31 20:43:27] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-07-31 20:43:27.369873254  [2024-07-31 20:43:27] frigate.app                    INFO    : Recording process started: 348
2024-07-31 20:43:27.375442716  [2024-07-31 20:43:27] frigate.app                    INFO    : Recording process started: 349
2024-07-31 20:43:27.378907392  [2024-07-31 20:43:27] frigate.app                    INFO    : go2rtc process pid: 99
2024-07-31 20:43:27.412905536  [2024-07-31 20:43:27] frigate.app                    INFO    : Output process started: 379
2024-07-31 20:43:27.436234744  [2024-07-31 20:43:27] detector.ov                    INFO    : Starting detection process: 378
2024-07-31 20:43:27.437093502  [2024-07-31 20:43:27] frigate.detectors.plugins.openvino WARNING : OpenVINO AUTO device type is not currently supported. Attempting to use GPU instead.
2024-07-31 20:43:27.641452493  [2024-07-31 20:43:27] frigate.app                    INFO    : Camera processor started for 3D_Printer: 395
2024-07-31 20:43:27.713167022  [2024-07-31 20:43:27] frigate.app                    INFO    : Capture process started for 3D_Printer: 422

Operating system

Proxmox

Install method

Docker Compose

Network connection

Wired

Camera make and model

Tapo C100

Any other information that may be helpful

docker_compose.yml:

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.14.0-rc1 #ghcr.io/blakeblackshear/frigate:stable
    shm_size: "256mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/dri:/dev/dri # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ~/frigate/config:/config
      - /frigate/media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8971:8971"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
      - "1984:1984" # go2rtc
    environment:
      FRIGATE_RTSP_PASSWORD: "password"
    labels:
      - "com.centurylinklabs.watchtower.scope=frigate"

[NickM-27]

This is not a bug and is working as expected, mse is the default as it is more compatible. 3 seconds is quite high for a delay, MSE for all of my cameras is ~500 ms across multiple types of devices.

[aegjoyce]

The delay is variable - sometimes it's sub-1s but I'm noticing that it frequently rewinds back a couple of seconds and continues streaming, an issue I never found with WebRTC.

The list here suggests that WebRTC has the lowest latency - if Frigate could detect which browser was accessing it, couldn't it default to WebRTC for compatible browsers, and then fall back to MSE for incompatible browsers?

Failing this - having the option to choose the stream source again would be fine...!

[hawkeye217]

If MSE is lagging but WebRTC "works perfectly", it's usually the case that WebRTC is dropping frames. MSE doesn't drop frames by default. Frigate 0.14.1 has improvements specifically for MSE. If your stream lags behind actual live time, it will dynamically compensate. What version of Frigate are you running?

[nljc88]

@hawkeye217 thanks for the deeper insight... WebRTC streams without loading errors and without what seems to be "lag". It is almost instant when I stand and wave in front of the camera - though MSE has a huge delay or "lag". Additionally, the MSE stream will drop and buffer and reload - however I can play WebRTC for long lengths of time without seeing this.

Sure maybe it's dropping frames to compensate - but the experience is 10 fold over MSE for my needs. Frigate docs and the internet in general favour WebRTC as the better technology for less lag and that's my goal.

I think giving frigate users the choice of how they want to stream, in a small box up top (defaulting to what frigate chooses, with ability to override), seems like a good compromise for advanced users to me.

Frigate (0.14.1-f4f3cfa)

[NickM-27]

It's not that simple, the necessary stream method to use will often vary based on the particular camera. For example WebRTC in most browsers does not support h265, so unless a user who chose webrtc gets a broken stream then we would need to have a way to handle that case. Also, for example, two way talk requires webrtc so that is another caveat. Another example is that Safari actually has issues with WebRTC in certain circumstances.

This is why the priority is for Frigate to control which stream is shown so no matter the feature or circumstance it can make the best choice. Especially after the changes in 0.14.1, MSE performs better than webrtc in many cases due to the dynamic playback rate changing allowing the buffering of MSE to not cause latency issues.

If you are having issues then I would suggest reading and applying https://docs.frigate.video/configuration/live#camera-settings-recommendations which many users have said helped immensely with their camera streams. Failing that, you should create your own support issue so we can more closely look in to the issue that you are seeing.

[nljc88]

@NickM-27 Appreciate the response here - highlights how much more is at play in the background that we're not aware of. Huge props for amazing software!!

I've just gotten my first Dahua and returned all Reolinks and am seeing huge improvement after following the recommendations at that link already - along with your specific Frame rate and interval settings. More Dahua camera's on the way!

It will just be a pesky Reolink doorbell to deal with - which isn't so bad ;) Will open separate issues if I need anything in future - cheers!

[NickM-27]

I have the reolink doorbell and it works just fine, make sure interframe space is set to 1x and the frame mode is set to constant