[Config Support]: Live view often switches to sub stream instead of main stream #13810

gunarser · 2024-09-18T06:47:39Z

gunarser
Sep 18, 2024

Describe the problem you are having

Live view often uses sub stream although it is specifically configured to use main stream. Also, when main stream is used, live view appears almost instantly but when sub stream is used there is some noticeable delay.

Version

0.14.1-f4f3cfa

Frigate config file

############################### MQTT ###############################
mqtt:
  host: 192.168.8.8 #Please replace with a valid mqtt broker
  user: mqtt
  password: pass

############################### CPU CONFIG ###############################
detectors:
  coral:
    type: edgetpu
    device: usb

############################### GLOBAL CONFIG ###############################
environment_vars:
  LIBVA_DRIVER_NAME: i965

ffmpeg:
  hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format
    yuv420p

objects:
  track:
    - person
    - bicycle
    - cat
    - dog
    - car

record:
  sync_recordings: false

snapshots:
  enabled: true
  timestamp: true
  bounding_box: true

motion:
  improve_contrast: true

#logger:
#  default: error

############################### go2rtc section ###############################

go2rtc:
  log:
    format: text
    exec: trace
#  webrtc:
#    candidates:
#    - 192.168.8.26:8555
#    - stun:8555
  streams:
    frontdoor_cam_main:
      - ffmpeg:http://192.168.8.26/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass#video=copy#audio=volume
      - rtsp://user:pass@192.168.8.26:554/Preview_01_main

    upstairs_cam_main:
      - ffmpeg:http://192.168.8.40/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass#video=copy#audio=copy#audio=opus
      - rtsp://user:pass@192.168.8.40:554/Preview_01_main
    upstairs_cam_sub:
      - rtsp://user:pass@192.168.8.40:554/Preview_01_sub

    downstairs_cam_main:
      - ffmpeg:http://192.168.8.43/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass#video=copy#audio=copy#audio=opus
      - rtsp://user:pass@192.168.8.43:554/Preview_01_main
    downstairs_cam_sub:
      - rtsp://user:pass@192.168.8.43:554/Preview_01_sub

    kitchen_cam_main:
      - ffmpeg:http://192.168.8.53/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass#video=copy#audio=copy#audio=opus
      - rtsp://user:pass@192.168.8.53:554/Preview_01_main

    darzs:
      - rtsp://user:pass@192.168.8.15:554/Streaming/channels/101
    darzs_cam_sub:
      - rtsp://user:pass@192.168.8.15:554/Streaming/channels/102

    iela:
      - rtsp://user:pass@192.168.8.16:554/Streaming/Channels/101
    iela_cam_sub:
      - rtsp://user:pass@192.168.8.16:554/Streaming/Channels/102

    garaza:
      - rtsp://user:pass@192.168.8.19:554/Streaming/channels/101
    garaza_cam_sub:
      - rtsp://user:pass@192.168.8.19:554/Streaming/channels/102

  ffmpeg:
    bin: ffmpeg
    volume: -af "volume=30dB"

############################### INDIVIDUAL CAMERAS ###############################
cameras:

############################### IELA ###############################
  iela:
    ffmpeg:
#      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format
#        yuv420p
      inputs:
        - path: rtsp://127.0.0.1:8554/iela_cam_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/iela
          input_args: preset-rtsp-restream
          roles:
            - record
    detect:
      width: 640
      height: 480
#      annotation_offset: -5

    zones:
      iebrauktuve:
        coordinates: 0.861,0.342,1,0.49,1,1,0.929,1,0.823,0.7,0.804,0.609,0.764,0.503,0.675,0.333
        objects:
          - person
          - bicycle
          - cat
          - dog
          - car
        inertia: 3
        loitering_time: 0
      cels:
        coordinates: 0.642,0.258,0.677,0.328,0.854,0.341,0.817,0.266

        inertia: 3
        loitering_time: 0
      ielas_darzs:
        coordinates: 0.641,0.265,0.759,0.501,0.839,0.764,0.925,1,0.687,1,0,1,0,1,0,0.269
        objects:
          - person
          - cat
          - dog
        inertia: 3
        loitering_time: 0
    record:
      enabled: true
      retain:
        days: 130
        mode: all
      events:
        retain:
          default: 130
          mode: active_objects
        pre_capture: 3
        post_capture: 10

    live:
      stream_name: iela

    snapshots:
      enabled: true
      required_zones:
        - iebrauktuve
        - cels
        - ielas_darzs
      retain:
        default: 130

 #   objects:
 #     filters:
 #       person:
 #         threshold: 0.75
 #         min_area: 4000
 #         max_area: 1000000
 #       cat:
 #         min_area: 1000
 #         max_area: 250000
 #       dog:
 #         min_area: 1000
 #         max_area: 500000
 #       bicycle:
 #         min_area: 1000
 #         max_area: 1000000
 #       car:
 #         min_area: 5000
 #         max_area: 1500000

    review:
      alerts:
        required_zones:
          - iebrauktuve
          - cels
          - ielas_darzs
      detections: {}
    motion:
      threshold: 45
      contour_area: 50
      improve_contrast: 'true'

############################### GARAZA ###############################

  garaza:
    ffmpeg:
#      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format
#        yuv420p
      inputs:
        - path: rtsp://127.0.0.1:8554/garaza_cam_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/garaza
          input_args: preset-rtsp-restream
          roles:
            - record
    objects:
      track:
        - person
        - cat
        - dog
#      filters:
#        person:
#          threshold: 0.70
#          min_area: 2000
#          max_area: 1000000
#        dog:
#          threshold: 0.70
#        cat:
#          threshold: 0.70

    detect:
      width: 640
      height: 360
#      annotation_offset: -15

    record:
      enabled: true
      retain:
        days: 130
        mode: all
      events:
        retain:
          default: 130
          mode: active_objects
        pre_capture: 3
        post_capture: 10

    live:
      stream_name: garaza

    snapshots:
      enabled: true
      retain:
        default: 130

    motion:
      threshold: 45
      contour_area: 50
      improve_contrast: 'true'
      mask: 
        0,0,0,0.143,0,0.327,0.07,0.153,0.213,0.119,0.497,0.081,0.595,0.068,0.696,0.059,0.753,0.058,0.786,0.071,0.832,0.112,0.929,0.226,1,0.338,1,0

############################### DARZS ###############################

  darzs:
    ffmpeg:
#      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format
#        yuv420p
      inputs:
        - path: rtsp://127.0.0.1:8554/darzs_cam_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/darzs
          input_args: preset-rtsp-restream
          roles:
            - record
    objects:
      track:
        - person
        - cat
        - dog
#      filters:
#        person:
#          threshold: 0.70
#          min_area: 5000
#          max_area: 1000000

    detect:
      width: 640
      height: 480
#      annotation_offset: -15

    record:
      enabled: true
      retain:
        days: 130
        mode: all
      events:
        retain:
          default: 130
          mode: active_objects
        pre_capture: 3
        post_capture: 10

    live:
      stream_name: darzs

    snapshots:
      enabled: true
      retain:
        default: 130

    motion:
      threshold: 45
      contour_area: 60
      improve_contrast: 'true'
      mask: 
        0.96,1,0.854,0.863,0.953,0.344,0.957,0.299,0.812,0.297,0.726,0.288,0.696,0.283,0.682,0.265,0.685,0.23,0.695,0.193,0.716,0.174,0.76,0.149,0.802,0.13,0.83,0.118,0.761,0.111,0.711,0.116,0.646,0.113,0.561,0.112,0.501,0.113,0.435,0.119,0.352,0.117,0.299,0.121,0.247,0.122,0.208,0.124,0.163,0.125,0.118,0.125,0.072,0.13,0.034,0.132,0,0.136,0,0,0.033,0,1,0,1,1

############################### Reolink Doorbell ###############################

  doorbell:
    enabled: false
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/frontdoor_cam_main?video&audio
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect

#     - path: rtsp://127.0.0.1:8554/frontdoor_cam_sub
#        input_args: preset-rtsp-restream
#        roles:
#        - detect
    objects:
      track:
        - person
        - cat
        - dog
    detect:
      width: 2560
      height: 1920

    motion:
      threshold: 55
#      contour_area: 25
      mask:
        - 2560,0,2560,626,0,623,0,0
      contour_area: 50
      improve_contrast: 'true'
    live:
      stream_name: frontdoor_cam_main
    record:
      enabled: true
      retain:
        days: 130
        mode: all
      events:
        retain:
          default: 130
          mode: active_objects
        pre_capture: 3
        post_capture: 10
    snapshots:
      enabled: true
      retain:
        default: 130

############################### UPSTAIRS ###############################

  upstairs:
    enabled: true
    onvif:
      host: 192.168.8.40
      port: 8000
      user: user
      password: pass
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/upstairs_cam_main?video&audio
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/upstairs_cam_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    objects:
      track:
        - person
    detect:
      width: 640
      height: 360
    live:
      stream_name: upstairs_cam_main
    record:
      enabled: true
      retain:
        days: 30
        mode: active_objects
      events:
        retain:
          default: 30
          mode: active_objects
        pre_capture: 3
        post_capture: 10
    snapshots:
      enabled: true
      retain:
        default: 30
    motion:
      mask: 0,0,0,0.073,0.382,0.077,0.381,1,0.752,1,0.754,0

############################### DOWNSTAIRS ###############################

  downstairs:
    enabled: true
    onvif:
      host: 192.168.8.43
      port: 8000
      user: user
      password: pass
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/downstairs_cam_main?video&audio
          input_args: preset-rtsp-restream
          roles:
            - record

        - path: rtsp://127.0.0.1:8554/downstairs_cam_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    objects:
      track:
        - person
    detect:
      width: 640
      height: 360
    live:
      stream_name: downstairs_cam_main
    record:
      enabled: true
      retain:
        days: 30
        mode: active_objects
      events:
        retain:
          default: 30
          mode: active_objects
        pre_capture: 3
        post_capture: 10
    snapshots:
      enabled: true
      retain:
        default: 30
    motion:
      mask: 0,0,0,0.08,0.312,0.079,0.312,0

############################### KITCHEN ###############################

  kitchen:
    enabled: true
    onvif:
      host: 192.168.8.53
      port: 8000
      user: user
      password: pass
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/kitchen_cam_main?video&audio
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
    objects:
      track:
        - person
    detect:
      width: 2560
      height: 1920
    live:
      stream_name: kitchen_cam_main
    record:
      enabled: true
      retain:
        days: 30
        mode: motion
      events:
        retain:
          default: 30
          mode: motion
        pre_capture: 3
        post_capture: 10
    snapshots:
      enabled: true
      retain:
        default: 30
    motion:
      mask:
        - 1669,0,1674,89,873,82,872,0
      threshold: 50
      contour_area: 55
      improve_contrast: 'true'

#### Camera groups ####

version: 0.14
camera_groups:
  Inside:
    order: 1
    icon: LuWarehouse
    cameras:
      - downstairs
      - kitchen
      - upstairs
  Outside:
    order: 2
    icon: LuTrees
    cameras:
      - darzs
      - doorbell
      - garaza
      - iela

Relevant Frigate log output

Perhaps this gives some clue:
<IP address> - - [18/Sep/2024:09:36:13 +0300] "GET /live/mse/api/ws?src=iela HTTP/1.1" 101 862 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"

Relevant go2rtc log output

<IP address> - - [18/Sep/2024:09:36:00 +0300] "GET /live/mse/api/ws?src=iela HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
<IP address> - - [18/Sep/2024:09:36:03 +0300] "GET /live/mse/api/ws?src=iela HTTP/1.1" 101 862 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
<IP address> - - [18/Sep/2024:09:36:13 +0300] "GET /live/jsmpeg/iela HTTP/1.1" 101 452308 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"

Frigate stats

No response

Operating system

Other Linux

Install method

Docker Compose

Object Detector

Coral

Screenshots of the Frigate UI's System metrics pages

No response

Any other information that may be helpful

No response

Answered by gunarser

Sep 20, 2024

I have older Hikvision cameras which were configured with H.265+. I downgraded to H.265 and now live view uses MSE stream which is great.
My concern now is how much more storage would be used. As I understand H.265+ is improvement over H.265 in this regard. I still have to do comparison.

View full answer

gunarser · 2024-09-18T07:36:50Z

gunarser
Sep 18, 2024
Author

In addition, I noticed that when there is motion detected (i.e. red dot pulses) from particular camera Live view mostly uses main stream as it should.

0 replies

hawkeye217 · 2024-09-18T11:56:34Z

hawkeye217
Sep 18, 2024
Collaborator

From the docs:

Frigate intelligently uses three different streaming technologies to display your camera streams on the dashboard and the single camera view, switching between available modes based on network bandwidth, player errors, or required features like two-way talk.

Frigate switches the stream type if:

The browser reports that buffering is occurring.
The browser reports a decoding error with the stream.
The player falls too far behind the actual live time.

When go2rtc is configured, Frigate starts with the MSE player as it's the most widely compatible among browsers and devices. If that fails per any of the conditions above, the WebRTC player is used (if configured). If that fails (or is not configured), Frigate falls back to the jsmpeg player, which is typically your low-res decoded stream.

So the behavior you're seeing is not unexpected. You may want to try to configure WebRTC to have a high-res fallback when the MSE player encounters any one of the above conditions.

0 replies

gunarser · 2024-09-19T09:24:19Z

gunarser
Sep 19, 2024
Author

Hi hawkeye217

I enabled WebRTC like this:

go2rtc:
  webrtc:
    candidates:
    - 192.168.8.30:8555
    - stun:8555

192.168.8.30 is the Frigate's local address

But it never uses WebRTC stream. When MSE works I see the following in Chrome Dev tools:

But when it fails I see fallback (MSE >WebRTC >JSMPEG) kick in:

And this is the same camera accessed just few seconds later. On Internet router I see no connections to tcp/udp/8555 port. But I have port forwarding using high ports not the default 80. 8555 is translated as is.

What surprises me most is inconsistency. If there was some configuration error it should always fall back but some times it uses MSE stream as expected.
Is there a way to force MSE stream somehow?

5 replies

hawkeye217 Sep 19, 2024
Collaborator

No, there's no way to force a stream type.

My guess is that your browser is reporting a decoding error with the MSE stream when you see it fall back, but you can look at Chrome's media internals to see exactly why the browser is failing to play back the stream.

As far as getting WebRTC working, there are more strict requirements on codecs with WebRTC than with MSE. Is the browser you're trying to view the WebRTC stream on the same network as your Frigate server? You shouldn't need any port forwarding if so.

gunarser Sep 20, 2024
Author

I have older Hikvision cameras which were configured with H.265+. I downgraded to H.265 and now live view uses MSE stream which is great.
My concern now is how much more storage would be used. As I understand H.265+ is improvement over H.265 in this regard. I still have to do comparison.

Answer selected by hawkeye217

hawkeye217 Sep 20, 2024
Collaborator

Hikvision's + codecs do some really bizarre things to the video data to have it use slightly less storage space. One of the things they change is the I-frame interval, which is a very important component for streaming. Without a consistent interval, live stream startup times can be very unpredictable.

gunarser Sep 20, 2024
Author

I see. I will then stick with the H.265 codec on my Hiks.
I guess this case can be considered resolved.

hawkeye217 Sep 20, 2024
Collaborator

Great, glad it's working for you.

Lallassu · 2024-09-19T11:17:44Z

Lallassu
Sep 19, 2024

I have the same issue now after upgrading to 0.14. Before I could select which type of stream. Now it randomly selects the "best" stream, but it usually ends up being the worst quality for a bunch of TP-Link VIGI cameras. It loads the same amount of time as it did with version 0.13, except it ending up with the detect stream. I also hope for some configuration option to force mse stream or maybe a drop-down selection as there were in version 0.13 :)

14 replies

hawkeye217 Sep 19, 2024
Collaborator

Thanks for the info... the configuration options for these (UVC G3) obsoleted cameras is minimal -- but I tried an experiment where I bumped the bitrate from 3,000 to 6,000, and the frame-rate from 15 to 30. The result is that the live streams now load instantly, almost all the time, a vast improvement!

Great. Increasing the framerate causes an i-frame to be sent more frequently, so I'm glad it improves it for you.

Lallassu Sep 20, 2024

I have 4 UVCG3 as well, they work fine and loads fast. But my Tp-link vigi cameras are still loading slow in live view (and downgrades to detection stream). Any tips for those? I'm already running them at 30fps and 6k someting bitrate, which is max for them.

hawkeye217 Sep 20, 2024
Collaborator

What is the video and audio codec being output by the TP Link cameras?

Lallassu Sep 20, 2024

Audio: G711A-law
Video: H264 (can run H265 as well).
I'm currently running 2560x1440, 30fps, VBR, 4096kbps on 3 of them. Some of VIGI cams have a bit different settings as I'm running a mix of Vigi C330/C340.

There is this option for "Smart coding", I turned it off and it seems to load a bit faster now. But it's very random. Sometimes the live view loads within 1 sec with full quality, sometimes it takes time.

hawkeye217 Sep 20, 2024
Collaborator

See this note for the TP Link Vigi cameras in the docs about codecs: https://docs.frigate.video/configuration/camera_specific/#tp-link-vigi-cameras

Smart coding should be disabled as it likely removes a regular i-frame interval, much like Hikvision's + codecs (h.264+, etc).

See the camera settings recommendation docs for more info about i-frames/keyframes and live stream startup time.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Config Support]: Live view often switches to sub stream instead of main stream #13810

{{title}}

Replies: 4 comments 19 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

[Config Support]: Live view often switches to sub stream instead of main stream #13810

gunarser Sep 18, 2024

Describe the problem you are having

Version

Frigate config file

Relevant Frigate log output

Relevant go2rtc log output

Frigate stats

Operating system

Install method

Object Detector

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Replies: 4 comments · 19 replies

gunarser Sep 18, 2024 Author

hawkeye217 Sep 18, 2024 Collaborator

gunarser Sep 19, 2024 Author

hawkeye217 Sep 19, 2024 Collaborator

gunarser Sep 20, 2024 Author

hawkeye217 Sep 20, 2024 Collaborator

gunarser Sep 20, 2024 Author

hawkeye217 Sep 20, 2024 Collaborator

Lallassu Sep 19, 2024

hawkeye217 Sep 19, 2024 Collaborator

Lallassu Sep 20, 2024

hawkeye217 Sep 20, 2024 Collaborator

Lallassu Sep 20, 2024

hawkeye217 Sep 20, 2024 Collaborator

gunarser
Sep 18, 2024

Replies: 4 comments 19 replies

gunarser
Sep 18, 2024
Author

hawkeye217
Sep 18, 2024
Collaborator

gunarser
Sep 19, 2024
Author

hawkeye217 Sep 19, 2024
Collaborator

gunarser Sep 20, 2024
Author

hawkeye217 Sep 20, 2024
Collaborator

gunarser Sep 20, 2024
Author

hawkeye217 Sep 20, 2024
Collaborator

Lallassu
Sep 19, 2024

hawkeye217 Sep 19, 2024
Collaborator

hawkeye217 Sep 20, 2024
Collaborator

hawkeye217 Sep 20, 2024
Collaborator