Audio/video stream issues with an h.265 camera

[bikemike]

What is your question?

I'm using Frigate 0.14 and I am not able to get video or audio in the frigate live view. Additionally, I am not able to get audio in the Home Assistant Frigate card. Can anyone help me figure this out?

For the live view issue, the camera thumbnails display fine but shows black when I click on the camera. If I go into Settings->Debug I can see the video fine.

For the Home Assistant issue, I set the live view provider to go2rtc and set the go2rtc stream name to either jrcam_low or jrcam_high but no audio stream seems to be available.

The audio stream works for recordings and if I open the go2rtc stream in VLC, I am able to get both audio and video.

Below is my config and the stream info.

Thanks!

cameras:
  jrcam:
    birdseye:
      order: 1
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/jrcam_low # detect from low res stream
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
        - path: rtsp://127.0.0.1:8554/jrcam_high # restream from go2rtc, record high res stream
          input_args: preset-rtsp-restream
          roles:
            - record
    live:
      stream_name: jrcam_low
    detect:
      width: 800
      height: 600


go2rtc:
  streams:
    jrcam_low:
      - rtsp://192.168.10.179/2
      - "ffmpeg:jrcam_low#audio=aac#audio=opus"
    jrcam_high:
      - rtsp://192.168.10.179/1
      - "ffmpeg:jrcam_high#audio=aac#audio=opus"

"jrcam_high": {
    "consumers": [
      {
        "medias": [
          "audio, sendonly, ANY"
        ],
        "remote_addr": "127.0.0.1:58888",
        "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.2\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=audio 0 RTP/AVP 96\r\na=rtpmap:96 PCMA/8000\r\na=fmtp:96 packetization-mode=1\r\na=control:trackID=0\r\n",
        "send": 1205600,
        "senders": [
          "96 PCMA/8000, bytes=1096000, receivers=1"
        ],
        "type": "RTSP passive consumer",
        "url": "rtsp://127.0.0.1:8554/jrcam_high?audio",
        "user_agent": "go2rtc/ffmpeg"
      },
      {
        "medias": [
          "video, sendonly, H264, H265",
          "audio, sendonly, MPEG4-GENERIC"
        ],
        "remote_addr": "127.0.0.1:58804",
        "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.2\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H265/90000\r\na=fmtp:96 profile-space=0;profile-id=1;tier-flag=0;level-id=1;interop-constraints=800000000000;sprop-vps=AAAAAUABDAH//wFgAAADALAAAAMAAAMAmawJ;sprop-sps=AAAAAUIBAQFgAAADALAAAAMAAAMAmaABQCAHgWNruTL83AQEBAI=;sprop-pps=AAAAAUQBwPKwOzQ=\r\na=control:trackID=0\r\nm=audio 0 RTP/AVP 97\r\na=rtpmap:97 MPEG4-GENERIC/8000\r\na=fmtp:97 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=158856E500\r\na=control:trackID=1\r\n",
        "send": 8757362,
        "senders": [
          "96 H265, bytes=8189222, receivers=1",
          "97 MPEG4-GENERIC/8000, bytes=449868, receivers=1"
        ],
        "type": "RTSP passive consumer",
        "url": "rtsp://127.0.0.1:8554/jrcam_high",
        "user_agent": "FFmpeg Frigate/0.14.0-da913d8"
      }
    ],
    "producers": [
      {
        "medias": [
          "video, recvonly, H265",
          "audio, recvonly, PCMA/8000"
        ],
        "receivers": [
          "96 H265, bytes=8189222, senders=1",
          "8 PCMA/8000, bytes=1096000, senders=1"
        ],
        "recv": 9451782,
        "remote_addr": "192.168.10.179:554",
        "sdp": "v=0\r\no=- 38990265062388 38990265062388 IN IP4 192.168.10.179\r\ns=10\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\na=control:*\r\na=range:npt=0-\r\nm=video 0 RTP/AVP 96\r\na=control:trackID=0\r\na=rtpmap:96 H265/90000\r\na=range:npt=0-\r\na=framerate:0S\r\na=fmtp:96 profile-space=0;profile-id=1;tier-flag=0;level-id=1;interop-constraints=800000000000;sprop-vps=AAAAAUABDAH//wFgAAADALAAAAMAAAMAmawJ;sprop-sps=AAAAAUIBAQFgAAADALAAAAMAAAMAmaABQCAHgWNruTL83AQEBAI=;sprop-pps=AAAAAUQBwPKwOzQ=\r\na=x-dimensions: 2560, 1920\r\na=x-framerate: 15\r\nm=audio 0 RTP/AVP 8\r\na=control:trackID=1\r\na=rtpmap:8 PCMA/8000/1\r\na=ptime:20\r\na=fmtp:8 packetization-mode=1\r\n",
        "type": "RTSP active producer",
        "url": "rtsp://192.168.10.179/1/",
        "user_agent": "go2rtc/1.9.2"
      },
      {
        "medias": [
          "audio, recvonly, MPEG4-GENERIC/8000",
          "audio, recvonly, OPUS/48000/2"
        ],
        "receivers": [
          "96 MPEG4-GENERIC/8000, bytes=449868, senders=1",
          "97 OPUS/48000/2, bytes=1171841, senders=0"
        ],
        "recv": 1709513,
        "remote_addr": "127.0.0.1:58916",
        "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=go2rtc/1.9.2\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\na=tool:libavformat LIBAVFORMAT_VERSION\r\nm=audio 0 RTP/AVP 96\r\nb=AS:48\r\na=rtpmap:96 MPEG4-GENERIC/8000/1\r\na=fmtp:96 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=158856E500\r\na=control:streamid=0\r\nm=audio 0 RTP/AVP 97\r\nb=AS:64\r\na=rtpmap:97 opus/48000/2\r\na=control:streamid=1\r\n",
        "type": "RTSP passive producer",
        "url": "rtsp://127.0.0.1:8554/67125cb6b0c0230a27cb8ed76ed35357",
        "user_agent": "ffmpeg/go2rtc"
      }
    ]
  },
  "jrcam_low": {
    "consumers": [
      {
        "medias": [
          "audio, sendonly, ANY"
        ],
        "remote_addr": "127.0.0.1:58886",
        "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.2\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=audio 0 RTP/AVP 96\r\na=rtpmap:96 PCMA/8000\r\na=fmtp:96 packetization-mode=1\r\na=control:trackID=0\r\n",
        "send": 1206480,
        "senders": [
          "96 PCMA/8000, bytes=1096800, receivers=1"
        ],
        "type": "RTSP passive consumer",
        "url": "rtsp://127.0.0.1:8554/jrcam_low?audio",
        "user_agent": "go2rtc/ffmpeg"
      },
      {
        "medias": [
          "video, sendonly, H264, H265",
          "audio, sendonly, MPEG4-GENERIC"
        ],
        "remote_addr": "127.0.0.1:58776",
        "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.2\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H265/90000\r\na=fmtp:96 profile-space=0;profile-id=1;tier-flag=0;level-id=1;interop-constraints=800000000000;sprop-vps=AAAAAUABDAH//wFgAAADALAAAAMAAAMAWqwJ;sprop-sps=AAAAAUIBAQFgAAADALAAAAMAAAMAWqAGQgCWWNruTL83AQEBAIA=;sprop-pps=AAAAAUQBwPKwOzQ=\r\na=control:trackID=0\r\nm=audio 0 RTP/AVP 97\r\na=rtpmap:97 MPEG4-GENERIC/8000\r\na=fmtp:97 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=158856E500\r\na=control:trackID=1\r\n",
        "send": 2817847,
        "senders": [
          "96 H265, bytes=2402206, receivers=1",
          "97 MPEG4-GENERIC/8000, bytes=449518, receivers=1"
        ],
        "type": "RTSP passive consumer",
        "url": "rtsp://127.0.0.1:8554/jrcam_low",
        "user_agent": "FFmpeg Frigate/0.14.0-da913d8"
      },
      {
        "medias": [
          "video, sendonly, H264, H265",
          "audio, sendonly, MPEG4-GENERIC"
        ],
        "remote_addr": "127.0.0.1:58820",
        "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.2\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H265/90000\r\na=fmtp:96 profile-space=0;profile-id=1;tier-flag=0;level-id=1;interop-constraints=800000000000;sprop-vps=AAAAAUABDAH//wFgAAADALAAAAMAAAMAWqwJ;sprop-sps=AAAAAUIBAQFgAAADALAAAAMAAAMAWqAGQgCWWNruTL83AQEBAIA=;sprop-pps=AAAAAUQBwPKwOzQ=\r\na=control:trackID=0\r\nm=audio 0 RTP/AVP 97\r\na=rtpmap:97 MPEG4-GENERIC/8000\r\na=fmtp:97 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=158856E500\r\na=control:trackID=1\r\n",
        "send": 2817847,
        "senders": [
          "96 H265, bytes=2402206, receivers=1",
          "97 MPEG4-GENERIC/8000, bytes=449518, receivers=1"
        ],
        "type": "RTSP passive consumer",
        "url": "rtsp://127.0.0.1:8554/jrcam_low",
        "user_agent": "FFmpeg Frigate/0.14.0-da913d8"
      }
    ],

[NickM-27]

I would suggest setting up webrtc

[bikemike]

In Frigate or in Home Assistant? Frigate used to have the option to select between jsmpeg, mse, and webrtc but I don't see that in 0.14.

In Home Assistant I checked off "Enable WebRTC for camera streams" in the Frigate integration but it doesn't show any controls for audio when viewing a stream. If I use the Frigate Card and then set the provider to "Automatic" it also doesn't show audio controls. The only way I can get audio from my other camera streams in Home Assistant is if I use the Frigate Card and set the provider to go2rtc and set the go2rtc stream name. But this doesn't work for this particular h.265 stream.

Also, I read a comment of yours from March 2023 that says that webrtc does not support h.265 but perhaps that is no longer valid. (#5783 (comment))

[NickM-27]

your config does not have webrtc configured https://docs.frigate.video/configuration/live#webrtc-extra-configuration

Also, I read a comment of yours from March 2023 that says that webrtc does not support h.265 but perhaps that is no longer valid.

it is still valid

[bikemike]

Ok, I think I figured out what I needed to do. I set the WebRTC candidates list as you linked to and I installed WebRTC Camera integration in Home Assistant. I am now able to get audio from the h.265 camera when I set up the card with "media: audio" but the h.265 video stream does not work with mse or webrtc.