WIP - Attempt to add av1 support

[Dishwasha]

For some reason Chrome doesn't detect the codec despite definitely supporting AV1 decoding. Nothing telling show up in console, chrome://webrtc-internals/, or chrome://media-internals/. Testing via docker-compose up.

[m1k1o]

Thank you for your conttribution. I tried it, same result. No error, just no video.

[m1k1o]

Fixes #49.

[Dishwasha]

Not sure how I missed that issue!

Just force pushed up a change to use rav1enc instead of svt1enc and now chrome://webrtc-internals/ is actually showing video data transmitting, but it still can't detect the decoder implementation to use. Using the gst-launch-1.0 tool, rav1enc seems a little slower than svt1enc but at least something is transmitting now.

[Dishwasha]

FYI went through the painstaking process of upgrading to gstreamer 1.22 to test and see if the AV1 video codec support improvements would solve the decoder detection problem, but ended up getting the same results :( Fortunately bookworm had most of the packages ready for testing 1.22. See https://gist.github.com/Dishwasha/8d43e375719b7c4ed155b6795f6cae2b.

[m1k1o]

@Dishwasha i tried your image but it fails with this error:

================================== 108/108 ===================================
test:         gst_gstbus
start time:   22:43:47
duration:     11.19s
result:       exit status 0
command:      MALLOC_PERTURB_=220 GST_PLUGIN_SCANNER_1_0=/src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu/libs/gst/helpers/gst-plugin-scanner CK_DEFAULT_TIMEOUT=20 GST_PLUGIN_LOADING_WHITELIST=gstreamer GST_REGISTRY=/src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu/tests/check/gst_gstbus.registry GST_PLUGIN_PATH_1_0=/src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu GST_STATE_IGNORE_ELEMENTS='' GST_PLUGIN_SYSTEM_PATH_1_0='' LD_LIBRARY_PATH=/src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu/libs/gst/check:/src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu/gst:/src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu/libs/gst/base:/src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu/libs/gst/net:/src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu/libs/gst/controller /src/gir1.2-gstreamer-1.0/obj-x86_64-linux-gnu/tests/check/gst_gstbus
----------------------------------- stdout -----------------------------------
Running suite(s): GstBus
100%: Checks: 14, Failures: 0, Errors: 0
Check suite gst_bus ran in 11.166s (tests failed: 0)
----------------------------------- stderr -----------------------------------

(gst_gstbus:19752): GStreamer-WARNING **: 22:43:47.518: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.
==============================================================================


Summary of Failures:

  3/108 validate.simplest             ERROR            0.21s   (exit status 255 or signal 127 SIGinvalid)
 31/108 gst_gstregistry               FAIL             0.16s   exit status 1

Ok:                 106 
Expected Fail:      0   
Fail:               2   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   
dh_auto_test: error: cd obj-x86_64-linux-gnu && LC_ALL=C.UTF-8 MESON_TESTTHREADS=32 meson test returned exit code 2
make[1]: *** [debian/rules:71: override_dh_auto_test] Error 25
make[1]: Leaving directory '/src/gir1.2-gstreamer-1.0'
make: *** [debian/rules:51: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui -b failed

Do you have any idea what it can be? Does it work for you?

[Dishwasha]

Do you have any idea what it can be? Does it work for you?

This is based off of bookworm which is still unstable so packages are constantly in flux. I wouldn't expect things to work consistently for more than a few days or a week. Fortunately once bookworm is released later this year, upgrading neko to it will allow for a more recent gstreamer and is probably a worthwhile upgrade.

The main point of this exercise was to validate that using an older version of gstreamer ISN'T the reason the browser can't seem to detect what codec to use to decode the AV1 video stream. I wish I knew more about why it isn't working, but I'm hoping somebody with more experience in this realm takes an interest.

[Dishwasha]

So a small update. I've discovered Chrome won't choose to decode AV1 without the SVC Extension for WebRTC enabled and the recently released Chrome 113. https://groups.google.com/g/discuss-webrtc/c/-QQ3pxrl-fw?pli=1 mentions needing scalabilityMode and a scaleResolutionDownBy specified.

I tested this out using PeerJS strictly between two Chrome browsers and when I set setCodecPreferences([RTCRtpSender.getCapabilities('video').codecs.find(codec => codec.mimeType == 'video/AV1')]) in the transceiver the video encoding defaulted back to VP8. When I set the transceivers sender encodings with scalabilityMode L1T2 & scaleResolutionDownBy 1, it finally used AV1! In chrome://webrtc-internals the outbound-rtp will show scalabilityMode L1T2 if it's being sent in the sender encodings properly.

I don't have time to hack on it at the moment, but we should be able to get similar results with Pion WebRTC by following the example https://github.com/cretz/webrtc/blob/simulcast-hack/examples/simulcast-hack/main.go.

* Also see the simulcast-playground example and this pion fiddle which is just missing the scalabilityMode.

[mbattista]

You should not need this extension for AV1.

I used and decoded AV1 in galene ( https://github.com/jech/galene/blob/06a0a2c36e4686f7e1bcb09e0123173586bbbd2a/codecs/codecs.go#L48 )
and in janus gateway ( https://github.com/meetecho/janus-gateway ) with just the normal chrome.

My problem was, that I received av1 packages, but the gstreamer encoder was not fast enough for realtime on my hardware.

I will try some different encoder soonish.

[Dishwasha]

So I tested this AV1 configuration out by screencasting 1080p from the Neko Chrome container to Chrome on another computer, so the hardware I'm using is definitely capable of real-time encoding (CPU barely used). Chrome uses libaom, so perhaps that would be the better encoder to try.

[ehfd]

The news is that libaom or rav1e are slower than svt-av1 recently.

[Dishwasha]

Small update. Gstreamer now supports nvav1enc in main branch but not yet part of releases. I've got AV1 Nvidia hardware offloading working in games-on-whales/wolf#98, but using the same settings in neko, I still can't get chrome to recognize the AV1 codec though it's successfully sending continuous bytes.

[ehfd]

Try my code.
https://github.com/selkies-project/selkies-gstreamer/blob/main/src/selkies_gstreamer/gstwebrtc_app.py

[Dishwasha]

Try my code. https://github.com/selkies-project/selkies-gstreamer/blob/main/src/selkies_gstreamer/gstwebrtc_app.py

I had to rebuild https://github.com/selkies-project/selkies-gstreamer/blob/main/addons/gstreamer/Dockerfile from gstreamer main branch before gst-py-example would work, but it totally does thanks!