README.USAGE

Introduction
============

This document serves as a starting guide on effectively using the GST-MFX plugins
for your custom GStreamer application, by demonstrating a considerable number of examples for common use cases.
Also, some useful command-line utilities will be shown facilitating the usage of these plugins.


Available GST-MFX plugins
================================

Plugin name        | Description
------------------------------------------------------------------------
mfxdecode          | Generic MFX Decoder Plugin
mfxh264dec         | MFX H264 Decoder Plugin
mfxhevcdec         | MFX HEVC Decoder Plugin
mfxmpeg2dec        | MFX MPEG2 Decoder Plugin
mfxjpegdec         | MFX JPEG Decoder Plugin
mfxwmvdec          | MFX WMV3 Decoder Plugin
mfxvc1dec          | MFX VC1 Decoder Plugin
mfxvp8dec          | MFX VP8 Decoder Plugin
mfxvp9dec          | MFX VP9 Decoder Plugin
mfxvpp             | MFX VPP Plugin
mfxh264enc         | MFX H264 Encoder Plugin
mfxhevcenc         | MFX H265 Encoder Plugin
mfxmpeg2enc        | MFX MPEG2 Encoder Plugin
mfxjpegenc         | MFX JPEG Encoder Plugin
mfxsink            | X11 / Wayland / D3D11 Renderer Plugin (mfxvpp + mfxsinkelement GstBin element)
mfxsinkelement     | Standalone X11 / Wayland / D3D11 Renderer Plugin


GStreamer Debugging Tips
========================

To list the configurable properties of an MFX plugin listed above (in this case, mfxvpp):

  gst-inspect-1.0 mfxvpp

To view debug, log or error messages for a GStreamer pipeline using the MFX plugins:

  export GST_DEBUG=mfx*:<log_level>,mfxsink:<log_level>

before executing the GStreamer pipeline. For this particular case,
we are logging the runtime debug output of mfxsink and its associated gst-libs.
<log_level> represents a GStreamer debug value from 1 to 8.

To view running frame rate statistics on the terminal when decoding, encoding or rendering,
run the following command first:

  export GST_DEBUG=fpsdisplaysink:6

Some GStreamer pipelines to demonstrate performance benchmarking of MFX plugins:

# Benchmark mfxsink rendering performance
gst-launch-1.0 filesrc location=input.mkv ! matroskademux ! h265parse ! mfxhevcdec ! \
  fpsdisplaysink video-sink=mfxsink text-overlay=false signal-fps-measurements=true

# Benchmark HEVC decoder performance
gst-launch-1.0 filesrc location=input.mkv ! matroskademux ! h265parse ! mfxhevcdec ! \
  fpsdisplaysink video-sink=fakesink text-overlay=false signal-fps-measurements=true sync=false
  
  
Example GStreamer Pipelines
===========================

# Video playback with audio and subtitles (from GStreamer 1.8 onwards)
gst-play-1.0 /path/to/video.mkv

# Fullscreen video playback using playbin with VPP rotation and EGL rendering 
gst-launch-1.0 playbin uri=file:///path/to/video.mp4 \
  video-filter="mfxvpp rotation=90" video-sink="mfxsink display=egl fullscreen=true"

# Basic H264 decode with OSD frame rate tracking
gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! h264parse ! mfxh264dec ! \
  fpsdisplaysink video-sink=mfxsink

# Basic MPEG2 decode
gst-launch-1.0 filesrc location=input.mpg ! tsdemux ! mpegvideoparse ! mfxmpeg2dec ! mfxsinkelement

# Basic HEVC decode
gst-launch-1.0 filesrc location=input.mkv ! matroskademux ! h265parse ! mfxhevcdec ! mfxsinkelement

# Basic JPEG decode
gst-launch-1.0 filesrc location=input.avi ! avidemux ! mfxjpegdec ! mfxsinkelement

# Basic VC1 decode with custom VC1 parser
gst-launch-1.0 filesrc location=input.wmv ! asfdemux ! vc1parse ! mfxdecode ! mfxsinkelement

# Basic VP8 / VP9 decode
gst-launch-1.0 filesrc location=input.webm ! matroskademux ! mfxdecode ! mfxsinkelement

# H264 raw file encode
gst-launch-1.0 filesrc location=raw.yuv ! \
  videoparse width=<width> height=<height> format=<format> ! \
  mfxh264enc ! matroskamux ! filesink location=/path/to/output.mkv sync=false

# Trancode an H264 video to an HEVC video
gst-launch-1.0 filesrc location=video.mp4 ! qtdemux ! h264parse ! mfxdecode ! \
  mfxhevcenc ! matroskamux ! filesink location=/path/to/output.mkv sync=false

# Trancode a 4k H264 video to a resized 1080p HEVC video with AAC audio encoding and muxing
gst-launch-1.0 filesrc location=4k_video.mp4 ! qtdemux name=demux demux.video_0 ! h264parse ! \
  mfxh264dec ! mfxvpp width=1920 height=1080 ! mfxhevcenc ! mux. demux.audio_0 ! queue ! decodebin ! \
  avenc_aac ! mux. matroskamux name=mux ! filesink location=/path/to/output.mkv sync=false

# Video preview with transcoding
gst-launch-1.0 filesrc location=video.mp4 ! qtdemux ! h264parse ! mfxdecode ! \
  tee name=tp tp. ! queue ! mfxsinkelement  tp. ! queue ! \
  mfxmpeg2enc ! mpegtsmux ! filesink location=/path/to/output.mpg

# Simultaneous camera preview using dma-buffer sharing and H264 encoding
gst-launch-1.0 v4l2src io-mode=5 ! video/x-raw, width=<w>, height=<h> ! mfxvpp ! \
  tee name=tp ! queue ! mfxsinkelement tp. ! mfxh264enc ! 'video/x-h264, profile=baseline' ! \
  flvmux streamable=true ! filesink location=/path/to/output.flv
  
# RTP sender (H264)
gst-launch-1.0 v4l2src io-mode=5 ! mfxh264enc ! \
  'video/x-h264, stream-format=byte-stream, profile=baseline' ! \
  rtph264pay pt=96 ! udpsink host=<ip> port=<port>

# RTP receiver (H264)
gst-launch-1.0 -v udpsrc port=<port> caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" ! \
  rtph264depay ! h264parse ! mfxh264dec ! mfxsink

# Multi-window decode with different renderers
gst-launch-1.0 filesrc location=video.mp4 ! qtdemux ! h264parse ! mfxdecode ! \
  tee name=tp tp. ! queue ! mfxsink display=egl  tp. ! queue ! mfxsink display=x11

# VPP postprocessing after decode (gst-inspect-1.0 mfxvpp for <options>)
gst-launch-1.0 filesrc location=video.mkv ! matroskademux ! h265parse ! mfxhevcdec ! \
  mfxvpp <options> ! mfxsinkelement

# VPP postprocessing during MPEG2 to H264 transcoding (gst-inspect-1.0 mfxvpp for <options>)
gst-launch-1.0 filesrc location=video.mpg ! mpegpsdemux ! mpegvideoparse ! mfxdecode ! \
  mfxvpp <options> ! mfxh264enc ! qtmux ! filesink location=/path/to/output.mp4 sync=false


Optimized Usage with GstGL Plugins
==================================

The GST-MFX plugins include optimized support for GstGL plugins in Linux by enabling zero-copy
EGL texture transfers using dma-buffer sharing. This dma-buffer sharing optimization has been introduced
since GStreamer version 1.8, and hence only works from this version onwards. For Windows, DirectX 11 / OpenGL
interoperability with the WGL_NV_DX_interop2 extension is used to enable zero-copy GL texture sharing.

To enable this feature in Linux, run the following command before executing the GStreamer pipeline:

  export GST_GL_PLATFORM=egl

To demonstrate optimized zero-copy rendering with glimagesink,

  gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! h264parse ! mfxdecode ! \
    fpsdisplaysink video-sink=glimagesink

For a more useful real-world example of video playback with glimagesink,

  gst-play-1.0 /path/to/video.mp4 --videosink=glimagesink

Finally, we demonstrate a complex pipeline that creates a video wall running 4 video streams concurrently
while ensuring all frames are processed in video memory for full performance.

  gst-launch-1.0 glvideomixer name=m
    sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540
    sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540
    sink_2::xpos=0 sink_2::ypos=540 sink_2::width=960 sink_2::height=540
    sink_3::xpos=960 sink_3::ypos=540 sink_3::width=960 sink_3::height=540 ! glimagesink \
    v4l2src io-mode=5 ! mfxvpp ! m.sink_0 \
    filesrc location=video1.mp4 ! qtdemux ! h264parse ! mfxdecode ! mfxvpp brightness=-90 ! m.sink_1 \
    filesrc location=video2.mpg ! tsdemux ! mpegvideoparse ! mfxdecode ! m.sink_2 \
    filesrc location=video3.mkv ! matroskademux ! h265parse ! mfxdecode ! mfxvpp contrast=3 ! m.sink_3

    
Dealing with tee and queue elements
===================================

Since GStreamer and Intel MSDK are heavily threaded frameworks, each with their own internal thread synchronization
mechanisms to perform concurrent operations, there are some nasty issues that need to be dealt with carefully
especially with tee and queue elements. This can be illustrated as follows:

  gst-launch-1.0 filesrc location=/path/to/video.mp4 ! qtdemux ! h264parse ! mfxdecode ! tee name=t ! \
    queue ! mfxvpp contrast=3 ! mfxsink t. ! \
    queue ! mfxvpp brightness=-90 ! mfxsink

Running this pipeline will cause unpredictable behaviour. This is because there are 2 downstream VPP elements that
are contending to share the parent MFX session initialized by mfxdecode. For MSDK, a single MFX session can only be
shared by single instances of MFX decode, encode and VPP tasks. In this case, two VPP tasks are simultaneously contending
to share the parent MFX session initialized by the single MFX decode task.

To overcome the issues caused by the described pipeline, we can add an extra mfxvpp element and
configure a few more options as follows:

  gst-launch-1.0 filesrc location=/path/to/video.mp4 ! qtdemux ! h264parse ! mfxdecode async-depth=16 ! mfxvpp rotation=0 ! \
    tee name=t ! queue ! mfxvpp contrast=3 ! mfxsink t. ! \
    queue ! mfxvpp brightness=-90 ! mfxsink

The above pipeline works perfectly well since the parent MFX session from mfxdecode is only shared by single VPP task
instantiated by mfxvpp which pretty much works in pass-through mode. The rotation=0 option is necessary in mfxvpp, otherwise
mfxvpp will work in true pass-through mode without forcing the instantiation of the shared MFX VPP task required to prevent
contention of the parent MFX session by the downstream mfxvpp elements. This workaround ensures that the downstream mfxvpp
elements branched out from the tee element each initialize their own separate MFX sessions and process the identity output
of the first mfxvpp element. Hence, MSDK rules for sharing the parent MFX session are strictly followed here.

This is still not yet the end of the story. Due to the latency caused by the queue elements, there can be some unexpected
jitters and stuttering when viewing the mfxsink outputs. To resolve the jitters / stutters, we add the async-depth option
in mfxdecode and set it to a high enough value, in this case, 16, until we can no longer see the output issues. While this
last step may seem hack-ish, it is currently the only way to resolve the lack of proper synchronization between the
heavily threaded GStreamer and MSDK frameworks.