How to Compare?

[nothing2obvi]

Hey, thanks again for helping me with the Mac issue with FFmpeg. I just have a quick question, in your experience.

Since VQM only compares by one parameter (preset, CRF, etc.), using the results, how would you suggest comparing these two settings?:

• veryslow, CRF 18
• slower, CRF 16

Let's say just for example these were the results, using VMAF:

• veryslow = 97
• CRF 18 = 92
• slower = 95
• CRF 16 = 96

Would you just take the average?

• veryslow, CRF 18 = (97 + 92) / 2
• slower, CRF 16 = (95 + 96) / 2

I'm sure it's probably much more complicated than that, but how would you approach this, given the data and your experience? Would you include PSNR and SSIM in your calculations? Thanks!

[CrypticSignal]

I don't think this is possible. Because a transcode with simply a preset chosen will also be using the default CRF value (23 is the default when using libx264, 28 is the default when using libx265), and a transcode with simply a CRF value chosen will be using the default preset (medium). So, if using the x265 encoder, you'd be calculating the average of veryslow, CRF 28 and medium, CRF 18 in an attempt to calculate the score you'd acheive with veryslow, CRF 18, which doesn't make sense.

You need to perform one transcode that uses both preset veryslow and CRF 18, and another transcode that uses both preset slower and CRF 16, and compare the results. I can implement the ability to specify a particular CRF value when comparing presets, or a particular preset when comparing CRF values. I might add this ability this week.

So you'd run something like:

python main.py -i test_videos/Seeking_30_480_1050.mp4 -e libx264 -p preset -v veryslow -crf 18

Then:

python main.py -i test_videos/Seeking_30_480_1050.mp4 -e libx264 -p preset -v slower -crf 16

Then you can simply compare the results. This does mean having to run two commands, but it's a simpler thing for me to implement.

At some point, I can try to implement the ability to compare a combination of two parameters with a combination of another two parameters, but this would be a little more involved to implement as you can imagine. The command you'd need to run if I implement such a feature could be something like:

python main.py -i test_videos/Seeking_30_480_1050.mp4 -e libx264 --parameter-combination preset crf --combination-1 veryslow 18 --combination-2 slower 16

Somewhat ironically, I don't know much about VMAF, SSIM and PSNR. I have not been involved in the development of any of these metrics, VQM just uses them through FFmpeg. VMAF specific questions are best asked in the VMAF repository. I don't see an option to start a discussion, but I suppose you can open an issue to ask a question related to VMAF.

[nothing2obvi]

Hey, thanks for your response.

The best way is to perform one transcode that uses both veryslow and CRF 18, and another transcode that uses both slower and CRF 16, and compare the results.

How would I compare them? Can I use VQM? If so, is there a way to do it using Overview Mode so I don't have to compare the entire videos?

I can implement the ability to specify a particular CRF value when comparing presets, or a particular preset when comparing CRF values. I might add this ability this week.

If you do implement this, combining this with Overview Mode would be so helpful. Please lmk if you do!

[CrypticSignal]

How would I compare them?

Both commands would produce a table with a single row with the VMAF score (and optionally) SSIM and PSNR. Simply compare the values. I know that comparison would be easier if the data was on the same table, but unless I implement the more complex feature that allows you to compare combinations of different parameters and values, the data will be in different tables as each run of VQM will produce its own table.

Can I use VQM?

As implied with "I can implement the ability...", not yet. I'll let you know when VQM supports the ability to specify a particular CRF value when comparing presets, or a particular preset when comparing CRF values.

If so, is there a way to do it using Overview Mode

It will probably work with Overview Mode. I will test to confirm this, when implementing the aforementioned feature.

[CrypticSignal]

By the way, I have updated my first comment to explain why calculating the average score after performing a transcode with preset veryslow and another transcode with CRF 18 in an attempt to deduce the VMAF score you'd get with "veryslow, CRF 18" doesn't make sense.

[nothing2obvi]

Thanks again for your responses.

I'll let you know when VQM supports the ability to specify a particular CRF value when comparing presets, or a particular preset when comparing CRF values.

Looking forward to it!

By the way, I have updated my first comment...

Got it. This is helpful, thank you.

[CrypticSignal]

@nothing2obvi After commit e9cef91, VQM allows you to set global encoder options (using FFmpeg syntax) that apply to all transcodes. So you can not only set a particular CRF value or preset, but also things like -x264-params keyint=123:min-keyint=20. It works with Overview Mode too.

Here's an example of how to set the preset to veryslow and the CRF value to 18, and use Overview Mode:

python3 main.py -i "test_videos/ForBiggerFun.mp4" -e libx264 --encoder-options="-crf 18" -p preset -v veryslow --interval 10 -cl 5

Another example, where the preset is set to slower and the CRF value is set to 16:

python3 main.py -i "test_videos/ForBiggerFun.mp4" -e libx264 --encoder-options="-crf 16" -p preset -v slower --interval 10 -cl 5

If you have any questions, or run into any issues, let me know.

[nothing2obvi]

Just tested it. This is awesome. Thank you!

[CrypticSignal]

Combination Mode has also been added.

To activate Combination Mode, specify the -c or --combinations argument, followed by a list of combinations you wish to compare. The list of combinations must be surrounded in quotes, and each combination must be separated by a comma.

For example, if you want to compare the quality achieved with:

• The combination of preset veryslow and a CRF value of 18
• The combination of preset slower and CRF value of 16

You would run something like:

python main.py -i "test_videos/ForBiggerFun.mp4" -e libx265 -c "preset veryslow crf 18,preset slower crf 16"

The table produced will look something like this:

VMAF values are in the format: Min | Standard Deviation | Mean
+--------------------------+-------------------+----------+-----------+----------------------+
|       Combination        | Encoding Time (s) |   Size   |  Bitrate  |         VMAF         |
+--------------------------+-------------------+----------+-----------+----------------------+
| -preset veryslow -crf 18 |       325.79      | 19.13 MB | 2.55 Mbps | 94.99 | 1.27 | 99.06 |
|  -preset slower -crf 16  |       211.81      | 24.06 MB | 3.20 Mbps | 95.62 | 1.14 | 99.23 |
+--------------------------+-------------------+----------+-----------+----------------------+

• Combination Mode can be used alongside Overview Mode.
• You need to decide whether you want to use the regular mode, which compares the quality metrics achieved with various values of one particular encoder parameter (using the -p and -v arguments), OR Combination Mode. You cannot do both.

[InB4DevOps]

Awesome addition!

[nothing2obvi]

@CrypticSignal This is really, really great. Thank you so much for your work on this.

Not a complaint, just an FYI, I ran Combination Mode with Overview Mode using 8 different combinations, and the labeling of the graph is a little off:

[CrypticSignal]

@nothing2obvi Fixed in the latest code ;)

[nothing2obvi]

Great! Thanks again.

[CrypticSignal]

I set the max y-axis value to highest VMAF score + 1 (but no higher than 100), and the lowest y-axis value to mininum VMAF score - 1 (but no lower than 0). Are you satisfied with these limits?