"cuBLAS.and.cuDNN" version differences?

[ghost]

What are differences between v2 and v3 Cublas and cudnn except kepler chip support, I am running it on RTX 3050. Also do I have to download the pytorch that in the repos even tho I installed pytorch 2.1.2 with pip? I asked it cus maybe its some fine tuned stuff

[Purfview]

What are differences between v2 and v3 Cublas and cudnn except kepler chip support, I am running it on RTX 3050.

v3 currently contains the latest libs, v2 contains the last with Kepler support. I don't know the other differences, you can read the docs at NVIDIA site.
Or you can do own benchmarks, use longer audio and run 3 times with one libs then run 3 times with other libs, look at the transcription speeds. Then report the results of the benchmarks.

Use -fallback=None to ensure that benchmarks are consistent.

Also do I have to download the pytorch that in the repos even tho I installed pytorch 2.1.2 with pip?

You don't need installed Python at all.

[Purfview]

Benchmarks posted below were reported by @despairTK :

Benchmark results of 1:15:42 long audio on RTX 4090 with --model=large-v2 --compute_type=float16 -fallback=None

libs v1:

libs v2:

libs v3:

[ghost]

I was doing a Benchmark with rtx 3050 37 min audio --model=large-v2 --sentence -fallback=None

v3 
1 Transcription speed: 4.31 audio seconds/s
2 Transcription speed: 4.37 audio seconds/s
3 Transcription speed: 4.43 audio seconds/s

I am on v2 rn, but what is weird to me is that whisper-faster says that it runs on cuda but my gpu utilization is around %3-5 it feels like it does not use my gpu :D I do have CUDA 11.8

[Purfview]

If it says that it runs CUDA then you can be sure that it runs on GPU.
You can use medium model for faster benchmarks.

Btw, talking about libs, CTranslate2 is distributed with one cuDNN lib - cudnn64_8, I think it's v8.1.1, which is pretty old.
I wonder if it would make a difference to update it. I can make a test build with it updated.

[ghost]

Okay benchmark ended finally :D
Environment: Windows 10 Pro 22H2, Ryzen 5 5500 CPU, RTX 3050 GPU, 8x2 3200mhz DDR4 ram
Whisper command: --model=large-v2 --sentence -fallback=None
Audio: 37:48 min, opus, 48.000khz

V3 Results:

1st: Transcription speed: 4.31 audio seconds/s
2nd: Transcription speed: 4.37 audio seconds/s
3rd: Transcription speed: 4.43 audio seconds/s
Avg. speed: 4.37 audio seconds/s

V2 Results:

1st: Transcription speed: 4.52 audio seconds/s
2nd: Transcription speed: 4.21 audio seconds/s
3rd: Transcription speed: 4.21 audio seconds/s
Avg. speed: 4.31333333333 audio second/a

[Purfview]

Looks like v3 is insignificantly faster, practically equal.
Benchmark r167.2-test vs "V3 Results", it's posted below.

Btw did you used the latest r167.2 version?

[ghost]

I used non-test version but yeah latest one that i could find in the repo, started to benchmark rn with large-v2 again, i will send the benchmark probably 1-2 hours later cus I have to do something.

[Purfview]

Here you can download r167.2-test for benchmarks -> https://we.tl/t-qZHfGTUxgc

It has cudnn64_8 updated to v8.9.6.50, same version as the rest of cuDNN libs from the v3 libs pack.

[ghost]

Next time i will do it like that 😆😆

[Purfview]

Use medium model for the faster tests.

[Purfview]

In your tests you didn't wrote what exact compute type did you tested.

And I just now noticed that there are newer cuDNN libs released, there I uploaded them -> https://we.tl/t-ibgWViCW0b

[ghost]

@Purfview cus i use it on auto mode, i did nto see the cudnn libs so i did benchmark with old libs i am now int8_float32 of v3, it will be finished like in 30 min ig.

[Purfview]

i use it on auto mode

You can see it with --verbose=true.

[ghost]

Environment: Windows 10 Pro 22H2, Ryzen 5 5500 CPU, RTX 3050 GPU, 8x2 3200mhz DDR4 ram
Whisper command: --model=medium --sentence -fallback=None
Audio: 37:48 min, opus, 48.000khz
Whisper-Faster Version: r167.2 non-test version.

v2 Results:

bfloat16:      Average Speed: 7.98 audio seconds/s
float16:       Average Speed: 7.46 audio seconds/s
int8_bfloat16: Average Speed: 6.18 audio seconds/s
int8_float16:  Average Speed: 5.88 audio seconds/s
float32:       Average Speed: 5.26 audio seconds/s
int8_float32:  Average Speed: 4.96 audio seconds/s

v3 Results:

bfloat16:      Average Speed: 8.21 audio seconds/s
float16:       Average Speed: 7.03 audio seconds/s
int8_float16:  Average Speed: 5.39 audio seconds/s
int8_bfloat16: Average Speed: 5.26 audio seconds/s
int8_float32:  Average Speed: 5.04 audio seconds/s
float32:       Average Speed: 4.87 audio seconds/s

[Purfview]

Expected... Allegedly auto should select the fastest. But it doesn't...
Maybe I need to write my own "auto".

I still don't know what to select which compute type i should use since accuracy and stuff u know

There is no "accuratest", so, imho don't bother.
On the different audios - the different quantizations can be more/less accurate.

[ghost]

alr so i will be using fastest one which is bfloat16 on my end

[Purfview]

In theory int8 should be a bit less accurate, in practice - I've seen the clear opposite effect on some audios.

[ghost]

Environment, commands etc. everything is the same as above, but since v3 is faster, I benchmarked only with v3, each compute_type was tested 3 times, but I only put the average values.

v3 [updated with v8.9.7.29] Results:

bfloat16:      Average Speed: 8.46 audio seconds/s
float16:       Average Speed: 7.27 audio seconds/s
int8_float16:  Average Speed: 6.79 audio seconds/s
int8_bfloat16: Average Speed: 6.59 audio seconds/s
int8_float32:  Average Speed: 5.26 audio seconds/s
float32:       Average Speed: 5.11 audio seconds/s

[Purfview]

Nice, looks like those new libs are a bit faster. bfloat16 is faster by 3%.

[Purfview]

Here you go, test2 build with CUDA12 libs ->https://we.tl/t-H37FJfdgkF

[Purfview]

You need to disable "CUDA Sysmem Fallback",
And normal r167.2 + v4 libs test on the same driver.

[ghost]

ok will do it in sometime

[ghost]

btw here is the new benchmark about cuda12 version.
Environment: Windows 10 Pro 22H2, Ryzen 5 5500 CPU, RTX 3050 GPU, 8x2 3200mhz DDR4 ram
Whisper command: --model=medium --sentence -fallback=None -pp --compute_type=
Audio: 37:48 min, opus, 48.000khz
System's CUDA Version: 12.3
Nvidia Driver Version: 546.33

CUDA Sysmem Fallback Policy: Pref. no sysmem fallback

test2cudaw2Results:

bfloat16:     Average Speed: 13.09  audio second/s
float16:      Average Speed: 12.17  audio second/s 
int8_float16: Average Speed: 10.80  audio second/s

[ghost]

Environment: Windows 10 Pro 22H2, Ryzen 5 5500 CPU, RTX 3050 GPU, 8x2 3200mhz DDR4 ram
Whisper command: --model=medium --sentence -fallback=None -pp --compute_type=
Audio: 37:48 min, opus, 48.000khz
System's CUDA Version: 12.3
Nvidia Driver Version: 546.33
CUDA Sysmem Fallback Policy: Pref. no sysmem fallback

v4 Results:

bfloat16:     Average Speed: 13.73 audio second/s
float16:      Average Speed: 12.24 audio second/s
int8_float16: Average Speed: 13.68 audio second/s

[Purfview]

In new tests sticks out that int8 is not slow anymore with CUDA11, it still slow with CUDA12.
Just that now int8_float16 was tested, but I don't think that int8 types should differ much.
Maybe test int8_bfloat16, int8_float32 and float32 too under this new environment.

Conclusion of the new tests:

float16:       -1% drop in speed 
bfloat16:      -5% drop in speed
int8_float16: -21% drop in speed

For the reference the old one [different environment]:

float16:      -10% drop in speed 
bfloat16:      -8% drop in speed
int8_bfloat16:  0% same

[ghost]

v4 for linux when @Purfview

[Purfview]

Did you switched to Linux?
"cuBLAS.and.cuDNN_linux_v4.7z" is uploaded now.

[ghost]

yup currently using arch