v1 / v2 Speed Comparison

[akaikite]

TheBloke_FreeWilly2-GPTQ_gptq-4bit-32g-actorder_True [70b]

gpu - a6000

v2

Output generated in 316.86 seconds (9.96 tokens/s, 3156 tokens, context 941, seed 839325157)

Output generated in 327.32 seconds (9.64 tokens/s, 3156 tokens, context 941, seed 964072975)

Output generated in 11.19 seconds (9.38 tokens/s, 105 tokens, context 941, seed 2138389161)

Output generated in 32.63 seconds (10.97 tokens/s, 358 tokens, context 1046, seed 698368887)

Output generated in 45.70 seconds (10.42 tokens/s, 476 tokens, context 1404, seed 267215585)

Output generated in 50.37 seconds (9.73 tokens/s, 490 tokens, context 1880, seed 1519311258)

Output generated in 71.70 seconds (9.16 tokens/s, 657 tokens, context 2370, seed 1991901992)

Output generated in 62.19 seconds (8.43 tokens/s, 524 tokens, context 3027, seed 1124908778)

Output generated in 46.36 seconds (7.49 tokens/s, 347 tokens, context 3551, seed 1028076373)

v1

Output generated in 398.96 seconds (7.91 tokens/s, 3156 tokens, context 941, seed 1913709604)

Output generated in 13.17 seconds (7.52 tokens/s, 99 tokens, context 941, seed 632398902)

Output generated in 50.28 seconds (8.81 tokens/s, 443 tokens, context 1040, seed 1079219730)

Output generated in 57.31 seconds (8.45 tokens/s, 484 tokens, context 1483, seed 464509141)

Output generated in 67.15 seconds (8.07 tokens/s, 542 tokens, context 1967, seed 633887777)

Output generated in 45.11 seconds (7.78 tokens/s, 351 tokens, context 2509, seed 2075010672)

Output generated in 42.57 seconds (7.56 tokens/s, 322 tokens, context 2860, seed 358840736)

Output generated in 55.01 seconds (7.38 tokens/s, 406 tokens, context 3182, seed 12195232)

Output generated in 71.95 seconds (7.07 tokens/s, 509 tokens, context 3588, seed 486408052)

It's definitely faster. I applaud this work.