-max-rate gives much lower throughput than desired

[tgrabiec]

 :~/src/scylla-bench $ ./scylla-bench -workload=sequential -mode=write -replication-factor=1 -partition-count=10 -clustering-row-count=10000000 -clustering-row-size=5120 -concurrency=200 -rows-per-request=10 -max-rate=100000
Configuration
Mode:			 write
Workload:		 sequential
Timeout:		 5s
Consistency level:	 quorum
Partition count:	 10
Clustering rows:	 10000000
Clustering row size:	 5120
Rows per request:	 10
Page size:		 1000
Concurrency:		 200
Connections:		 4
Maximum rate:		 100000 op/s
Client compression:	 true

time		operations/s	rows/s		errors	max		99.9th		99th		95th		90th		median
^C
interrupted

Results
Time (avg):	 30.239109ms
Total ops:	 3026
Total rows:	 30250
Operations/s:	 500.349773593381
Rows/s:		 5001.84423370779
Latency:
  max:		 2.850815ms 
  99.9th:	 1.376255ms 
  99th:		 1.015807ms 
  95th:		 753.663µs 
  90th:		 688.127µs 
  median:	 622.591µs

 :~/src/scylla-bench $ ./scylla-bench -workload=sequential -mode=write -replication-factor=1 -partition-count=10 -clustering-row-count=10000000 -clustering-row-size=5120 -concurrency=200 -rows-per-request=10 -max-rate=80000
Configuration
Mode:			 write
Workload:		 sequential
Timeout:		 5s
Consistency level:	 quorum
Partition count:	 10
Clustering rows:	 10000000
Clustering row size:	 5120
Rows per request:	 10
Page size:		 1000
Concurrency:		 200
Connections:		 4
Maximum rate:		 80000 op/s
Client compression:	 true

time		operations/s	rows/s		errors	max		99.9th		99th		95th		90th		median
^C
interrupted

Results
Time (avg):	 7.300328ms
Total ops:	 585
Total rows:	 5850
Operations/s:	 400.72400209705376
Rows/s:		 4007.240020970538
Latency:
  max:		 2.719743ms 
  99.9th:	 1.441791ms 
  99th:		 1.081343ms 
  95th:		 819.199µs 
  90th:		 720.895µs 
  median:	 655.359µs

 :~/src/scylla-bench $ ./scylla-bench -workload=sequential -mode=write -replication-factor=1 -partition-count=10 -clustering-row-count=10000000 -clustering-row-size=5120 -concurrency=200 -rows-per-request=10 -max-rate=70000
Configuration
Mode:			 write
Workload:		 sequential
Timeout:		 5s
Consistency level:	 quorum
Partition count:	 10
Clustering rows:	 10000000
Clustering row size:	 5120
Rows per request:	 10
Page size:		 1000
Concurrency:		 200
Connections:		 4
Maximum rate:		 70000 op/s
Client compression:	 true

time		operations/s	rows/s		errors	max		99.9th		99th		95th		90th		median
^C
interrupted

Results
Time (avg):	 11.012372ms
Total ops:	 773
Total rows:	 7720
Operations/s:	 350.9807348881303
Rows/s:		 3505.2668477831385
Latency:
  max:		 1.310719ms 
  99.9th:	 1.212415ms 
  99th:		 1.081343ms 
  95th:		 786.431µs 
  90th:		 720.895µs 
  median:	 655.359µs

[dahankzter]

Was the generator machine fully utilized in this cases?

[dahankzter]

Did you update scylla-bench? We switched to using the shard aware driver a few days ago. Perhaps it causes a regression. Can you try to down grade? Should be only the top commit or so.