How to properly setup buffer read test. fio and dd shows very different result.

[jtweng]

Hello all,

I am currently undertaking a performance comparison study to evaluate read/write performance in different systems. While testing with directIO (direct=1), the results using fio seem promising.

However, when I employ fio for a sequential buffer read test to assess performance with cached files, I observe a significant discrepancy between the numbers reported by dd and fio. The dd command reports a throughput of 5GB/s, as expected, whereas fio reports the figure as 200MB/s.

In addition, during the fio run, I noticed heavy disk usage as observed with iostat 1. I expected that, with the test file in memory, the read operation wouldn't need to trigger disk read, which is what dd test shows.

Commands and results for clarity:
file is created and read once to make sure it sits in page cache.
dd command:
dd if=fio.data of=/dev/null bs=8k
fio command:
fio --name=io_uring --iodepth=1 --filename=./fio.data --rw=read --ioengine=io_uring --bs=8K --numjobs=1 --size=20G --direct=0

dd result:
2621440+0 records in
2621440+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 4.40825 s, 4.9 GB/s
fio result:
io_uring: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=1
......
Run status group 0 (all jobs):
READ: bw=188MiB/s (197MB/s), 188MiB/s-188MiB/s (197MB/s-197MB/s), io=20.0GiB (21.5GB), run=108961-108961msec

Disk stats (read/write):
dm-1: ios=81825/28, merge=0/0, ticks=195432/6644, in_queue=202076, util=99.87%, aggrios=82245/24, aggrmerge=0/0, aggrticks=179344/2979, aggrin_queue=182323, aggrutil=99.73%
md122: ios=82245/24, merge=0/0, ticks=179344/2979, in_queue=182323, util=99.73%, aggrios=41041/1301, aggrmerge=160/1137, aggrticks=91610/37929, aggrin_queue=129539, aggrutil=80.42%
sdd: ios=65625/1307, merge=256/1131, ticks=145871/51779, in_queue=197650, util=80.42%
sdc: ios=16457/1295, merge=64/1143, ticks=37350/24079, in_queue=61428, util=26.32%

I am running fio-3.35 on Red Hat Enterprise Linux release 9.3 (Kernel: 5.14.0-362.8.1.el9_3.x86_64).

Any insights into this discrepancy and suggestions for further investigation would be greatly appreciated.

Thank you!
Jerry

[axboe]

Fio will invalidate the cache if the file before reading it, as without that you're just measuring how fast you can copy pages and not doing any io. You can then this off with invalidate=0.

[jtweng]

Thank you for the response.

After fill in invalidate=0, I increase the throughput from 150MB/s to 1GB/s.
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=1095MiB/s (1148MB/s), 1095MiB/s-1095MiB/s (1148MB/s-1148MB/s), io=20.0GiB (21.5GB), run=18700-18700msec

It is still reporting 1/5 throughput comparing with DD.
I further increase iodepth to from 1 to a bigger value, and then the throughput increases and capped at 4GB/s
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
READ: bw=3948MiB/s (4140MB/s), 3948MiB/s-3948MiB/s (4140MB/s-4140MB/s), io=20.0GiB (21.5GB), run=5187-5187msec

These performance number now makes a lot more sense. Thanks!