ioengines: Implement dircreate, dirstat, dirdelete engines

[friendy-su]

test log:

directory create:

$ ../fio --group_reporting dircreate-ioengine.fio
t0: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dircreate, iodepth=1
t1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dircreate, iodepth=1
t2: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dircreate, iodepth=1
t3: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dircreate, iodepth=1
t4: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dircreate, iodepth=1
t5: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dircreate, iodepth=1
fio-3.36
Starting 6 processes

t0: (groupid=0, jobs=6): err= 0: pid=610064: Fri Jan 26 12:21:39 2024
read: IOPS=90.0k, BW=352MiB/s (369MB/s)(720KiB/2msec)
clat (usec): min=35, max=105, avg=55.18, stdev= 9.26
clat percentiles (usec):
| 1.00th=[ 37], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49],
| 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 57],
| 70.00th=[ 57], 80.00th=[ 59], 90.00th=[ 63], 95.00th=[ 66],
| 99.00th=[ 92], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106],
| 99.99th=[ 106]
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=66
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=180,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=720KiB (737kB), run=2-2msec

directory stat:

$ ../fio --group_reporting dirstat-ioengine.fio
t0: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dirstat, iodepth=1
...
t1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dirstat, iodepth=1
...
fio-3.36
Starting 20 threads

t0: (groupid=0, jobs=20): err= 0: pid=610107: Fri Jan 26 12:22:40 2024
read: IOPS=50.0k, BW=195MiB/s (205MB/s)(400KiB/2msec)
clat (nsec): min=1096, max=19725, avg=3404.90, stdev=3309.08
clat percentiles (nsec):
| 1.00th=[ 1096], 5.00th=[ 1224], 10.00th=[ 1272], 20.00th=[ 1464],
| 30.00th=[ 2040], 40.00th=[ 2160], 50.00th=[ 2224], 60.00th=[ 2320],
| 70.00th=[ 2512], 80.00th=[ 3632], 90.00th=[ 7776], 95.00th=[ 7968],
| 99.00th=[19584], 99.50th=[19840], 99.90th=[19840], 99.95th=[19840],
| 99.99th=[19840]
cpu : usr=0.00%, sys=0.00%, ctx=5, majf=0, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=100,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=195MiB/s (205MB/s), 195MiB/s-195MiB/s (205MB/s-205MB/s), io=400KiB (410kB), run=2-2msec

directory delete:

$ ../fio --group_reporting dirdelete-ioengine.fio
t0: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dirdelete, iodepth=1
t1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dirdelete, iodepth=1
t2: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dirdelete, iodepth=1
t3: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dirdelete, iodepth=1
t4: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dirdelete, iodepth=1
t5: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=dirdelete, iodepth=1
fio-3.36
Starting 6 processes

t0: (groupid=0, jobs=6): err= 0: pid=610164: Fri Jan 26 12:23:38 2024
read: IOPS=70.6k, BW=276MiB/s (289MB/s)(4800KiB/17msec)
clat (usec): min=5, max=8170, avg=70.37, stdev=443.95
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 23],
| 30.00th=[ 25], 40.00th=[ 26], 50.00th=[ 27], 60.00th=[ 28],
| 70.00th=[ 33], 80.00th=[ 48], 90.00th=[ 105], 95.00th=[ 122],
| 99.00th=[ 265], 99.50th=[ 412], 99.90th=[ 8094], 99.95th=[ 8160],
| 99.99th=[ 8160]
cpu : usr=0.00%, sys=58.54%, ctx=836, majf=0, minf=168
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1200,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=4800KiB (4915kB), run=17-17msec

[vincentkfu]

The code in the new source file duplicates a lot of the code in fileoperations.c. Why not just add code for these new ioengines to fileoperations.c?

Also, in the documentation point readers to the example jobs or some other location documenting the unique way that these ioengines operate to forestall issues like #1699

[friendy-su]

The code in the new source file duplicates a lot of the code in fileoperations.c. Why not just add code for these new ioengines to fileoperations.c?

fileoperations.c and diroperations.c code framework is similar. But there are still many difference. E.g. file operation engine is no need to prepare files for stat/delete. But dir operation need prepare directories by itself, since fio will not do mkdir in setup_files(). Another difference like file operation uses unlink(), dir operation uses rmdir(). These trivial difference causes hard to reuse code. So I think it is more clear to keep independent fileoperations.c and diroperations.c.

Also, in the documentation point readers to the example jobs or some other location documenting the unique way that these ioengines operate to forestall issues like #1699

OK. I will create another patch to append more info into document.

[vincentkfu]

The first 60 lines of fileoperations.c and diroperations.c are the same and there are another 40 lines beginning with static int invalidate_do_nothing(struct thread_data *td, struct fio_file *f) that are the same. Why do you want to duplicate these 100 lines across the two source files?