Skip to content

Commit

Permalink
engines/fileoperations: add more description for file/directory opera…
Browse files Browse the repository at this point in the history 
…tion engines

file/directory operations are quite different from I/O operation.
Add more description for how to configure the measurement
and how to utilize the data.

Signed-off-by: friendy-su <friendy.su@sony.com>
  • Loading branch information
@friendy-su
friendy-su committed Apr 15, 2024
1 parent 4eef23f commit 2cd920c
Show file tree
Hide file tree
Showing 2 changed files with 171 additions and 63 deletions.
111 changes: 80 additions & 31 deletions HOWTO.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1992,7 +1992,9 @@ I/O engine

.. option:: ioengine=str

Defines how the job issues I/O to the file. The following types are defined:
fio supports 2 kinds of performance measurement: I/O and file/directory operation.

I/O engines define how the job issues I/O to the file. The following types are defined:

**sync**
Basic :manpage:`read(2)` or :manpage:`write(2)`
Expand Down Expand Up @@ -2177,36 +2179,6 @@ I/O engine
absolute or relative. See :file:`engines/skeleton_external.c` for
details of writing an external I/O engine.

**filecreate**
Simply create the files and do no I/O to them. You still need to
set `filesize` so that all the accounting still occurs, but no
actual I/O will be done other than creating the file.

**filestat**
Simply do stat() and do no I/O to the file. You need to set 'filesize'
and 'nrfiles', so that files will be created.
This engine is to measure file lookup and meta data access.

**filedelete**
Simply delete the files by unlink() and do no I/O to them. You need to set 'filesize'
and 'nrfiles', so that the files will be created.
This engine is to measure file delete.

**dircreate**
Simply create the directories and do no I/O to them. You still need to
set `filesize` so that all the accounting still occurs, but no
actual I/O will be done other than creating the directories.

**dirstat**
Simply do stat() and do no I/O to the directories. You need to set 'filesize'
and 'nrfiles', so that directories will be created.
This engine is to measure directory lookup and meta data access.

**dirdelete**
Simply delete the directories by rmdir() and do no I/O to them. You need to set 'filesize'
and 'nrfiles', so that the directories will be created.
This engine is to measure directory delete.

**libpmem**
Read and write using mmap I/O to a file on a filesystem
mounted with DAX on a persistent memory device through the PMDK
Expand Down Expand Up @@ -2276,6 +2248,50 @@ I/O engine
several instances to access the same device or file
simultaneously, but allow it for threads.

File/directory operation engines define how the job operates file or directory. The
following types are defined:

**filecreate**
Simply create the files and do no I/O to them. You still need to
set `filesize` so that all the accounting still occurs, but no
actual I/O will be done other than creating the file.
Example job file: filecreate-ioengine.fio.

**filestat**
Simply do stat() and do no I/O to the file. You need to set 'filesize'
and 'nrfiles', so that files will be created.
This engine is to measure file lookup and meta data access.
Example job file: filestat-ioengine.fio.

**filedelete**
Simply delete the files by unlink() and do no I/O to them. You need to set 'filesize'
and 'nrfiles', so that the files will be created.
This engine is to measure file delete.
Example job file: filedelete-ioengine.fio.

**dircreate**
Simply create the directories and do no I/O to them. You still need to
set `filesize` so that all the accounting still occurs, but no
actual I/O will be done other than creating the directories.
Example job file: dircreate-ioengine.fio.

**dirstat**
Simply do stat() and do no I/O to the directories. You need to set 'filesize'
and 'nrfiles', so that directories will be created.
This engine is to measure directory lookup and meta data access.
Example job file: dirstat-ioengine.fio.

**dirdelete**
Simply delete the directories by rmdir() and do no I/O to them. You need to set 'filesize'
and 'nrfiles', so that the directories will be created.
This engine is to measure directory delete.
Example job file: dirdelete-ioengine.fio.

For file and directory operation engines, there is no I/O throughput, then the
statistics data in report have different meanings. The meaningful output indexes are: 'iops' and 'clat'.
'bw' is meaningless. Refer to section: "Interpreting the output" for more details.


I/O engine specific parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -4583,6 +4599,21 @@ writes in the example above). In the order listed, they denote:
commit if available) functions were completed to when the I/O's
completion was reaped by fio.

For file and directory operation engines, **clat** denotes the time
to complete one file or directory operation.

**filecreate engine**:the time cost to create a new file

**filestat engine**: the time cost to look up an existing file

**filedelete engine**:the time cost to delete a file

**dircreate engine**: the time cost to create a new directory

**dirstat engine**: the time cost to look up an existing directory

**dirdelete engine**: the time cost to delete a directory

**lat**
Total latency. Same names as slat and clat, this denotes the time from
when fio created the I/O unit to completion of the I/O operation.
Expand All @@ -4601,12 +4632,30 @@ writes in the example above). In the order listed, they denote:
are on the same disk, since they are then competing for disk
access.

For file and directory operation engines, **bw** is meaningless.

**iops**
IOPS statistics based on measurements from discrete intervals.
For details see the description for bw above. See
:option:`iopsavgtime` to control the duration of the intervals.
Same values reported here as for bw except for percentage.

For file and directory operation engines, **iops** is the most
fundamental index to denote the performance.
It means how many files or directories can be operated per second.

**filecreate engine**:number of files can be created per second

**filestat engine**: number of files can be looked up per second

**filedelete engine**:number of files can be deleted per second

**dircreate engine**: number of directories can be created per second

**dirstat engine**: number of directories can be looked up per second

**dirdelete engine**: number of directories can be deleted per second

**lat (nsec/usec/msec)**
The distribution of I/O completion latencies. This is the time from when
I/O leaves fio and when it gets completed. Unlike the separate
Expand Down
123 changes: 91 additions & 32 deletions fio.1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1794,8 +1794,9 @@ started on the result.
.SS "I/O engine"
.TP
.BI ioengine \fR=\fPstr
Defines how the job issues I/O to the file. The following types are defined:
.RS
fio supports 2 kinds of performance measurement: I/O and file/directory operation.

I/O engines define how the job issues I/O to the file. The following types are defined:
.RS
.TP
.B sync
Expand Down Expand Up @@ -1989,36 +1990,6 @@ ioengine `foo.o' in `/tmp'. The path can be either
absolute or relative. See `engines/skeleton_external.c' in the fio source for
details of writing an external I/O engine.
.TP
.B filecreate
Simply create the files and do no I/O to them. You still need to set
\fBfilesize\fR so that all the accounting still occurs, but no actual I/O will be
done other than creating the file.
.TP
.B filestat
Simply do stat() and do no I/O to the file. You need to set 'filesize'
and 'nrfiles', so that files will be created.
This engine is to measure file lookup and meta data access.
.TP
.B filedelete
Simply delete files by unlink() and do no I/O to the file. You need to set 'filesize'
and 'nrfiles', so that files will be created.
This engine is to measure file delete.
.TP
.B dircreate
Simply create the directories and do no I/O to them. You still need to set
\fBfilesize\fR so that all the accounting still occurs, but no actual I/O will be
done other than creating the directories.
.TP
.B dirstat
Simply do stat() and do no I/O to the directory. You need to set 'filesize'
and 'nrfiles', so that directories will be created.
This engine is to measure directory lookup and meta data access.
.TP
.B dirdelete
Simply delete directories by unlink() and do no I/O to the directory. You need to set 'filesize'
and 'nrfiles', so that directories will be created.
This engine is to measure directory delete.
.TP
.B libpmem
Read and write using mmap I/O to a file on a filesystem
mounted with DAX on a persistent memory device through the PMDK
Expand Down Expand Up @@ -2079,6 +2050,55 @@ instance is used per process, so all jobs setting option \fBthread\fR will share
a single instance (with one queue per thread) and must specify compatible
options. Note that some drivers don't allow several instances to access the same
device or file simultaneously, but allow it for threads.
.TP
.RE
.P
File/directory operation engines define how the job operates file or directory.
The following types are defined:
.RS
.TP
.B filecreate
Simply create the files and do no I/O to them. You still need to
set \fBfilesize\fP so that all the accounting still occurs, but no
actual I/O will be done other than creating the file.
Example job file: filecreate-ioengine.fio.
.TP
.B filestat
Simply do stat() and do no I/O to the file. You need to set \fBfilesize\fP
and \fBnrfiles\fP, so that files will be created.
This engine is to measure file lookup and meta data access.
Example job file: filestat-ioengine.fio.
.TP
.B filedelete
Simply delete the files by unlink() and do no I/O to them. You need to set \fBfilesize\fP
and \fBnrfiles\fP, so that the files will be created.
This engine is to measure file delete.
Example job file: filedelete-ioengine.fio.
.TP
.B dircreate
Simply create the directories and do no I/O to them. You still need to
set \fBfilesize\fP so that all the accounting still occurs, but no
actual I/O will be done other than creating the directories.
Example job file: dircreate-ioengine.fio.
.TP
.B dirstat
Simply do stat() and do no I/O to the directories. You need to set \fBfilesize\fP
and \fBnrfiles\fP, so that directories will be created.
This engine is to measure directory lookup and meta data access.
Example job file: dirstat-ioengine.fio.
.TP
.B dirdelete
Simply delete the directories by rmdir() and do no I/O to them. You need to set \fBfilesize\fP
and \fBnrfiles\fP, so that the directories will be created.
This engine is to measure directory delete.
.TP
.RE
.P
For file and directory operation engines, there is no I/O throughput, then the statistics \
data in report have different meanings. The meaningful output indexes are: \fBiops\fP and \fBclat\fP. \
\fBbw\fP is meaningless. Refer to section: "Interpreting the output" for more details.
.RE
.P
.SS "I/O engine specific parameters"
In addition, there are some parameters which are only valid when a specific
\fBioengine\fR is in use. These are used identically to normal parameters,
Expand Down Expand Up @@ -4236,6 +4256,24 @@ submission to completion of the I/O pieces. For sync I/O, clat will
usually be equal (or very close) to 0, as the time from submit to
complete is basically just CPU time (I/O has already been done, see slat
explanation).

For file and directory operation engines, \fBclat\fP denotes the time
to complete one file or directory operation.
.RS
.TP
\fBfilecreate engine\fP:\tthe time cost to create a new file
.TP
\fBfilestat engine\fP:\tthe time cost to look up an existing file
.TP
\fBfiledelete engine\fP:\tthe time cost to delete a file
.TP
\fBdircreate engine\fP:\tthe time cost to create a new directory
.TP
\fBdirstat engine\fP:\tthe time cost to look up an existing directory
.TP
\fBdirdelete engine\fP:\tthe time cost to delete a directory
.TP
.RE
.TP
.B lat
Total latency. Same names as slat and clat, this denotes the time from
Expand All @@ -4250,12 +4288,33 @@ xlat stats, but also includes the number of samples taken (\fIsamples\fR) and an
approximate percentage of total aggregate bandwidth this thread received in its
group (\fIper\fR). This last value is only really useful if the threads in this
group are on the same disk, since they are then competing for disk access.

For file and directory operation engines, \fBbw\fR is meaningless.
.TP
.B iops
IOPS statistics based on measurements from discrete intervals.
For details see the description for \fBbw\fR above. See
\fBiopsavgtime\fR to control the duration of the intervals.
Same values reported here as for \fBbw\fR except for percentage.

For file and directory operation engines, \fBiops\fP is the most
fundamental index to denote the performance.
It means how many files or directories can be operated per second.
.RS
.TP
\fBfilecreate engine\fP:\tnumber of files can be created per second
.TP
\fBfilestat engine\fP:\tnumber of files can be looked up per second
.TP
\fBfiledelete engine\fP:\tnumber of files can be deleted per second
.TP
\fBdircreate engine\fP:\tnumber of directories can be created per second
.TP
\fBdirstat engine\fP:\tnumber of directories can be looked up per second
.TP
\fBdirdelete engine\fP:\tnumber of directories can be deleted per second
.TP
.RE
.TP
.B lat (nsec/usec/msec)
The distribution of I/O completion latencies. This is the time from when
Expand Down

0 comments on commit 2cd920c

Please sign in to comment.