DAOS-13216 dfuse: Add a pre-read feature for non-cached files. #12015

When the kernel cache is in use but a file is not cached then pre-read the file on open. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> # ------------------------ >8 ------------------------ Skip-func-hw-test: true Skip-func-test: true Quick-Functional: true Test-tag: dfuse

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 · 2023-04-24T14:46:55Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/4/execution/node/1015/log

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-04-24T16:22:37Z

Test stage Build on Leap 15.4 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/6/execution/node/345/log

daosbuild1 · 2023-04-24T16:22:58Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/6/execution/node/371/log

daosbuild1 · 2023-04-24T16:27:10Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/6/execution/node/366/log

daosbuild1 · 2023-04-24T16:27:38Z

Test stage Build RPM on Leap 15.4 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/6/execution/node/353/log

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 · 2023-08-23T02:42:11Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/37/execution/node/1258/log

Features: dfuse

daosbuild1

LGTM. No errors found by checkpatch.

mchaarawi · 2023-09-12T20:14:35Z

have not dug into the code yet in this PR and have just some high level questions based on the comment of pre-read you added:

if we are doing this ls -R or recursive readdir like resnet does initially from every thread before reading any data, are we going to trigger this pre-read? (that would be disastrous). i don't think the file is opened in this case, but not sure.
in the case of a pre-read during the open, do we read 1 mib or up to the size of the file? is this querying the file size, or does fuse or dfuse already have queried the file size before issuing the open?
is there a practical application this is aimed for?

ashleypittman · 2023-09-12T20:18:40Z

have not dug into the code yet in this PR and have just some high level questions based on the comment of pre-read you added:

if we are doing this ls -R or recursive readdir like resnet does initially from every thread before reading any data, are we going to trigger this pre-read? (that would be disastrous). i don't think the file is opened in this case, but not sure.

ls -R will not call open so no, there would be no difference to that code path.

in the case of a pre-read during the open, do we read 1 mib or up to the size of the file? is this querying the file size, or does fuse or dfuse already have queried the file size before issuing the open?

The file size is known during open and returned to the kernel. This is then used for the read size. The open performance should be unaffected (and there's no I/O changes during open).

is there a practical application this is aimed for?

This was tested with the Resnet training stage and helped, it should be useful for any dataset ingest which is dominated by read of smaller files.

mchaarawi

looks good to me. im not clear though what happens in the case if the file was written to after the pre-read happens. Is the pre-read buffer dropped?

Features: dfuse Required-githooks: true

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-09-25T22:27:48Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/39/execution/node/1292/log

daosbuild1

LGTM. No errors found by checkpatch.

ashleypittman · 2023-09-26T10:29:02Z

looks good to me. im not clear though what happens in the case if the file was written to after the pre-read happens. Is the pre-read buffer dropped?

If the pre-read buffer is used then once it's exhausted (which might take several reads depending on application I/O sizes) then it will be released. If there is other I/O, either writes or reads from elsewhere in the file then the buffer will not be used, but it won't be released until close.

daltonbohning · 2023-09-26T16:15:37Z

src/client/dfuse/ops/read.c

+ if (oh->doh_linear_read_pos != position) {
+ DFUSE_TRA_DEBUG(oh, "disabling readahead");


So "linear read" is synonymous with "readahead"?

Apparently yes. This PR started are read-ahead after the first read before I realised that the speculative read could happen on open. This is debugging so I think it's Ok though.

I was thinking from the perspective of the variable names, etc., but not asking you to change this

https://github.com/daos-stack/daos/blob/master/src/client/dfuse/dfuse.h#L117-L126

linear read means the file was opened, read from start to end without seeking and closed. We already track this to short-circuit the last zero-length read so we use that data to know if to use the pre-read buffer.

daosbuild1 · 2023-09-27T00:46:01Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/40/execution/node/1268/log

daosbuild1 · 2023-09-27T02:58:25Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-12015/40/testReport/

daosbuild1 · 2023-09-27T11:30:34Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-12015/40/display/redirect

Required-githooks: true

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-09-29T03:21:32Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12015/41/execution/node/1267/log

Required-githooks: true

daosbuild1

LGTM. No errors found by checkpatch.

When the kernel cache is in use but a file is not cached then pre-read the file on open. This works for files up to the read buffer size (1Mb) and is enabled based on the I/O pattern of the last file closed in the same directory. Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> Signed-off-by: Jeff Olivier <jeffolivier@google.com>

daosbuild1 previously requested changes Apr 21, 2023

View reviewed changes

src/client/dfuse/ops/read.c Outdated Show resolved Hide resolved

ashleypittman force-pushed the amd/pre-read branch from eadb559 to 9561b04 Compare April 21, 2023 11:33

daosbuild1 reviewed Apr 21, 2023

View reviewed changes

ashleypittman added 2 commits April 24, 2023 10:05

Skip some logging.

b3af6ba

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

ashleypittman force-pushed the amd/pre-read branch from 9561b04 to b3af6ba Compare April 24, 2023 13:17

daosbuild1 reviewed Apr 24, 2023

View reviewed changes

Fix a potential crash on free.

5134632

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 reviewed Apr 24, 2023

View reviewed changes

Add some comments on testing.

6e8f258

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

ashleypittman changed the title ~~DAOS-0000 dfuse: Add a pre-read feature for non-cached files.~~ DAOS-13216 dfuse: Add a pre-read feature for non-cached files. Apr 24, 2023

Fix another crash.

332628c

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 reviewed Apr 24, 2023

View reviewed changes

Fix another crash.

ade793e

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 reviewed Apr 24, 2023

View reviewed changes

Fix compile.

2871a77

Test-tag: dfuse Required-githooks: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

Merge branch 'master' into amd/pre-read

036d047

Features: dfuse

daosbuild1 reviewed Sep 7, 2023

View reviewed changes

ashleypittman requested review from mchaarawi, jolivier23 and daltonbohning September 12, 2023 14:38

mchaarawi previously approved these changes Sep 22, 2023

View reviewed changes

Merge branch 'master' into amd/pre-read

931b578

Features: dfuse Required-githooks: true

ashleypittman dismissed mchaarawi’s stale review via 931b578 September 25, 2023 11:22

daosbuild1 reviewed Sep 25, 2023

View reviewed changes

daosbuild1 reviewed Sep 26, 2023

View reviewed changes

daltonbohning approved these changes Sep 26, 2023

View reviewed changes

ashleypittman requested a review from mchaarawi September 26, 2023 16:59

Merge branch 'master' into amd/pre-read

661c033

Required-githooks: true

daosbuild1 reviewed Sep 28, 2023

View reviewed changes

Merge branch 'master' into amd/pre-read

4d3f28e

Required-githooks: true

daosbuild1 reviewed Oct 5, 2023

View reviewed changes

jolivier23 approved these changes Oct 6, 2023

View reviewed changes

ashleypittman merged commit 875e233 into master Oct 7, 2023
32 of 34 checks passed

ashleypittman deleted the amd/pre-read branch October 7, 2023 07:34

ashleypittman mentioned this pull request Oct 9, 2023

Add a test for container handle evict ashleypittman/daos#23

Closed

		if (oh->doh_linear_read_pos != position) {
		DFUSE_TRA_DEBUG(oh, "disabling readahead");

DAOS-13216 dfuse: Add a pre-read feature for non-cached files. #12015

DAOS-13216 dfuse: Add a pre-read feature for non-cached files. #12015

Conversation

ashleypittman commented Apr 21, 2023

github-actions bot commented Apr 21, 2023 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Apr 21, 2023

daosbuild1 commented Apr 21, 2023

daosbuild1 commented Apr 21, 2023

daosbuild1 commented Apr 21, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Apr 21, 2023

daosbuild1 commented Apr 21, 2023

daosbuild1 commented Apr 21, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Apr 24, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Apr 24, 2023

daosbuild1 commented Apr 24, 2023

daosbuild1 commented Apr 24, 2023

daosbuild1 commented Apr 24, 2023

daosbuild1 commented Aug 23, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

mchaarawi commented Sep 12, 2023

ashleypittman commented Sep 12, 2023

mchaarawi left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Sep 25, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

ashleypittman commented Sep 26, 2023

daltonbohning Sep 26, 2023

Choose a reason for hiding this comment

ashleypittman Sep 26, 2023

Choose a reason for hiding this comment

daltonbohning Sep 26, 2023

Choose a reason for hiding this comment

ashleypittman Sep 26, 2023

Choose a reason for hiding this comment

daosbuild1 commented Sep 27, 2023

daosbuild1 commented Sep 27, 2023

daosbuild1 commented Sep 27, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Sep 29, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 21, 2023 •

edited

Loading