From c8533563eb559b3675ed256d7ad3aa62acfc0775 Mon Sep 17 00:00:00 2001 From: Sean Lee Date: Tue, 6 Aug 2024 20:31:14 +0800 Subject: [PATCH] Improve http_host, filename in docs and example/http-s3.fio In fio.1 and HOWTO.rst: * http_host: add details on vHost- vs Path-style hostname differences * filename: add details on bucket prefix for Path-style hostname In examples/http-s3.fio: * add comments at the top to clarify vHost vs Path S3 and change the hostname in existing example to disambiguate two scenarios Signed-off-by: Sean Lee --- HOWTO.rst | 14 ++++++++++++-- examples/http-s3.fio | 16 +++++++++++++--- fio.1 | 11 +++++++++-- 3 files changed, 34 insertions(+), 7 deletions(-) diff --git a/HOWTO.rst b/HOWTO.rst index 9eeb203e58..11b1c66934 100644 --- a/HOWTO.rst +++ b/HOWTO.rst @@ -805,6 +805,11 @@ Target file/device Note: Windows and FreeBSD (refer to geom(4)) prevent write access to areas of the disk containing in-use data (e.g. filesystems). + For HTTP and S3 access, specify a valid URL path or S3 key, respectively. + A filename for path-style S3 includes a bucket name (:file:`/bucket/k/e.y`) + while a virtual-hosted-style S3 filename :file:`/k/e.y` does not because + its bucket name is specified in :option:`http_host`. + The filename "`-`" is a reserved name, meaning *stdin* or *stdout*. Which of the two depends on the read/write direction set. @@ -2934,8 +2939,13 @@ with the caveat that when used on the command line, they must come after the .. option:: http_host=str : [http] - Hostname to connect to. For S3, this could be the bucket hostname. - Default is **localhost** + Hostname to connect to. HTTP port 80 is used automatically when the value of + the https parameter is *off*, and HTTPS port 443 if it is *on*. A + virtual-hosted-style S3 hostname starts with a bucket name, while a + path-style S3 hostname does not. See + https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html for + detailed examples. + Default is **localhost** (path-style S3 hostname) .. option:: http_user=str : [http] diff --git a/examples/http-s3.fio b/examples/http-s3.fio index 2dcae364a9..c16e89b169 100644 --- a/examples/http-s3.fio +++ b/examples/http-s3.fio @@ -1,19 +1,29 @@ # Example test for the HTTP engine's S3 support against Amazon AWS. # Obviously, you have to adjust the S3 credentials; for this example, # they're passed in via the environment. -# +# For non-AWS S3 implementations, refer to your S3 vendor's region +# settings. Note that the region value appears twice, in http_host and +# http_s3_region. +# This example uses virtual-hosted-style requests: +# https://bucket-name.s3.region-code.amazonaws.com/k/e.y +# For path-style, prefix the key with the bucket name in the filename +# so that filename=/bucket-name/k/e.y: +# https://s3.region-code.amazonaws.com/bucket-name/k/e.y + +# Reference for Virtual-hosted-style vs. Path-style URLs: +# https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html [global] ioengine=http name=test direct=1 -filename=/larsmb-fio-test/object +filename=/k/e.y http_verbose=0 https=on http_mode=s3 http_s3_key=${S3_KEY} http_s3_keyid=${S3_ID} -http_host=s3.eu-central-1.amazonaws.com +http_host=bucket-name.s3.eu-central-1.amazonaws.com http_s3_region=eu-central-1 group_reporting diff --git a/fio.1 b/fio.1 index 15bafbf9ba..6db76b2cd2 100644 --- a/fio.1 +++ b/fio.1 @@ -585,6 +585,11 @@ the first device, `\\\\.\\PhysicalDrive1' for the second etc. Note: Windows and FreeBSD prevent write access to areas of the disk containing in-use data (e.g. filesystems). .P +For HTTP and S3 access, specify a valid URL path or S3 key, respectively. +A filename for path-style S3 includes a bucket name (`/bucket/k/e.y') +while a virtual-hosted-style S3 filename (`/k/e.y') does not because its +bucket name is specified in \fBhttp_host\fR. +.P The filename `\-' is a reserved name, meaning *stdin* or *stdout*. Which of the two depends on the read/write direction set. .RE @@ -2554,8 +2559,10 @@ Touching all objects affects ceph caches and likely impacts test results. Enabled by default. .TP .BI (http)http_host \fR=\fPstr -Hostname to connect to. For S3, this could be the bucket name. Default -is \fBlocalhost\fR +Hostname to connect to. HTTP port 80 is used automatically when the value +of the \fBhttps\fP parameter is \fRoff\fP, and HTTPS port 443 if it is \Ron\fP. +A virtual-hosted-style S3 hostname starts with a bucket name, while a +path-style S3 hostname does not. Default is \fBlocalhost\fR. .TP .BI (http)http_user \fR=\fPstr Username for HTTP authentication.