Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve http_host, filename in docs and example/http-s3.fio #1798

Merged
merged 1 commit into from
Aug 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions HOWTO.rst
Original file line number Diff line number Diff line change
Expand Up @@ -805,6 +805,11 @@ Target file/device
Note: Windows and FreeBSD (refer to geom(4)) prevent write access to areas
of the disk containing in-use data (e.g. filesystems).

For HTTP and S3 access, specify a valid URL path or S3 key, respectively.
A filename for path-style S3 includes a bucket name (:file:`/bucket/k/e.y`)
while a virtual-hosted-style S3 filename :file:`/k/e.y` does not because
its bucket name is specified in :option:`http_host`.

The filename "`-`" is a reserved name, meaning *stdin* or *stdout*. Which
of the two depends on the read/write direction set.

Expand Down Expand Up @@ -2934,8 +2939,13 @@ with the caveat that when used on the command line, they must come after the

.. option:: http_host=str : [http]

Hostname to connect to. For S3, this could be the bucket hostname.
Default is **localhost**
Hostname to connect to. HTTP port 80 is used automatically when the value of
the https parameter is *off*, and HTTPS port 443 if it is *on*. A
virtual-hosted-style S3 hostname starts with a bucket name, while a
path-style S3 hostname does not. See
https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html for
detailed examples.
Default is **localhost** (path-style S3 hostname)

.. option:: http_user=str : [http]

Expand Down
16 changes: 13 additions & 3 deletions examples/http-s3.fio
Original file line number Diff line number Diff line change
@@ -1,19 +1,29 @@
# Example test for the HTTP engine's S3 support against Amazon AWS.
# Obviously, you have to adjust the S3 credentials; for this example,
# they're passed in via the environment.
#
# For non-AWS S3 implementations, refer to your S3 vendor's region
# settings. Note that the region value appears twice, in http_host and
# http_s3_region.
# This example uses virtual-hosted-style requests:
# https://bucket-name.s3.region-code.amazonaws.com/k/e.y
# For path-style, prefix the key with the bucket name in the filename
# so that filename=/bucket-name/k/e.y:
# https://s3.region-code.amazonaws.com/bucket-name/k/e.y

# Reference for Virtual-hosted-style vs. Path-style URLs:
# https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html

[global]
ioengine=http
name=test
direct=1
filename=/larsmb-fio-test/object
filename=/k/e.y
http_verbose=0
https=on
http_mode=s3
http_s3_key=${S3_KEY}
http_s3_keyid=${S3_ID}
http_host=s3.eu-central-1.amazonaws.com
http_host=bucket-name.s3.eu-central-1.amazonaws.com
http_s3_region=eu-central-1
group_reporting

Expand Down
11 changes: 9 additions & 2 deletions fio.1
Original file line number Diff line number Diff line change
Expand Up @@ -585,6 +585,11 @@ the first device, `\\\\.\\PhysicalDrive1' for the second etc.
Note: Windows and FreeBSD prevent write access to areas
of the disk containing in-use data (e.g. filesystems).
.P
For HTTP and S3 access, specify a valid URL path or S3 key, respectively.
A filename for path-style S3 includes a bucket name (`/bucket/k/e.y')
while a virtual-hosted-style S3 filename (`/k/e.y') does not because its
bucket name is specified in \fBhttp_host\fR.
.P
The filename `\-' is a reserved name, meaning *stdin* or *stdout*. Which
of the two depends on the read/write direction set.
.RE
Expand Down Expand Up @@ -2554,8 +2559,10 @@ Touching all objects affects ceph caches and likely impacts test results.
Enabled by default.
.TP
.BI (http)http_host \fR=\fPstr
Hostname to connect to. For S3, this could be the bucket name. Default
is \fBlocalhost\fR
Hostname to connect to. HTTP port 80 is used automatically when the value
of the \fBhttps\fP parameter is \fRoff\fP, and HTTPS port 443 if it is \Ron\fP.
A virtual-hosted-style S3 hostname starts with a bucket name, while a
path-style S3 hostname does not. Default is \fBlocalhost\fR.
.TP
.BI (http)http_user \fR=\fPstr
Username for HTTP authentication.
Expand Down