Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement statfs with synthetic values #1118

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

c-hagem
Copy link
Contributor

@c-hagem c-hagem commented Nov 7, 2024

Description of change

This PR adds support for calling statfs on virtual file system created using mountpoint.
Some applications depend on the filesystem reporting non-zero available space; currently mountpoint reports 0 as number of available blocks, which can cause these applications to not work as expected.

This PR (building on #871) implements statfs with synthetic values (4611686018427387904 free blocks).
For example, the DF output now is:

mountpoint-s3  4611686018427387904        0 4611686018427387904   0% /local/home/chagem/mnt/bucket

Thus, checks for available space should no longer fail.

Relevant issues: #710.
This change impacts existing behaviour, as Mountpoint will report non-zero value for total blocks, free blocks, free inodes and maximum file name length.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

Signed-off-by: Christian Hagemeier <chagem@amazon.com>
Signed-off-by: Christian Hagemeier <chagem@amazon.com>
@c-hagem c-hagem temporarily deployed to PR integration tests November 13, 2024 13:17 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 13, 2024 13:17 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 13, 2024 13:18 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 13, 2024 13:18 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 13, 2024 13:18 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 13, 2024 13:18 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 13, 2024 13:18 — with GitHub Actions Inactive
Copy link
Contributor

@dannycjones dannycjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Awesome to see this working.

There are a few comments inline.

Can you add some tests for this, just to verify these values make it out of the file system? I'd recommend creating a new file in mountpoint-s3/tests/fuse_tests/ named statfs_test.rs, and write some tests that test end-to-end. statvfs isn't actually in the Rust standard library, but we have some example of using a crate named nix for querying this information.

CacheLimit::AvailableSpace { min_ratio } => {
let stats = match nix::sys::statvfs::statvfs(&self.cache_directory) {
Ok(stats) if stats.blocks() == 0 => {
warn!("unable to determine available space (0 blocks reported)");
return false;
}
Ok(stats) => stats,
Err(error) => {
warn!(?error, "unable to determine available space");
return false;
}
};
(stats.blocks_free() as f64) < min_ratio * (stats.blocks() as f64)
}

mountpoint-s3/src/fs.rs Outdated Show resolved Hide resolved
Comment on lines 830 to 831
const FREE_BLOCKS: u64 = u64::MAX / 2;
const FREE_INODES: u64 = u64::MAX / 2;
Copy link
Contributor

@dannycjones dannycjones Nov 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure where any boundaries may lie, but I think we can take this down by 1024 at least. (In this case, I think we are fine to adjust these values in the future.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed it, assuming taking down means / 1024 instead of / 2.

mountpoint-s3/CHANGELOG.md Outdated Show resolved Hide resolved
mountpoint-s3/src/fs.rs Outdated Show resolved Hide resolved
Signed-off-by: Christian Hagemeier <chagem@amazon.com>
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:30 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:30 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:30 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:30 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:30 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:30 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:30 — with GitHub Actions Inactive
Signed-off-by: Christian Hagemeier <chagem@amazon.com>
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:58 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:58 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:58 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:58 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:58 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:58 — with GitHub Actions Inactive
@c-hagem c-hagem temporarily deployed to PR integration tests November 18, 2024 10:58 — with GitHub Actions Inactive
Comment on lines +110 to +129
#[derive(Debug)]
/// Reply to a 'statfs' call
pub struct StatFs {
/// Total number of blocks
pub total_blocks: u64,
/// Number of free blocks
pub free_blocks: u64,
/// Number of free blocks available to unprivileged user
pub available_blocks: u64,
/// Number of inodes in file system
pub total_inodes: u64,
/// Available inodes
pub free_inodes: u64,
/// Optimal transfer block size
pub block_size: u32,
/// Maximum name length
pub maximum_name_length: u32,
/// Fragement size
pub fragment_size: u32,
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: didn't catch this last time, but we always put Rustdoc comments above macros (so its clear what macros are applied to the struct)

Suggested change
#[derive(Debug)]
/// Reply to a 'statfs' call
pub struct StatFs {
/// Total number of blocks
pub total_blocks: u64,
/// Number of free blocks
pub free_blocks: u64,
/// Number of free blocks available to unprivileged user
pub available_blocks: u64,
/// Number of inodes in file system
pub total_inodes: u64,
/// Available inodes
pub free_inodes: u64,
/// Optimal transfer block size
pub block_size: u32,
/// Maximum name length
pub maximum_name_length: u32,
/// Fragement size
pub fragment_size: u32,
}
/// Reply to a 'statfs' call
#[derive(Debug)]
pub struct StatFs {
/// Total number of blocks
pub total_blocks: u64,
/// Number of free blocks
pub free_blocks: u64,
/// Number of free blocks available to unprivileged user
pub available_blocks: u64,
/// Number of inodes in file system
pub total_inodes: u64,
/// Available inodes
pub free_inodes: u64,
/// Optimal transfer block size
pub block_size: u32,
/// Maximum name length
pub maximum_name_length: u32,
/// Fragement size
pub fragment_size: u32,
}

non-blocking

assert_ne!(stats.blocks(), 0);
}

/// Tests that default values from FUSER are reported for mpst fields
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is mpst, is it typo for most?

Comment on lines +20 to +25
// These five aren't default values but set by us, so maybe drop
assert_eq!(stats.blocks(), u64::MAX / 1024);
assert_eq!(stats.blocks_free(), u64::MAX / 1024);
assert_eq!(stats.blocks_available(), u64::MAX / 1024);
assert_eq!(stats.files(), u64::MAX / 1024);
assert_eq!(stats.files_available(), u64::MAX / 1024);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd drop these, but actually maybe we just combine with the test above. No need to separate them.

let test_session = creator_fn(prefix, Default::default());
let mount_dir = test_session.mount_path();
let stats = nix::sys::statvfs::statvfs(mount_dir.into()).unwrap();
//assert_eq!(stats.name_max(), 255);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drop commented out code

mountpoint-s3/tests/fuse_tests/statfs_test.rs Show resolved Hide resolved
Comment on lines +64 to +69
#[cfg(feature = "s3_tests")]
#[test_case(""; "no prefix")]
#[test_case("statfs_report_nonzero_test"; "prefix")]
fn statfs_report_fuser_defaults_s3(prefix: &str) {
statfs_test_available_nonzero(fuse::s3_session::new, prefix);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test case is wrong

Suggested change
#[cfg(feature = "s3_tests")]
#[test_case(""; "no prefix")]
#[test_case("statfs_report_nonzero_test"; "prefix")]
fn statfs_report_fuser_defaults_s3(prefix: &str) {
statfs_test_available_nonzero(fuse::s3_session::new, prefix);
}
#[cfg(feature = "s3_tests")]
#[test_case(""; "no prefix")]
#[test_case("statfs_report_fuser_defaults"; "prefix")]
fn statfs_report_fuser_defaults_s3(prefix: &str) {
statfs_report_fuser_defaults(fuse::s3_session::new, prefix);
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants