Skip to content

Config File

Jump to bottom
James Fantin-Hardesty edited this page Oct 17, 2023 · 10 revisions

The Cloudfuse configuration file is the main way to configure options for Cloudfuse.

The Pipeline and Components

Each configuration file must contain a pipeline section that defines the components that will be used in the pipelines.

The available components are as follows in the following order of priority:

  • libfuse
  • stream
  • block_cache
  • file_cache
  • attr_cache
  • azstorage
  • s3storage

You should following the following guide for the component section.

  • Every pipeline should contain libfuse, which is interfaces with the FUSE filesystem.

  • Each pipeline should also have either stream, block_cache, or file_cache. Each of these components cannot coexist with the others so only one can be used.

    • The file_cache component should be used when you are doing write heavy workloads, or if you are reading many small files that can be stored on your local disk. file_cache will cache reads and writes to improve future access times on the same files. See here to see how to configure the file_cache component.

    • The stream component should be used when you are doing read heavy workloads that involve large files. Stream will only read part of the file requested so it improves read performance, especially on large files. See here to see how to configure the stream component.

    • The block_cache component should be used when doing read-only workloads and is similar to stream, but caches its reads so future reads of the same part of a file are faster.

  • Each pipeline should contains attr_cache which caches file and directory attributes to improve performance of the filesystem.

  • Each pipeline should contain either azstorage for Azure storage or s3storage for S3 storage.

For example, use file caching with S3 storage you would have the following in your configuration file.

components:
  - libfuse
  - file_cache
  - attr_cache
  - s3storage

Or to use streaming with Azure storage you would have the following in your configuration file.

components:
  - libfuse
  - stream
  - attr_cache
  - azstorage

Other Options

Read the logging documentation for information about setting up logging in the configuration file.

Read the health monitor documentation for information about setting up the health monitor in the configuration file.

Please see the base configuration file for a list of all settings and a brief explanation of each setting.

Sample Configuration Files

The following are some sample configuration files to help get you started.

Clone this wiki locally