Skip to content

Commit

Permalink
[enhancement](cloud) refine block file cache evict policy (apache#42451)
Browse files Browse the repository at this point in the history
- Enhance cache performance and reduce cache misses by:
    - Prevent starvation of any particular cache.
    - Improve disk space utilization.
    - Maintain strict and systematic eviction priorities.
- Enhance the observability of caching strategies, allowing cache
behavior to be intuitively self-explanatory through monitoring.
- Improve code comprehensibility: unify the cache framework to avoid
ad-hoc handling of TTL, increasing the modularity of the code.
- Reduce operational complexity by eliminating and standardizing
configuration items.

---------

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
  • Loading branch information
freemandealer authored Nov 4, 2024
1 parent 90ae514 commit dacba03
Show file tree
Hide file tree
Showing 6 changed files with 1,784 additions and 432 deletions.
4 changes: 2 additions & 2 deletions be/src/common/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1011,7 +1011,7 @@ DEFINE_Bool(enable_file_cache_query_limit, "false");
DEFINE_mInt32(file_cache_enter_disk_resource_limit_mode_percent, "90");
DEFINE_mInt32(file_cache_exit_disk_resource_limit_mode_percent, "80");
DEFINE_mBool(enable_read_cache_file_directly, "false");
DEFINE_mBool(file_cache_enable_evict_from_other_queue_by_size, "false");
DEFINE_mBool(file_cache_enable_evict_from_other_queue_by_size, "true");
DEFINE_mInt64(file_cache_ttl_valid_check_interval_second, "0"); // zero for not checking
// If true, evict the ttl cache using LRU when full.
// Otherwise, only expiration can evict ttl and new data won't add to cache when full.
Expand Down Expand Up @@ -1292,7 +1292,7 @@ DEFINE_Int64(num_s3_file_upload_thread_pool_min_thread, "16");
// The max thread num for S3FileUploadThreadPool
DEFINE_Int64(num_s3_file_upload_thread_pool_max_thread, "64");
// The max ratio for ttl cache's size
DEFINE_mInt64(max_ttl_cache_ratio, "90");
DEFINE_mInt64(max_ttl_cache_ratio, "50");
// The maximum jvm heap usage ratio for hdfs write workload
DEFINE_mDouble(max_hdfs_wirter_jni_heap_usage_ratio, "0.5");
// The sleep milliseconds duration when hdfs write exceeds the maximum usage
Expand Down
Loading

0 comments on commit dacba03

Please sign in to comment.