Thanos Receiver: Error logs "try lock file: open <>.tmp-for-creation/index_tmp_po: no such file or directory"

[SM-9870]

Hi All,

I am facing some errors in the Thanos receiver when I am using S3 Storage for mounting PV/PVC for Thanos. But I do not see this error If I moved to NFS or the local node storage. In my setup I am using same S3 Storage across all the Pods like Compactor, Receiver, Storagegateway etc. but each service had their own Folder inside the S3 mounted storage.

Along with this error, Thanos is not able to upload the metrics to the S3 storage for long term support.

Please help!

ts=2024-09-12T01:43:13.164952074Z caller=db.go:1014 level=error component=receive component=multi-tsdb tenant=default-tenant msg="compaction failed" err="try lock file: open /data/receiver-0/default-tenant/01J7HYBVS79NPA7JJ6292NXCEJ.tmp-for-creation/index_tmp_po: no such file or directory"
ts=2024-09-12T01:44:23.166065172Z caller=db.go:1014 level=error component=receive component=multi-tsdb tenant=default-tenant msg="compaction failed" err="try lock file: open /data/receiver-0/default-tenant/01J7HYE0KDYBKFX92ZACZ2FZ01.tmp-for-creation/index_tmp_po: no such file or directory"

Similar error also logged in the Storage gateway:

  ts=2024-09-12T01:37:49.186686337Z caller=binary_reader.go:569 level=debug msg="failed to read index-header from disk; recreating" path=/data/storegw/01J7HXTB9PM6MJPYHCJ4AC3DY1/index-header err="try lock file: open /data/storegw/01J7HXTB9PM6MJPYHCJ4AC3DY1/index-header: no such file or directory"

[dosubot[bot]]

Hello @sahilmahajan88! I'm here to help you with any bugs, questions, or contributions you have. Let's work together to solve this issue.

The error you're encountering is likely due to the eventual consistency model of S3. S3's eventual consistency can lead to situations where files are not immediately available after being created, causing the "no such file or directory" error. In contrast, NFS and local node storage typically provide stronger consistency guarantees, which is why you do not encounter the same issue with them ^[1].

To mitigate these issues, you can configure specific options in Thanos:

1. Sync Delay: Use the syncDelay option to ensure blocks are considered ready to be read only after a certain period (default is 15 minutes) from their creation time. This helps account for the eventual consistency of S3.

2. Delete Delay: Use the deleteDelay option to ensure blocks marked for deletion are not immediately removed, allowing time for all readers to sync and be aware of the deletion.

3. Block Discovery Strategy: Use the --block-discovery-strategy flag to control the listing strategy for blocks. This can help manage the trade-offs between speed and cost, especially in environments with eventual consistency issues ^[2].

Additionally, ensure your S3 configuration is optimized for Thanos:

type: S3
config:
  bucket: "<bucket>"
  endpoint: "<endpoint>"
  region: "<region>"
  access_key: "<access_key>"
  secret_key: "<secret_key>"
  aws_sdk_auth: false
  http_config:
    idle_conn_timeout: 1m30s
    response_header_timeout: 2m
    tls_handshake_timeout: 10s
    expect_continue_timeout: 1s
    max_idle_conns: 100
    max_idle_conns_per_host: 100
  signature_version2: false

By following these practices and configurations, you can better handle eventual consistency issues in S3 when using Thanos ^[3].

To continue talking to Dosu, mention @dosu.

[SM-9870]

This error is coming in Thanos Receiver when I attach S3 mounted folder for TSDB directory. If I use local node storage, its working fine.