Thanos store/compact error: Truncated response should have continuation token set

[LukaszWasko]

I got error today on working instances (for about 3 months with no problems) Thanos store/compact on NetApp S3 :

Mar 03 16:04:21 thanos thanos[67666]: {"caller":"http.go:74","component":"store","err":"bucket store initial sync: sync block: BaseFetcher: iter bucket: Truncated response should have continuation token set","level":"info","msg":"internal server is shutting down","service":"http/server","ts":"2022-03-03T15:04:21.156903187Z"}"

I cannot launch them.
Thanos Receiver works fine with pushing metrics to S3.

I googled around and I found a possible cause to be returning truncated amounts of results because of some kind of page limit on S3 API.

NetApp shows me that it has about 3800 objects:

NetApp::> bucket show metrics-prod -fields object-count
  (vserver object-store-server bucket show)
vserver                      bucket           object-count
---------------------------- ---------------- ------------
metrics.prod metrics-prod 3872

Adding to s3.yaml below line doesn't help:

list_objects_version: "v1"

After couple of minutes thanos throw an error context deadline exceeded

I did not find a clear answer, should I change any parameter on the Thanos Store / Minio client (s3.yaml) side or on the NetApp side?
Has anyone had experience with this?

Thanos version: thanos-0.24.0-1.el8.x86_64
System: CentOS 8

thanos store parameters:

--objstore.config-file /etc/thanos-store/s3.yaml --http-address=0.0.0.0:10904 --grpc-address=0.0.0.0:10901 --data-dir="/var/lib/thanos/store" --log.level=debug --log.format=json'

/etc/thanos-store/s3.yaml:

type: S3
config:
  bucket: "metrics-prod"
  endpoint: "metrics.prod"
  region: "us-east-1a"
  access_key: XXX
  insecure: false
  signature_version2: false
  secret_key: XXX
  http_config:
    idle_conn_timeout: 0s
  sse_config:
    type: "SSE-S3"

[LukaszWasko]

Any ideas? Problem still exists and I cannot run thanos store.

[wiardvanrij]

No, but perhaps this might get more attention in an issue. Feel free to make that as well and provide as much information as possible. Thanks!

[LukaszWasko]

I have the same issue on my another S3 storage (from two days so far).

netapp::> bucket show -vserver prometrics.prod -bucket prometrics -fields object-count
  (vserver object-store-server bucket show)
vserver                         bucket               object-count
------------------------------- -------------------- ------------
prometrics.prod prometrics 8462

thanos compact and thanos storage do not work.
I used S3 Browser to connect to s3 backend and delete couple of recent objects to reach below some_kind_of_limit and it worked.
Thanos Compactor started and created some new objects and again the mistical object's limit has been reached.
Looks like Compactor has been bored for the last 3 weeks:

and my thanos compact config:

--objstore.config-file /etc/thanos/s3.yaml \
  --data-dir=/var/lib/thanos/compact \
  --http-address=0.0.0.0:10904 \
  --deduplication.replica-label="replica" \
  --deduplication.replica-label="receive_replica" \
  --deduplication.func=penalty \
  --compact.enable-vertical-compaction \
  --delete-delay=2h \
  --wait \
  --wait-interval 5m \
  --log.format=json'