Unable to get metrics from S3

[pablokbs]

Hello, I'm currently having issues retrieving metrics from my S3 storage.

I'm able to see the blocks in the Thanos-Store interface

I'm testing it by only using the storegateway as a store:

Query:

      containers:
      - args:
        - query
        - --log.level=debug
        - --log.format=logfmt
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --query.replica-label=prometheus_replica
        - --store=dnssrv+_grpc._tcp.thanos-storegateway.monitoring.svc.cluster.local
        image: docker.io/bitnami/thanos:0.24.0-scratch-r0

StoreGateway

      containers:
      - args:
        - store
        - --log.level=debug
        - --log.format=logfmt
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --data-dir=/data
        - --objstore.config-file=/conf/objstore.yml
        image: docker.io/bitnami/thanos:0.24.0-scratch-r0

Query:

Am I missing something?

thanos-query logs:

thanos-query-5f7b7d869c-424tt query level=debug ts=2022-08-29T17:41:44.413871452Z caller=proxy.go:332 component=proxy request="min_time:1661102810000 max_time:1661124881000 matchers:<name:\"label_app\" value:\"go-test-app\" > matchers:<name:\"__name__\" value:\"kube_pod_labels\" > aggregates:COUNT aggregates:SUM partial_response_disabled:true step:2419000 " msg="Series: started fanout streams" status="Store Addr: 172.20.39.125:10901 LabelSets: {replica=\"thanos-ruler-0\", ruler_cluster=\"\"},{replica=\"thanos-ruler-1\", ruler_cluster=\"\"} Mint: 1653523217096 Maxt: 1661788800000 queried"
thanos-query-5f7b7d869c-424tt query level=debug ts=2022-08-29T17:41:44.413939435Z caller=proxy.go:332 component=proxy request="min_time:1661127000000 max_time:1661211965000 matchers:<name:\"label_app\" value:\"go-test-app\" > matchers:<name:\"__name__\" value:\"kube_pod_labels\" > aggregates:COUNT aggregates:SUM partial_response_disabled:true step:2419000 " msg="Series: started fanout streams" status="Store Addr: 172.20.39.125:10901 LabelSets: {replica=\"thanos-ruler-0\", ruler_cluster=\"\"},{replica=\"thanos-ruler-1\", ruler_cluster=\"\"} Mint: 1653523217096 Maxt: 1661788800000 queried"
thanos-query-5f7b7d869c-9mtfs query level=debug ts=2022-08-29T17:41:44.417177971Z caller=proxy.go:332 component=proxy request="min_time:1661385833000 max_time:1661470798000 matchers:<name:\"label_app\" value:\"go-test-app\" > matchers:<name:\"__name__\" value:\"kube_pod_labels\" > aggregates:COUNT aggregates:SUM partial_response_disabled:true step:2419000 " msg="Series: started fanout streams" status="Store Addr: 172.20.39.125:10901 LabelSets: {replica=\"thanos-ruler-0\", ruler_cluster=\"\"},{replica=\"thanos-ruler-1\", ruler_cluster=\"\"} Mint: 1653523217096 Maxt: 1661788800000 queried"
thanos-query-5f7b7d869c-9mtfs query level=debug ts=2022-08-29T17:41:44.417177981Z caller=proxy.go:332 component=proxy request="min_time:1661472917000 max_time:1661557882000 matchers:<name:\"label_app\" value:\"go-test-app\" > matchers:<name:\"__name__\" value:\"kube_pod_labels\" > aggregates:COUNT aggregates:SUM partial_response_disabled:true step:2419000 " msg="Series: started fanout streams" status="Store Addr: 172.20.39.125:10901 LabelSets: {replica=\"thanos-ruler-0\", ruler_cluster=\"\"},{replica=\"thanos-ruler-1\", ruler_cluster=\"\"} Mint: 1653523217096 Maxt: 1661788800000 queried"
thanos-query-5f7b7d869c-424tt query level=debug ts=2022-08-29T17:41:44.416086575Z caller=proxy.go:332 component=proxy request="min_time:1661644666000 max_time:1661707860000 matchers:<name:\"label_app\" value:\"go-test-app\" > matchers:<name:\"__name__\" value:\"kube_pod_labels\" > aggregates:COUNT aggregates:SUM partial_response_disabled:true step:2419000 " msg="Series: started fanout streams" status="Store Addr: 172.20.39.125:10901 LabelSets: {replica=\"thanos-ruler-0\", ruler_cluster=\"\"},{replica=\"thanos-ruler-1\", ruler_cluster=\"\"} Mint: 1653523217096 Maxt: 1661788800000 queried"
thanos-query-5f7b7d869c-424tt query level=debug ts=2022-08-29T17:41:44.416167438Z caller=proxy.go:332 component=proxy request="min_time:1661214084000 max_time:1661299049000 matchers:<name:\"label_app\" value:\"go-test-app\" > matchers:<name:\"__name__\" value:\"kube_pod_labels\" > aggregates:COUNT aggregates:SUM partial_response_disabled:true step:2419000 " msg="Series: started fanout streams" status="Store Addr: 172.20.39.125:10901 LabelSets: {replica=\"thanos-ruler-0\", ruler_cluster=\"\"},{replica=\"thanos-ruler-1\", ruler_cluster=\"\"} Mint: 1653523217096 Maxt: 1661788800000 queried"
thanos-query-5f7b7d869c-424tt query level=debug ts=2022-08-29T17:41:44.416547643Z caller=proxy.go:332 component=proxy request="min_time:1661560001000 max_time:1661642547000 matchers:<name:\"label_app\" value:\"go-test-app\" > matchers:<name:\"__name__\" value:\"kube_pod_labels\" > aggregates:COUNT aggregates:SUM partial_response_disabled:true step:2419000 " msg="Series: started fanout streams" status="Store Addr: 172.20.39.125:10901 LabelSets: {replica=\"thanos-ruler-0\", ruler_cluster=\"\"},{replica=\"thanos-ruler-1\", ruler_cluster=\"\"} Mint: 1653523217096 Maxt: 1661788800000 queried"
thanos-query-5f7b7d869c-424tt query level=debug ts=2022-08-29T17:41:44.416580954Z caller=proxy.go:332 component=proxy request="min_time:1661301168000 max_time:1661383714000 matchers:<name:\"label_app\" value:\"go-test-app\" > matchers:<name:\"__name__\" value:\"kube_pod_labels\" > aggregates:COUNT aggregates:SUM partial_response_disabled:true step:2419000 " msg="Series: started fanout streams" status="Store Addr: 172.20.39.125:10901 LabelSets: {replica=\"thanos-ruler-0\", ruler_cluster=\"\"},{replica=\"thanos-ruler-1\", ruler_cluster=\"\"} Mint: 1653523217096 Maxt: 1661788800000 queried"
thanos-query-5f7b7d869c-424tt query level=debug ts=2022-08-29T17:41:44.613949524Z caller=endpointset.go:309 component=endpointset msg="starting to update API endpoints" cachedEndpoints=1
thanos-query-5f7b7d869c-424tt query level=debug ts=2022-08-29T17:41:44.614689262Z caller=endpointset.go:312 component=endpointset msg="checked requested endpoints" activeEndpoints=1 cachedEndpoints=1
thanos-query-5f7b7d869c-9mtfs query level=debug ts=2022-08-29T17:41:45.593861674Z caller=endpointset.go:309 component=endpointset msg="starting to update API endpoints" cachedEndpoints=1
thanos-query-5f7b7d869c-9mtfs query level=debug ts=2022-08-29T17:41:45.595151825Z caller=endpointset.go:312 component=endpointset msg="checked requested endpoints" activeEndpoints=1 cachedEndpoints=1

thanos-store-gateway logs:

thanos-storegateway-1 storegateway level=debug ts=2022-08-29T17:41:44.414538677Z caller=bucket.go:1114 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:256340 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:1890}" err=null
thanos-storegateway-1 storegateway level=debug ts=2022-08-29T17:41:44.414664222Z caller=bucket.go:1114 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionEr
rors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:248239 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:1150}" err=null
thanos-storegateway-1 storegateway level=debug ts=2022-08-29T17:41:44.417409145Z caller=bucket.go:1114 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:451917 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:2200}" err=null
thanos-storegateway-1 storegateway level=debug ts=2022-08-29T17:41:44.417664724Z caller=bucket.go:1114 msg="stats query processed" stats="&{blocksQueried:8 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:568392 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:2450}" err=null
thanos-storegateway-1 storegateway level=debug ts=2022-08-29T17:41:44.417753808Z caller=bucket.go:1114 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:503309 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:1530}" err=null
thanos-storegateway-1 storegateway level=debug ts=2022-08-29T17:41:44.417833781Z caller=bucket.go:1114 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:476748 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:1010}" err=null
thanos-storegateway-1 storegateway level=debug ts=2022-08-29T17:41:44.418363431Z caller=bucket.go:1114 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionEr
rors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:191437 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:1190}" err=null
thanos-storegateway-1 storegateway level=debug ts=2022-08-29T17:41:44.418464744Z caller=bucket.go:1114 msg="stats query processed" stats="&{blocksQueried:2 postingsTouched:0 postingsTouchedSizeSum:0 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:0 seriesTouchedSizeSum:0 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:0 chunksTouchedSizeSum:0 chunksFetched:0 chunksFetchedSizeSum:0 chunksFetchCount:0 chunksFetchDurationSum:0 getAllDuration:149225 mergedSeriesCount:0 mergedChunksCount:0 mergeDuration:1050}" err=null