Sidecar is really slow compared to prometheus HTTP API usage

[rishabhkumar92]

Hi Team,

I am trying to setup sidecar + querier to fetch data from multiple clusters (Each cluster uses Grafana Mimir). Listing down details of setup:

• Thanos Sidecar is deployed with Mimir's Query frontend (1 per replica)
• Thanos Queriers are deployed as separate app to query data from sidecars from multiple clusters

We are observing very slow performance (15-20s) for queries due to remote read API usage while prometheus HTTP query based federated solution like promxy executes under 3 seconds. Is there something wrong with some my setup/default configuration?

Resource usage for sidecar/querier is not a bottleneck here, utilization stays below 20-30% on memory, CPU is spiky (goes beyond 80%) but below limit .

Sidecar: cpu: 4, memory: 4GB
Querier: cpu: 4, memory: 4GB

[dosubot[bot]]

I found some similar issues and discussions regarding slow performance with Thanos Sidecar:

1. Open Issue: Sidecar is significantly slower than the underlying Prometheus queries. The suggested solution is to switch back to the Prometheus engine in the Thanos querier, which has been reported to improve query times significantly ^[1].

2. Open Issue: Thanos query responds too slow. It suggests setting up distributed tracing and continuous profiling tools to identify delays and reviewing configuration settings related to timeouts and intervals ^[2].

3. Discussion: How to get Thanos sidecar use mimir query frontend API rather than remote read endpoint to query metrics. This is an unsolved discussion that might be relevant to your setup ^[3].

The slow performance you're experiencing could be due to how Thanos handles query execution. Thanos needs to pull in all data from stores in memory before evaluating a query, which can increase latency. Consider optimizing your configuration settings and exploring the use of distributed query execution to improve performance ^[4].

To continue talking to Dosu, mention @dosu.