How does Thanos receiver HPA works?

[yliu138repo]

Hi team,

I currently have Thanos Receiver set up with 2 pods on K8s (Openshift acutally) cluster using Thanos helm chart from Bitnami, and there is a HPA as well with minimum of 2 pods. Before Thanos Receiver, there was a opentelmetry collector with prometheusremotewrite exporter that writes samples to Thanos receiver. The Thanos Receiver is set up with Ketama Hashing algorithm. I know when traffic comes, same metrics will be routed to the same receiver to ensure efficient writing. Only new connection could be routed to a different receiver to ensure not a single receiver is overloaded.

Load testing details: I run with k6 - 6 concurrent k6 runs - each run with 2000 virtual users over 1 hour.

Question 1
It seems that every time running load testing, all the metrics get to routed to the first instance only. When I looked into the hashrings.json , it only has the 1st receiver in the instance. So how could we dynamically update the configmap if HPA autoscales the receiver statefulset??

Question 2
If there are 2 receivers for example, if metrics are routed to first receiver, and the received somehow crashed. what will happen? Would the process stop sending, and wait until the first receiver comes back?? or it will be routed to the next avaialbe receiver, which is the 2nd receiver?

Thanks.

[kaiohenricunha]

Anyone? Interesting question

[alecrajeev]

The thanos-receive-controller can be used to automatically scale up and down a thanos-receive statefulset: https://github.com/observatorium/thanos-receive-controller

It will automatically generate the hashring configmap. The --allow-dynamic-scaling flag should be used to allow for scaling.