Refresh ServiceMonitors

[Evynglais]

Hi,

I've deployed the helm-prometheus-stack, and then deployed a ServiceMonitor, but it doesn't show up as a target in prometheus - it only shows if I reinstall the whole stack. Am I doing something wrong here? I was under the impression that the prom-operator should pick up the changes and then roll out some new config for my Prometheus pods.

For info, I've added the "release: " label that I believe we need for it to be picked up, and I'm not overriding any of the default settings in my values.yaml other than HTTPS configuration for Prometheus.

Thanks,
Evynglais

[paulfantom]

We don't provide helm chart as it lives in https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack.

I recommend the following:

1. Check prometheus-operator logs to verify if there is something abnormal happening
2. Go to prometheus UI /service-discovery and check if your service is present there. If it is then ServiceMonitor was picked up and the issue is with incorrectly selected port in endpoints field
3. If your serviceMonitor is not registered in /service-discovery then check what is the value of serviceMonitorSelector field in your Prometheus object and adjusting ServiceMonitor labels accordingly. Value of {} selects all serviceMonitors.
4. If you still don't have serviceMonitor registered, check a field serviceMonitorNamespaceSelector to verify if prometheus should look for serviceMonitor in that namespace. Value of {} selects all namespaces.
5. If after waiting 15 minutes you still don't have your serviceMonitor registered, check RBAC. Usually you shouldn't get this far as in most cases issue is in points 2-4 :)

[mikepattyn]

How can i check the prometheus-operator logs? The ServiceMonitor is beeing picked up by /service-discovery, yet all targets are dropped and im trying to figure out why... (monitoring a cockroachdb cluster)

[Evynglais]

Perfect, thanks. It was picked up in service-discovery and the port regex was wrong so no target was generated. Thanks!