fix(alerts): add instance label to KubeAggregatedAPIErrors

[sebastiangaiser]

Supersede of #774

@skl please let me know what you think about this ;)

[sebastiangaiser]

Another idea would be to add a for: 10m.

[skl]

Thanks for contributing and drawing my attention to this!

I ended up doing a bit of a deep dive to try and understand the metric properly, left you a suggestion on improving the PromQL query in the review.

Please could you add the 'for': '10m' to the alert?

And update the description to something like:

description: 'Kubernetes aggregated API {{ $labels.instance }}/{{ $labels.name }} has reported {{ $labels.reason }} errors%s.' % [
  (utils.ifShowMultiCluster($._config, ' on cluster {{ $labels.%(clusterLabel)s }}' % $._config),
],

I'm hoping all of that would resolve the underlying issue, wdyt?

[sebastiangaiser]

Sounds good. Thank you @skl for investing time here, I added all your suggestions.

[skl]

Fixed a syntax error for you in 21c0b87

[skl]

Lint failure inherited from main, this can be ignored as I'm dealing with that separately

prometheus_alerts.yaml:342-346 Warning: Aggregation using `without()` can be fragile when used inside binary expression because both sides must have identical sets of labels to produce any results, adding or removing labels to metrics used here can easily break the query, consider aggregating using `by()` to ensure consistent labels. (promql/fragile)