Improve the handling of sh.keptn.event.get-sli.triggered events where SLIs don't produce a value
#704
Replies: 2 comments 1 reply
-
|
@arthurpitman thanks for the summary and simplifying it further! |
Beta Was this translation helpful? Give feedback.
-
|
@johannes-b it would be good to be able to determine programmatically which indicator generated the warning. For this we would need to extend or change the |
Beta Was this translation helpful? Give feedback.
-
|
As a starter, it would be nice to see at least which metrics failed. At the moment there is zero output. If I have an SLO with What would help tremendously is at least showing (somehow that A customer I'm working with has exactly this issue which has lead to me having to create a utility which reads an |
Beta Was this translation helpful? Give feedback.
-
Current situation
SLIs that produce no data during a given evaluation result in
sh.keptn.event.get-sli.finishedevent withresultset tofail. This in turn causes the evaluation to fail, as the lighthouse-service does not process the event further.Desired behavior
In cases where the dynatrace-service is able to query the Dyntrace tenant successfully, but the processing of the result leads to no single numerical value (i.e. when no data is available during the timeframe or no single value can be selected from the return results),
successof the corresponding element in theindicatorValuesis set tofalsebut the overallresultis set towarning.This would mean the dynatrace-service would only function as an SLI provider, providing indicators with as much information as possible, making its behavior more obvious and easy to document. It would no longer perform any sort of evaluation, this would be done entirely by the lighthouse-service, depending on the scoring of the SLO configuration.
In short, if the user has set up the evaluation to tolerate a certain number of failing objectives, then it could still pass even when some SLIs don't produce a value. Other classes of errors would still be handled as before: situations where it was unable to process the request would lead to
sh.keptn.event.get-sli.finishedevent withstatusoferrored, while situations where it is misconfigured (bad SLI syntax or errors from the Dynatrace tenant) would still result in aresultset tofail.The overall behavior can be summarized as follows:
statusresultIf the dynatrace-service was able to retrieve the metric from the Dynatrace API, i.e. the chain of calls succeeded (~HTTP 200-class) and process the result to produce a single value
succeededpasssuccessfield of each indicator inindicatorValuesis set totrue- an SLI retrieval was performed
-> good case (lighthouse-service can continue with evaluation task)
As above, but if the processing of the result cannot produce a single value
- due to no data points being returned,
- multiple data points being returned,
- or not being able to select a single result value (USQL queries).
succeededwarningsuccessfield of the individual indicator inindicatorValuesis set tofalse- an SLI retrieval was performed
-> lighthouse-service has to deal with SLIs that have no value
If an HTTP 400- class error occurs, due to e.g.:
- syntax errors in SLI files,
- dashboard syntax errors,
- or failure to retrieve the metric from the Dynatrace API
succeededfailsuccessfield of each affected indicator inindicatorValuesis set tofalse- a partial SLI retrieval may have been performed but it should be disregarded
->
get-sli.finishedterminates the sequence executionIf the dynatrace-service was unable to process the request (encounters HTTP 500-class errors)
erroredfailsuccessfield of all indicators inindicatorValuesis set tofalse- most likely no SLI retrieval was performed
->
get-sli.finishedterminates the sequence executionBeta Was this translation helpful? Give feedback.
All reactions