[ffresty] [metric] HTTP Response Time and Complete Gauge Support

[onelapahead]

For Resty, we enhance the existing before/after hooks so that we record the elapsed time on responses. And use more labels and smarter context's for consistently deriving any request metadata like the host.

Then, within the metric manager itself, we didn't offer the ability to use gauge inc/dec which can be very useful rather than using set.

[Chengxuan]

@onelapahead what's your consideration between Histogram and Summary for this timing metrics?

[onelapahead]

Histogram is more costly in terms of cardinality bc of all the buckets you make. So this felt light enough to only calculate average response time across a few basic dimensions.

Would love to toggle btwn histogram and summaries based on certain settings eventually, but that was a bigger change than I had an appetite for.

[Chengxuan]

Some size metrics for data transmitted would be useful to add as well.

[Chengxuan]

That is a great step forward, thanks @onelapahead .

I asked a question about the metrics type for the response time. Switching to histogram will have the following benefits, which could be compelling:

1. reduce observation costs on the client
2. enable aggregation across instances

^^ from https://prometheus.io/docs/practices/histograms/

[onelapahead]

enable aggregation across instances

To clarify - this is still entirely possible with summaries. They are similar to a histogram but less counters:

# summary example
http_response_time_sum{}
http_response_time_count{}

# histogram example
http_response_time_buckets{le="1.0"}
http_response_time_buckets{le="0.5"}
http_response_time_buckets{le="0.1"}
http_response_time_sum{}
http_response_time_count{}

Histograms are great/preferred for aggregate percentiles and therefore performance engineering. Summaries are great for aggregated averages and therefore reliability engineering.

Quantiles are a special version of summary, which to your point are basically worthless at any real scale bc they cannot be aggregated. However, they are less expensive cardinality than histograms. So if you care about individual "thing's" performance they are useful.

Do hope ff-common can offer histograms as an option, but not by default since they are quite expensive. But will save that for a future contributor.

[EnriqueL8]

This looks good @onelapahead

Getting Error: pkg/ffresty/ffresty.go:201:1: cyclomatic complexity 37 of func NewWithConfig is high (> 30) (gocyclo) from build, given this function is doing a lot of just configuration we can probably ignore this complexity warning by adding the ignore comment. Or if you are up for it we can break it down.

We could split adding the onBeforeRequest to another function but it might be messy