automatically debug-log upstream http-response-sizes

[gabor]

work in progress.

[gabor]

@wbrowne @marefr could you please take a look at this draft? it's based on our discussion about logging "upstream" response-sizes... this is not polished yet, i'd just like to know if i'm going in the right direction.. notes:

• we could alternatively generate metrics instead of logs, but i'm not sure it's really possible to produce metrics from plugin-sdk-go.. logging seems easier
• my first idea was to create this as a middleware in the default-middleware array, but when i checked this, it seemed that if you configure the http-client with your own middlewares, then you have to manually append the default-middlewares... the current approach seemed more robust
• i'm checking for a label to be sure this is a datasource-request, plugin-sdk-go-http-clients are also used in other situations where we do not want to do this logging
• as i said, no polish yet, variable names may be inconsistent etc, will fix that later

in general, WDYT? thanks!

[marefr]

I think this make sense with some minor changes.

After reviewing the code I just realized that we already have something like this in core grafana, see https://github.com/grafana/grafana/blob/main/pkg/infra/httpclient/count_bytes_reader.go and https://github.com/grafana/grafana/blob/ddee95cb6d737ba845380c89456ede9998e12f48/pkg/infra/httpclient/httpclientprovider/datasource_metrics_middleware.go#L115-L119. So I think it would make sense to move the count_bytes_reader.go to the SDK instead replacing your ReportSizeReader

[marefr]

You could put this logic in the middleware as well, see this example to get a hold of the options

grafana-plugin-sdk-go/backend/httpclient/basic_auth_middleware.go

Line 15 in 7470982

return NamedMiddlewareFunc(BasicAuthenticationMiddlewareName, func(opts Options, next http.RoundTripper) http.RoundTripper {

. And I think then adding the middleware via DefaultMiddlewares() would allow both

grafana-plugin-sdk-go/backend/httpclient/provider.go

Line 64 in 7470982

providerOpts.Middlewares = DefaultMiddlewares()

and http_client.go to support this.

One important thing is that this middleware should probably/maybe execute after the https://github.com/grafana/grafana-plugin-sdk-go/blob/main/backend/httpclient/max_bytes_reader.go so if a limit is in place your middleware isn't continuing to read the response to the end 🤷

One thing to consider might be to somehow share the response size reader between limit and reporting middlewares, but maybe something for the future.

[gabor]

this was achieved by #1130