feat: add subscribed message metrics instrumentations

[Xunzhuo]

After: #1982
Part of: #700

refer: #2089

[codecov]

Codecov Report

Merging #2080 (3316827) into main (131178b) will decrease coverage by 0.02%.
The diff coverage is 44.11%.

@@            Coverage Diff             @@
##             main    #2080      +/-   ##
==========================================
- Coverage   64.42%   64.40%   -0.02%     
==========================================
  Files         109      109              
  Lines       15049    15094      +45     
==========================================
+ Hits         9695     9721      +26     
- Misses       4752     4770      +18     
- Partials      602      603       +1     

Files	Coverage Δ
internal/xds/translator/runner/runner.go	69.81% <100.00%> (+0.58%)	⬆️
internal/gatewayapi/runner/runner.go	20.80% <25.00%> (-0.34%)	⬇️
internal/message/watchutil.go	88.57% <84.00%> (-11.43%)	⬇️
internal/xds/server/runner/runner.go	28.18% <0.00%> (-0.53%)	⬇️
internal/provider/kubernetes/controller.go	51.92% <30.30%> (-0.40%)	⬇️

... and 1 file with indirect coverage changes

[zirain]

I think you can add a simple e2e test for prometheus

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"strings"
	"time"

	"github.com/prometheus/common/expfmt"
)

func main() {
	_, body, err := HTTPGet(fmt.Sprintf("http://127.0.0.1:%d/stats/prometheus", 9901))
	if err != nil {
		fmt.Println("get metric error: %v", err)
		return
	}
	reader := strings.NewReader(body)
	_, err = (&expfmt.TextParser{}).TextToMetricFamilies(reader)
	if err != nil {
		fmt.Println("parse metric error: %v body: %s", err, body)
	}
}

func HTTPGet(url string) (code int, respBody string, err error) {
	log.Println("HTTP GET", url)
	client := &http.Client{Timeout: time.Minute}
	resp, err := client.Get(url)
	if err != nil {
		log.Println(err)
		return 0, "", err
	}
	defer resp.Body.Close()
	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Println(err)
		return 0, "", err
	}
	respBody = string(body)
	code = resp.StatusCode
	return code, respBody, nil
}

[Xunzhuo]

@zirain Can you help with it in a sparate PR?

[zirain]

any reason for use 10 for buffer size?

[Xunzhuo]

We need some buffers to make sure the errChan will not be stuck when observing errors from snapshot.Updates. As for the size, I am not sure about it, a suggested value from chatgpt.

[zirain]

can you add some comments for it, or a TODO: find a suitable value for this.

[Xunzhuo]

Done

[arkodg]

@Xunzhuo are we following metric naming conventions similar to EnvoyProxy ?

[arkodg]

hey @Xunzhuo, ive made some suggestions for now mainly around naming, prefer- watchable_subscribed_duration_seconds{resource="provider-resources", runner="gateway-api"}

would also like to replace subscribed with something else, but can't think of anything rn

[arkodg]

thanks @Xunzhuo, hey @zirain can you ptal

[zirain]

can udpate an image of the latest output?

[Xunzhuo]

@zirain updated the pic in PR desc.