Add a key to the access default log text format

[arkodg]

Before

[2023-08-18T22:15:14.085Z] "GET /get HTTP/1.1" 200 - 0 509 1 0 "10.1.3.156" "curl/7.86.0" "ec7ca022-80c8-4e7c-8271-167b260bd190" "www.example.com" "10.102.226.25:3000"

After

{"start_time":"2023-08-30T19:39:16.694Z","method":"GET","x-envoy-origin-path":"/get","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"508","duration":"1","x-envoy-upstream-service-time":"0","x-forwarded-for":"10.1.4.12","user-agent":"curl/7.86.0","x-request-id":"df9cc5ed-8f92-4376-b045-34669dde0ea1",":authority":"www.example.com","upstream_host":"10.110.21.140:3000","upstream_cluster":"httproute/default/backend/rule/0","upstream_local_address":"10.1.4.12:36496","downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:53826","requested_server_name":"-","route_name":"httproute/default/backend/rule/0/match/0/www_example_com"}

Fixes: #1446

[arkodg]

apart from adding a key for every field, this PR also adds more fields.

• There's a bump in char length from 168 to 835

$ echo '[2023-08-18T22:15:14.085Z] "GET /get HTTP/1.1" 200 - 0 509 1 0 "10.1.3.156" "curl/7.86.0" "ec7ca022-80c8-4e7c-8271-167b260bd190" "www.example.com" "10.102.226.25:3000"' | wc -c
     168
$ echo '"start_time":"2023-08-18T22:01:23.901Z","method":"GET","x-envoy-origin-path":"/get","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"511","duration":"0","x-envoy-upstream-service-time":"0","x-forwarded-for":"192.168.65.4","user-agent":"curl/7.86.0","x-request-id":"000985ce-a62d-4899-95f8-c60fa9e5f246",":authority":"www.example.com","upstream_host":"10.104.223.83:3000","upstream_cluster":"default/backend/rule/0/match/0-www.example.com","upstream_local_address":"10.1.3.145:55664","downstream_local_address":"10.1.3.145:10080","downstream_remote_address":"192.168.65.4:50042","requested_server_name":"-","route_name":"default/backend/rule/0/match/0-www.example.com"' | wc -c
     835

• And the separator is a comma / , instead of a space /

Hoping to hear back on preferences cc @envoyproxy/gateway-maintainers

[arkodg]

also blocked on #1799

[arkodg]

updated access log format to use spaces as the delimiter

[codecov]

Codecov Report

Merging #1800 (6838599) into main (3c372dd) will decrease coverage by 0.12%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1800      +/-   ##
==========================================
- Coverage   65.25%   65.14%   -0.12%     
==========================================
  Files          86       86              
  Lines       12480    12480              
==========================================
- Hits         8144     8130      -14     
- Misses       3821     3830       +9     
- Partials      515      520       +5     

Files Changed	Coverage Δ
internal/xds/translator/accesslog.go	98.19% <ø> (ø)

... and 3 files with indirect coverage changes

[zirain]

updated access log format to use spaces as the delimiter

with this format, can you use a json decoder directly in logstash/fluent?

[Alice-Lilith]

I like this format a lot more than the default EnvoyProxy access logs

[haq204]

might as well go all the way to format it to be a JSON string so that external log aggregators can parse it as json structured logs

[arkodg]

@zirain @haq204 imo the JSON string format

{"start_time":"2023-08-18T22:01:23.901Z","method":"GET","x-envoy-origin-path":"/get","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"511","duration":"0","x-envoy-upstream-service-time":"0","x-forwarded-for":"192.168.65.4","user-agent":"curl/7.86.0","x-request-id":"000985ce-a62d-4899-95f8-c60fa9e5f246",":authority":"www.example.com","upstream_host":"10.104.223.83:3000","upstream_cluster":"default/backend/rule/0/match/0-www.example.com","upstream_local_address":"10.1.3.145:55664","downstream_local_address":"10.1.3.145:10080","downstream_remote_address":"192.168.65.4:50042","requested_server_name":"-","route_name":"default/backend/rule/0/match/0-www.example.com"}

is a little harder to read compared to

"start_time":"2023-08-21T23:53:03.858Z" "method":"GET" "x-envoy-origin-path":"/get" "protocol":"HTTP/1.1" "response_code":"200" "response_flags":"-" "response_code_details":"via_upstream" "connection_termination_details":"-" "upstream_transport_failure_reason":"-" "bytes_received":"0" "bytes_sent":"511" "duration":"0" "x-envoy-upstream-service-time":"0" "x-forwarded-for":"192.168.65.4" "user-agent":"curl/7.86.0" "x-request-id":"4cb7e358-e1dc-4b94-abf2-bd699d5a11b7" ":authority":"www.example.com" "upstream_host":"10.105.188.194:3000" "upstream_cluster":"default/backend/rule/0/match/0-www.example.com" "upstream_local_address":"10.1.3.194:39338" "downstream_local_address":"10.1.3.194:10080" "downstream_remote_address":"192.168.65.4:47188" "requested_server_name":"-" "route_name":"default/backend/rule/0/match/0-www.example.com"

im fine proceeding either way, since adding a key makes life so much better than before

[zirain]

@zirain @haq204 imo the JSON string format

{"start_time":"2023-08-18T22:01:23.901Z","method":"GET","x-envoy-origin-path":"/get","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"511","duration":"0","x-envoy-upstream-service-time":"0","x-forwarded-for":"192.168.65.4","user-agent":"curl/7.86.0","x-request-id":"000985ce-a62d-4899-95f8-c60fa9e5f246",":authority":"www.example.com","upstream_host":"10.104.223.83:3000","upstream_cluster":"default/backend/rule/0/match/0-www.example.com","upstream_local_address":"10.1.3.145:55664","downstream_local_address":"10.1.3.145:10080","downstream_remote_address":"192.168.65.4:50042","requested_server_name":"-","route_name":"default/backend/rule/0/match/0-www.example.com"}

is a little harder to read compared to

"start_time":"2023-08-21T23:53:03.858Z" "method":"GET" "x-envoy-origin-path":"/get" "protocol":"HTTP/1.1" "response_code":"200" "response_flags":"-" "response_code_details":"via_upstream" "connection_termination_details":"-" "upstream_transport_failure_reason":"-" "bytes_received":"0" "bytes_sent":"511" "duration":"0" "x-envoy-upstream-service-time":"0" "x-forwarded-for":"192.168.65.4" "user-agent":"curl/7.86.0" "x-request-id":"4cb7e358-e1dc-4b94-abf2-bd699d5a11b7" ":authority":"www.example.com" "upstream_host":"10.105.188.194:3000" "upstream_cluster":"default/backend/rule/0/match/0-www.example.com" "upstream_local_address":"10.1.3.194:39338" "downstream_local_address":"10.1.3.194:10080" "downstream_remote_address":"192.168.65.4:47188" "requested_server_name":"-" "route_name":"default/backend/rule/0/match/0-www.example.com"

im fine proceeding either way, since adding a key makes life so much better than before

it's hard to use for other tools, same opinion with @haq204 .

[arkodg]

created a compatible json string, ptal @haq204 @zirain

[qicz]

so the text and json formats have the same log format?

[arkodg]

so the text and json formats have the same log format?

The default format is Text as JSON string 😄

[zirain]

cc @AliceProxy

[zhaohuabing]

Looks much better than the default one. I always had to check the docs to find out the meaning of the Envoy logs before because lacking of the key before the value :-).