Data Plane Debugging and Troubleshooting

[maleck13]

What

Document useful ways to debug what is happening when hitting the Gateway

• Getting logs from the WASMPlugin
• Getting debug / access logs from envoy
• Logs and log level setting for Limitador and/or Authorino
• Logs from Kuadrant Operator
• Common "gotya" type scenarios that we have hit as the developers and testers
• etc

Why

The goal here is to enable users and developers to better understand how RatelimitPolicy & AuthPolicy interacts with the various components and how a user can debug and troubleshoot issues they may hit when applying Policies at the dataplane

[maleck13]

@eguzki @alexsnaps I think this would be a useful thing for the team and also for our users. I am adding one for DNS WDYT

[alexsnaps]

This is probably useful in any case, but I've been toying around with doing this a little differently (as well?). Did a few edits, not specific to RL, but rather the entire data plane tbh (and RateLimit- & AuthPolicy do interact with each other as well...)

I'd like to get Kuadrant as a whole in some "development mode". So that when you'd hit a URL through the Gateway it would, while leveraging the distributed tracing probing mechanism already present in the different code bases, gather the spans and "pipe" them thru http headers (can't, looks like you don't get access to the headers of the response from the host) the dynamic_metadata: well_known_types::Struct of both gRPC responses back to the wasm-shim. That would then gather it all and, on error, use that to populate the response body with a nice message of what happened; or, append it all to the response. Either as headers again (tho that'd be less nice to debug) or find a way to attach it to the actual body.

I was thinking something like some "secret (i.e. a string or something)" possibly attached to some resource, the kuadrant CR? kuadrant_dev_key: lol and now you could get that data on per request by passing ?kuadrant_debug=lol to your URL requested from the Gateway or something...

Anyways, not completely fleshed out, but I think I could put a prototype together in a few days... and looks like I might be able to free that time up in the next couple of weeks... wdyt?

Two advantages I think: more user friendly and it won't get out of date, like a doc would as this automatically updates itself with reality.