Allow Engines method to take additional parameters

[yeya24]

This allows RemoteEndpoints implementation to use different engines based on the query itself.
For example, in my usecase I will select different engines based on query, start, end and lookback delta value.

[yeya24]

Upgraded Prometheus version to latest main and also includes prometheus/prometheus@fd3630b.

The point of having ctx is to allow downstream optimizer to be aware of the user. We can extract user ID from the request ctx.

[yeya24]

Kindly ping for review.

[fpetkovski]

Is this needed because we cannot add new logical optimizers from outside the engine?

[yeya24]

No, even if we can define own optimizer, the engines interface is still unaware of the query, time and other metadata like userid so it cannot know which remote engines it should pick.

[fpetkovski]

But could you then prune engines in the optimizer itself? The same parameters are present in the logical plan optimizer interface.

[yeya24]

Got it, I think it works but the RemoteEngine interface will be bypassed for me. However, filtering by external labels doesn't work for me so I need to write my own logic so this way I am just using the Optimizer, not the RemoteEngine interface. In some sense, this is a workaround.

But as you said, since we cannot define external optimizer, it is still not doable.

[fpetkovski]

I think it's fine to expose the parser publicly because optimizers are an extension point for the engine. Feel free to raise a PR exposing what you need if that is blocking you right now. Ideally we could use the PromQL structs directly, but we can try to get there incrementally.