[Feature Request] Prompt caching

[gius]

Do you plan to make use of Prompt caching? It should make multiple commands for a single MR faster and cheaper.

https://openai.com/index/api-prompt-caching/
https://www.anthropic.com/news/prompt-caching

[mrT23]

According to the documentation, caching happens automatically.

[gius]

My use case is when you run multiple commands (/describe, /review, /improve) on a single MR. In that case, the part you want to cache is the MR diff.

As I understand the documentation, in case of OpenAI, you'd need all three commands to start with the same text (probably a shared intro followed with the diff) and move the specific instructions to the end of the prompt.
For Claude, you need to put the diff into a separate section marked with "cache_control": {"type": "ephemeral"}

[mrT23]

The prompt needs to be different for the different command

Even the code diff is presented differently (with\without lines, token cap, metadata, ...)

So to your question - we probably dont support auto caching