Anthropic prompt caching support

[monotykamary]

It isn't immediately obvious that when using Anthropic models whether prompt caching is enabled. Some workflow use-cases make it pretty overkill, however coding use-cases would definitely need this enabled to avoid balooning costs.

[evalstate]

More sophisticated Context Management is high up the roadmap, including caching (I'd be interested in hearing any thoughts for features in this area.

I've thought on the Anthropic case a bit, as unexploited cache write costs can remove savings depending on use-case.

As an interim measure, something like if the first generate() call on Anthropic is > 1200 characters then it should deploy a cache-control block, catching production use-cases where you might be using load_prompt_multipart or apply_prompt to setup the agent context. Would that work?

[monotykamary]

Maybe not needed as the minimum cacheable prompt is 1024 tokens (2048 for Haiku), which would otherwise fallback to processing without caching. Anything with a chat history usually needs it as it saves more often than not. A lot of applications always flag cache_control for the last two messages for good measure.

However, workflows that are mostly one-shot agents or handle large unique data can remove those savings quite fast.

[njbbaer]

Are there any plans to support this feature? Prompt caching can offer significant savings for Anthropic models. It could be made optional for situations where it doesn't make sense.

[monotykamary]

I think 9.5 times out of 10, we can just leave it passively enabled by default. Let me cook a quick PR for the anthropic provider and let everyone start from there.

[evalstate]

About to respond to this -- @monotykamary that would be much appreciated. If at all possible a couple of things I would love:

• Provider config for Anthropic to have a cache_mode of off, prompt or auto.
• prompt would set a cache block on the apply_prompt. That will allow large, common prompts / non-history maintaining responders to be efficient.
• auto should do whatever you're planning :)