Add method to reset model KV cache (llama_reset)

[shashankb-cc]

Feature Request

I’m using LLM.swift for on-device inference in a macOS app. I noticed that there is currently no exposed way to reset the model context or KV cache between runs without reinitializing the entire model. This becomes an issue in apps that perform multiple inferences in sequence (e.g., analyzing transcript segments or handling multiple short prompts in a session).

Problem

Calling .predict(...) multiple times on the same LLMModel instance can cause:

• Errors like:
• decode: failed to find KV cache slot for ubatch of size 512
• llama_decode: failed to decode, ret = 1
• Memory buildup or inference inconsistencies over time.
• Reinitializing the model every time is inefficient and causes performance degradation, especially on smaller prompts.

Proposed Solution

Expose a method on LLMModel like this:

public func resetContext() {
  llama_reset(context)
}

[lockieluke]

is that why it's picking up stuff that was mentioned before?

[shashankb-cc]

is that why it's picking up stuff that was mentioned before?

Yep, I fixed it for now by running some memory-related functions. Let me know if you’d like me to share them.

[daviddahl]

is that why it's picking up stuff that was mentioned before?

Yep, I fixed it for now by running some memory-related functions. Let me know if you’d like me to share them.

Would like to see what you have done here

[shashankb-cc]

is that why it's picking up stuff that was mentioned before?

Yep, I fixed it for now by running some memory-related functions. Let me know if you’d like me to share them.

Would like to see what you have done here

thats a lot of code do you want me to paste it here itself ?

[daviddahl]

is that why it's picking up stuff that was mentioned before?

Yep, I fixed it for now by running some memory-related functions. Let me know if you’d like me to share them.

Would like to see what you have done here

thats a lot of code do you want me to paste it here itself ?

Perhaps in a gist?

[daviddahl]

BTW, this is some analysis I had Claude do looking at my crashes and llama.cpp and LLM.swift code: https://gist.github.com/daviddahl/138d4f3fdcf6c939b43db1717f3a4264

I hope this is helpful, if so I will file an issue, @eastriverlee

[shashankb-cc]

sorry mate the solution I added worked temporarily so I didn't wanted to give wrong solution .

is that why it's picking up stuff that was mentioned before?

Yep, I fixed it for now by running some memory-related functions. Let me know if you’d like me to share them.

Would like to see what you have done here

thats a lot of code do you want me to paste it here itself ?

Perhaps in a gist?