Feature Request: Deepthink with Confidence

[ipdeft]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

"DeepConf leverages model internal confidence signals to dynamically filter out low-quality reasoning traces
during or after generation. It requires no additional model training or hyperparameter tuning and can be seamlessly integrated into existing serving frameworks."

References
https://jiaweizzhao.github.io/deepconf/
https://jiaweizzhao.github.io/deepconf/static/pdfs/deepconf_arxiv.pdf

Motivation

Increased accuracy while decreasing generated tokens

Possible Implementation

Hopefully a port of the provided patched VLLM implementation is possible into llama.cpp (https://jiaweizzhao.github.io/deepconf/static/htmls/code_example.html)

[Mushoz]

It doesn't reduce token usage, it actually strongly increases it because you are doing parallel inference. It has reduced token usage compared to the naive implementation of parallel inference, since this technique does early stopping of reasoning streams with low confidence scores.

[pnocera]

FYI I'm giving it a try in a WIP here https://github.com/pnocera/llama.cpp/blob/master/docs/deepconf.md

[Mushoz]

Oh wow that was quick! Thank you so much. Curious to see how it will work and perform. Do you have any plans if upstreaming this via a PR?

[pnocera]

I'm trying to build it right now. I have no plan, just experimenting... I'm curious too !

Edit : Now I did integrate it in llama-client and llama-server. didn't do benchmarks yet

Note : my fork has been forked with several enhancements that I didn't check yet https://github.com/R-Dson/llama.cpp

[Mushoz]

@pnocera They actually created a PR back to your branch. Might be worth checking out so we can keep all the work in one place. What's the status of this feature in llama-server? Is it in a place where it can be used? If so, I'd be happy to help test.

[pnocera]

I merged the PR. The status is I built it and ran it, did a few tests but didn't do any benchmarks. @Mushoz if you can help testing that would be great as I don't have much time these days with my work.