Eval bug: llama.cpp/tools/server/server.cpp:3562: GGML_ASSERT(batch.n_tokens > 0) failed

[vt-alt]

Name and Version

G:\llama_cpp> llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from G:\llama_cpp\ggml-cuda.dll
load_backend: loaded RPC backend from G:\llama_cpp\ggml-rpc.dll
load_backend: loaded CPU backend from G:\llama_cpp\ggml-cpu-haswell.dll
version: 6387 (4fd1242)
built with clang version 19.1.5 for x86_64-pc-windows-msvc

Operating systems

Windows

GGML backends

CUDA

Hardware

NVIDIA GeForce RTX 3060

Models

Voxtral-Mini-3B-2507-GGUF:Q4_K_M and mistralai_Voxtral-Small-24B-2507-GGUF

Problem description & steps to reproduce

Voxtral-Mini-3B-2507-GGUF works via llama-mtmd-cli but fails via llama-server.

This command works (taken from the tests): llama-mtmd-cli -hf ggml-org/Voxtral-Mini-3B-2507-GGUF:Q4_K_M --image test-2.mp3 -p "what is the publisher name of the newspaper?" --temp 0 -n 128

API request to the llama-server fails with: server.cpp:3562: GGML_ASSERT(batch.n_tokens > 0) failed message. Full log below.

Similar bug reports there #13433

First Bad Commit

No response

Relevant log output

srv  log_server_r: request: GET /props 192.168.1.1 200
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 131072, n_keep = 0, n_prompt_tokens = 190
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 3, n_tokens = 3, progress = 0.015789
slot update_slots: id  0 | task 0 | kv cache rm [3, end)
srv  process_chun: processing audio...
encoding audio slice...
audio slice encoded in 5542 ms
decoding audio batch 1/1, n_tokens_batch = 187
audio decoded (batch 1/1) in 15 ms
srv  process_chun: audio processed in 5560 ms
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 190, n_tokens = 0, progress = 1.000000
D:/a/llama.cpp/llama.cpp/tools/server/server.cpp:3562: GGML_ASSERT(batch.n_tokens > 0) failed

[vt-alt]

@ngxson Can you look into this?

The reproducer using curl on Linux:

echo '{ "model": "Voxtral-Mini-3B-2507-GGUF:Q4_K_M", "temperature": 0, "max_tokens": 128, "prompt": { "prompt_string": "what is the publisher name of the newspaper? <__media__>", "multimodal_data": ["'$(base64 -w0 tools/mtmd/test-2.mp3)'"] } }' \
  | curl -X POST http://localhost:8080/completion \
     -H "Content-Type: application/json" \
     -d @-
curl: (52) Empty reply from server

llama-server console log:

$ llama-server -hf ggml-org/Voxtral-Mini-3B-2507-GGUF:Q4_K_M
...
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 198
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 11, n_tokens = 11, progress = 0.055556
slot update_slots: id  0 | task 0 | kv cache rm [11, end)
srv  process_chun: processing audio...
encoding audio slice...
audio slice encoded in 146 ms
decoding audio batch 1/1, n_tokens_batch = 187
audio decoded (batch 1/1) in 2963 ms
srv  process_chun: audio processed in 3109 ms
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 198, n_tokens = 0, progress = 1.000000
/usr/src/RPM/BUILD/llama.cpp-6351/tools/server/server.cpp:3542: GGML_ASSERT(batch.n_tokens > 0) failed
Aborted (core dumped)

[vt-alt]

I thought of the reproducer for OpenAI-compatible API. Perhaps, this is different bug. First request is OK, following up request crashes the llama-server:

$ echo '{
    "model": "Voxtral-Mini-3B-2507-GGUF:Q4_K_M",
    "temperature": 0,
    "max_tokens": 128,
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "what is the publisher name of the newspaper?" },
            {
              "type": "input_audio",
              "input_audio": {
                "data": "'"$(base64 -w0 tools/mtmd/test-2.mp3)"'",
                "format": "mp3"
            }
          }
        ]
      }
    ]
  }' | curl -X POST http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d @-
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"The publisher name of the newspaper \"The New York Times\" is **The New York Times Company**."}}],"created":1757129121,"model":"Voxtral-Mini-3B-2507-GGUF:Q4_K_M","system_fingerprint":"b6351-5d804a49 [alt1]","object":"chat.completion","usage":{"completion_tokens":21,"prompt_tokens":199,"total_tokens":220},"id":"chatcmpl-6G0V8g8ptHtz00PHlzxFXlUlpTHrMff5","timings":{"prompt_n":199,"prompt_ms":197.43,"prompt_per_token_ms":0.9921105527638191,"prompt_per_second":1007.9521855847642,"predicted_n":21,"predicted_ms":86.323,"predicted_per_token_ms":4.110619047619047,"predicted_per_second":243.27236078449545}}

$ echo '{
    "model": "Voxtral-Mini-3B-2507-GGUF:Q4_K_M",
    "temperature": 0,
    "max_tokens": 128,
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "what is the publisher name of the newspaper?" },
            {
              "type": "input_audio",
              "input_audio": {
                "data": "'"$(base64 -w0 tools/mtmd/test-2.mp3)"'",
                "format": "mp3"
            }
          }
        ]
      }
    ]
  }' | curl -X POST http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d @-
curl: (52) Empty reply from server

llama-server log for the first request:

srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 199
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 11, n_tokens = 11, progress = 0.055276
slot update_slots: id  0 | task 0 | kv cache rm [11, end)
srv  process_chun: processing audio...
encoding audio slice...
audio slice encoded in 124 ms
decoding audio batch 1/1, n_tokens_batch = 187
audio decoded (batch 1/1) in 20 ms
srv  process_chun: audio processed in 144 ms
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 199, n_tokens = 1, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 199, n_tokens = 1
slot      release: id  0 | task 0 | stop processing: n_past = 219, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =     197.75 ms /   199 tokens (    0.99 ms per token,  1006.34 tokens per second)
       eval time =      86.24 ms /    21 tokens (    4.11 ms per token,   243.50 tokens per second)
      total time =     283.99 ms /   220 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200

llama-server log for the second request (with a crash):

srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 23 | processing task
slot update_slots: id  0 | task 23 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 199
slot update_slots: id  0 | task 23 | need to evaluate at least 1 token for each active slot, n_past = 199, n_prompt_tokens = 199
slot update_slots: id  0 | task 23 | kv cache rm [198, end)
... call trace output ...
terminate called after throwing an instance of 'std::runtime_error'
  what():  Chunk not found
Aborted (core dumped)