Bug: The streaming every couple of rows blocks for 5-8s

[ciprianveg]

What happened?

Although I obtained good sweep-bench results for 235b UD_Q5_XL as shown below, and with the q4 quant they were 20% faster, in both cases, this annoying blocking happens every couple of rows. I tried changing from 16 threads to 12, but same thing happens. Wilth main llama, is like 25% slower, but is cursive.
My system is a TR 3955wx with 16 cores, 256 ddr4 3200, 2x3090..
Any ideas?
./build/bin/llama-sweep-bench --model /home/ciprian/ai/models/Qwen3-235B-UD_Q5_XL/Qwen3-235B-A22B-UD-Q5_K_XL-00001-of-00004.gguf --alias Qwen3-235B-A22B-UD-Q5_K_XL -fa -fmoe -ctk q8_0 -ctv q8_0 -c 40960 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 0.5 -ot "blk.[0-9].ffn_up_exps=CUDA0,blk.[0-9].ffn_gate_exps=CUDA0,blk.2[0-4].ffn_up_exps=CUDA0,blk.2[0-4].ffn_gate_exps=CUDA0,blk.1[0-9].ffn_up_exps=CUDA1,blk.1[0-9].ffn_gate_exps=CUDA1,blk.2[5-8].ffn_up_exps=CUDA1,blk.2[5-8].ffn_gate_exps=CUDA1,exps=CPU" -ngl 99 --threads 16 --host 0.0.0.0 --port 5002 --ubatch-size 4096 --batch-size 4096

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
4096	1024	0	11.730	349.19	133.500	7.67
4096	1024	4096	12.079	339.11	136.944	7.48
4096	1024	8192	12.514	327.33	140.286	7.30
4096	1024	12288	13.038	314.17	144.478	7.09
4096	1024	16384	13.545	302.40	148.595	6.89
4096	1024	20480	13.943	293.76	151.881	6.74
4096	1024	24576	14.767	277.38	154.643	6.62
4096	1024	28672	15.621	262.21	158.355	6.47
4096	1024	32768	16.561	247.32	161.875	6.33
4096	1024	36864	17.658	231.97	166.160	6.16

Name and Version

llama-server -model /home/ciprian/ai/models/Qwen3-235B-UD_Q5_XL/Qwen3-235B-A22B-UD-Q5_K_XL-00001-of-00004.gguf --alias Qwen3-235B-A22B-UD-Q5_K_XL -fa -fmoe -ctk q8_0 -ctv q8_0 -c 40960 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 0.5 -ot "blk.[0-9].ffn_up_exps=CUDA0,blk.[0-9].ffn_gate_exps=CUDA0,blk.2[0-4].ffn_up_exps=CUDA0,blk.2[0-4].ffn_gate_exps=CUDA0,blk.1[0-9].ffn_up_exps=CUDA1,blk.1[0-9].ffn_gate_exps=CUDA1,blk.2[5-8].ffn_up_exps=CUDA1,blk.2[5-8].ffn_gate_exps=CUDA1,exps=CPU" -ngl 99 --threads 16 --host 0.0.0.0 --port 5002 --ubatch-size 4096 --batch-size 4096

What operating system are you seeing the problem on?

No response

Relevant log output

[ikawrakow]

Not sure. Do you get many tokens at once after the 5-8 seconds pause, or it just did nothing for 5-8 seconds?

[ciprianveg]

It looks like it did nothing, sometimes a second 5-8 s pause comes after just 2 words, other times after 2 rows of text. I tried also with 2048 ubatch size and with using amb 512, no difference. For my hardware, what would be the most suitable build params. I am now setting gpu ggml sched copies to 1, cublast off and ggml cuda on

…

On Wed, 28 May 2025, 08:17 Kawrakow, ***@***.***> wrote: *ikawrakow* left a comment (ikawrakow/ik_llama.cpp#464) <#464 (comment)> Not sure. Do you get many tokens at once after the 5-8 seconds pause, or it just did nothing for 5-8 seconds? — Reply to this email directly, view it on GitHub <#464 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJTBYK5CNGBBG5RUZEC6VBT3AVBHVAVCNFSM6AAAAAB6BBXI7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMJUHE3DMNRSGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ikawrakow]

I'm trying to understand the root cause for this strange behavior. Can you reproduce it using llama-cli ?

[ciprianveg]

I will try this evening and let you know

[ciprianveg]

Something that maybe can give a clue is that my system is cpu limited, i have 8 channels ddr4 3200 ram but the memory read speed is limited to 85Mb/s instead of the theoretical >200Mb/s because the 16 cores are not enough to read at that speed. This is against the standard cpu systems where memory speed is the limiter, not the cpu..

[ciprianveg]

same issue also with llama-cli

[ikawrakow]

Is there disc activity during the pause? Have you looked at process activity during the pause? Are you running llama.cpp with the exact same parameters (apart from -fmoe)? Is there another memory hungry process running (e.g., another llama.cpp server)?

[ciprianveg]

Llama.cpp runs with exact params except fmoe. I have 256Gb ram and almost 100gb free. No other memory hungry process..

…

On Wed, 28 May 2025, 20:22 Kawrakow, ***@***.***> wrote: *ikawrakow* left a comment (ikawrakow/ik_llama.cpp#464) <#464 (comment)> Is there disc activity during the pause? Have you looked at process activity during the pause? Are you running llama.cpp with the exact same parameters (apart from -fmoe)? Is there another memory hungry process running (e.g., another llama.cpp server)? — Reply to this email directly, view it on GitHub <#464 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJTBYK64C6EOOT2D5LADL733AXWFPAVCNFSM6AAAAAB6BBXI7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMJXGA3TQMRXGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ikawrakow]

What about the first two questions? Is the CPU busy during the pauses or just sitting there doing nothing? But at the end it might be easier to just run in the debugger and when it pauses, hit Ctrl-C, type bt, and post the backtrace here.

[ciprianveg]

1. Disk activity, no
2. Top shows llama server between 100-500% when it works and same when it pauses

[kirnat]

Check your PCIe traffic with nvtop or similar when the pause happens. Does it happen if you don't offload any experts to the GPUs?

[ikawrakow]

To test the hypothesis that it gets stuck on copying tensors to the GPU, you can run with -op 26,0,27,0,29,0. This disables offloading tensors to the GPU for any type of matrix multiplication.

But running in the debugger, interrupting with Ctrl-C when it gets stuck, and sending the backtrace will hopefully also diagnose where (in which function) it hangs for so long.

[ciprianveg]

XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MUL_MAT to OFF
XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MUL_MAT_ID to OFF
XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MOE_FUSED_UP_GATE to OFF
XXXXXXXXXXXXXXXXXXXXXXXXXXXX offload(MUL_MAT) = 0
XXXXXXXXXXXXXXXXXXXXXXXXXXXX offload(MUL_MAT_ID) = 0
XXXXXXXXXXXXXXXXXXXXXXXXXXXX offload(MOE_FUSED_UP_GATE) = 0

same issue

[ciprianveg]

Thread 1 "llama-server" received signal SIGINT, Interrupt.
Download failed: Invalid argument. Continuing without source file ./nptl/./nptl/pthread_mutex_lock.c.
0x00007fffee4a014c in lll_mutex_lock_optimized (mutex=0x55555899a0d8) at ./nptl/pthread_mutex_lock.c:48
warning: 48 ./nptl/pthread_mutex_lock.c: No such file or directory

this is from debug

also, with nvtop, when pause happens, the gpus transfer speed is around 1,8GB/s and as soon as it unblocks drops to 50-100MB/s

[ciprianveg]

it happened also with ngl 0, with nothing sent to gpus, only that being slower, like 2-3tok/s also the pause was longer, cca 20s

llama-server --model /home/ciprian/ai/models/Qwen3-235B-UD_Q4_XL/Qwen3-235B-A22B-UD-Q4_K_XL-00001-of-00003.gguf --alias Qwen3-235B-A22B-UD-Q4_K_XL -fa -ctk q8_0 -ctv q8_0 -c 36864 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 0.5 -ngl 0 --threads 16 --host 0.0.0.0 --port 5002 --ubatch-size 4096 --batch-size 4096