AMD RX 9060XT ROCm error: invalid device function

[mudler]

Discussed in #6008

^{Originally posted by cybershaman August 10, 2025}
Hello all!

Been hitting my head against this issue for some time now so though I might reach out to the community here for advice :-)

Host - Proxmox 8.4 Bare Metal Install

• AMD RX 9060XT (gfx1200)
• Kernel 6.11.11 to leverage proper GPU detection
• amdgpu Kernel module compiled and inserted as dkms (probably not quite needed though?)
• "rocm-smi" functional
• "rocminfo" shows only 1 Agent (CPU)

Container (LXC) - Ubuntu 22.04.5 LTS

• ROCm 6.4.2 installed via amdgpu-install (of course no DKMS)
• following devices passed through from Host with proper cgroup perms for container:
  • /dev/kfd
  • /dev/dri/*
• "rocm-smi" functional
• "rocminfo" functional (2 Agents, CPU & GPU)

LocalAI compiled from git source:

• REBUILD=true BUILD_TYPE=hipblas GPU_TARGETS=gfx1200 GO_TAGS=stablediffusion,tts BUILD_GRPC_FOR_BACKEND_LLAMA=true BUILD_GRPC=true make build
• using environment variable HSA_OVERRIDE_GFX_VERSION=12.0.0 just in case
• testing with a LLAMA model, relevant config parts:
  • backend: rocm-llama-cpp
  • f16: true
  • threads: 0
  • gpu_layers: 200

LocalAI appears to be recognising and utilizing the GPU as there is VRAM movement and a tiny bit of GPU usage visible while querying the API.
However, throws an error eventually:

11:04AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr ggml_cuda_init: found 1 ROCm devices:
11:04AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr   Device 0: AMD Radeon RX 9060 XT, gfx1200 (0x1200), VMM: no, Wave Size: 32
[...]
11:04AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr load_tensors: offloading 32 repeating layers to GPU
11:04AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr load_tensors: offloading output layer to GPU
11:04AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr load_tensors: offloaded 33/33 layers to GPU
11:04AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr load_tensors:   CPU_Mapped model buffer size =    70.32 MiB
11:04AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr load_tensors:        ROCm0 model buffer size =  3877.56 MiB
[...]
11:05AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr ggml_cuda_compute_forward: MUL_MAT failed
11:05AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr ROCm error: invalid device function
11:05AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr   current device: 0, in function ggml_cuda_compute_forward at /LocalAI/backend/cpp/llama-cpp-fallback-build/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2513
11:05AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr   err
11:05AM DBG GRPC(discolm_german-127.0.0.1:37515): stderr /LocalAI/backend/cpp/llama-cpp-fallback-build/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:84: ROCm error

Anyone have any ideas and/or pointers?
Thank you very much in advance!

[cybershaman]

I would love to test this for you in any way shape or form. So far I've tried the following steps but failed to rebuild the backend:

• removed comment from lines 35, 36 and 38
• commented out line 37
• line 35 changed to:
  GPU_TARGETS ?= gfx803,gfx900,gfx906,gfx908,gfx90a,gfx942,gfx1010,gfx1030,gfx1032,gfx1100,gfx1101,gfx1102,gfx1200

Afterwards attempt to compile:
make -C LocalAI/backend/cpp/llama-cpp llama-cpp-fallback

results in error:

CMake Error at tools/grpc-server/CMakeLists.txt:21 (find_package):
  Could not find a package configuration file provided by "Protobuf" with any
  of the following names:

    ProtobufConfig.cmake
    protobuf-config.cmake

  Add the installation prefix of "Protobuf" to CMAKE_PREFIX_PATH or set
  "Protobuf_DIR" to a directory containing one of the above files.  If
  "Protobuf" provides a separate development package or SDK, be sure it has
  been installed.

System packages installed:

ii  libprotobuf-c-dev:amd64              1.3.3-1ubuntu2.1
ii  libprotobuf-c1:amd64                 1.3.3-1ubuntu2.1
ii  libprotobuf-dev:amd64                3.12.4-1ubuntu7.22.04.4
ii  libprotobuf-lite23:amd64             3.12.4-1ubuntu7.22.04.4
ii  libprotobuf23:amd64                  3.12.4-1ubuntu7.22.04.4
ii  protobuf-c-compiler                  1.3.3-1ubuntu2.1
ii  protobuf-compiler                    3.12.4-1ubuntu7.22.04.4
ii  protobuf-compiler-grpc               1.30.2-3build6

[mudler]

This seems to be not supported (yet?) I've tried to target gfx1200 with the current images but didn't worked out. To test it locally (but will complain as not recognized):

AMDGPU_TARGETS=gfx1200 BASE_IMAGE="rocm/dev-ubuntu-22.04:6.1" BUILD_TYPE=hipblas make backends/llama-cpp

I have to still test if builds correctly with the new rocm images. But you can already play with it if you manage to build successfully.

[mudler]

ok, bumping rocm images seems to be working, at least I'm compiling it locally now. I can't test it yet, but as soon as the PR is merged and images built it will be available in the backends gallery, you have to just install the "development" version of llama-cpp with hipblas

[cybershaman]

Just FYI and as verification, plus one finding:

• rocm-llama-cpp-development backend now works! Bumping ROCm to 6.4.x really did the trick here.
• first run took a bit of time while the backend was linking libraries via ld it seems?
• I had to use environment variable GPU_MAX_HW_QUEUES=1 for local-ai to circumvent the GPU remaining at 100% usage even after end token

Fantastic job, thank you! 👍