[Bug]: GPU inference produces completely incorrect results

[abrbird]

OpenVINO Version

2025.2.0

Operating System

Other (Please specify in description)

Device used for inference

GPU

Framework

ONNX

Model used

https://github.com/Peterande/D-FINE

Issue description

Inference on GPU using Openvino produces identical/blank results.
However, if I run the inference model on the CPU, the results are valid.
No matter what image I'm using as input, outputs stays same.

Running on Ubuntu 22.04.1 using Docker image openvino/ubuntu22_runtime:2025.2.0 bash.

Am I doing something wrong?

Step-by-step reproduction

1. Run Docker container
   sudo docker run --privileged -it --rm -v /dev:/dev --device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) openvino/ubuntu22_runtime:2025.2.0 bash

2. Clone the repo and install requirements

3. Download and export D-Fine (checked m and l sizes) model pth -> onnx (or pth -> onnx -> IR xml+bin) using script

python3 ./tools/deployment/export_onnx.py -c configs/dfine/objects365/dfine_hgnetv2_m_obj2coco.yml --simplify -r dfine_m_obj2coco.pth

4. Run inference setting device_name to GPU using script

python3 ./tools/inference/openvino_inf.py --ov_model ./dfine_m_obj2coco/dfine_m_obj2coco.onnx --image img.png

Relevant log output

On CPU:
scores: [0.95673555 0.93977463 0.88885874 0.62019074 0.4028598 ...]
labels: [1 16  7  2  3 58 58  1 58  3  0 58 ...]

On GPU:
scores: [0.05419922 0.05419922 0.05419922 0.05419922 0.05419922 0.05419922 ...]
labels: [0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 ...]

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[YuChern-Intel]

Could you directly share your dfine_m_obj2coco.onnx model as I did not find dfine_m_obj2coco.pth in the repository?

[abrbird]

Yes, sure
https://drive.google.com/file/d/17XUOSwZPVBPbassDaD-98wZKJjl9t8D8/view?usp=sharing

just in case if dfine_m_obj2coco.pth needed https://github.com/Peterande/storage/releases/download/dfinev1.0/dfine_m_obj2coco.pth

[YuChern-Intel]

Add the following codes in the openvino_inf.py file for GPU to produce good result:

ov_config = {"GPU_DISABLE_WINOGRAD_CONVOLUTION": "YES"}
self.compile_model = self.core.compile_model(self.model_path, device_name,ov_config)

[abrbird]

I added the config as you suggested and got results that were different from previous ones, but they vary greatly compared to the CPU's.
Model produces:
labels - object class id (int in the [0, 79])
scores - confidence in the objects class (float in the [0,1])
boxes - objects' bounding box (4 values in the [0, 640])

Therefore, I'm getting low scores, and all detected objects have incorrect labels and bounding boxes, also some boxes have negative values - looks very strange

CPU:

labels: [1 16  7  3  2 ...]
scores: [0.95798177 0.9448116  0.7443941  0.7154298  0.61971915 ...]

GPU:

labels: [1  3  3  1 58 ...]
scores: [0.2697754  0.1595459  0.15710449 0.1550293  0.10144043 ...]

[YuChern-Intel]

I replicate the results using native environment and found that GPU inference able to produce correct results with OpenVINO 2025.3.0.

Please try using the latest OpenVINO docker image, openvino/ubuntu22_runtime:2025.3.0.

[abrbird]

Thank you, ov 2025.3 and {"GPU_DISABLE_WINOGRAD_CONVOLUTION": "YES"} helped