GPUs: take device offline when unhealthy (build logic in go-nvlib)

[jgehrcke]

@klueska originally wrote:

We could start with the same health-check logic that we have in the k8s-device-plugin. This needs a major overhaul, but it is better than having nothing.

Once 26 is complete, we can then backport what was done here to the k8s-dra-driver repo as well.

Or better yet, we should build the logic for handling health checks in an abstract way in go-nvlib so that both projects can take advantage of it.

internal ref: cnt/issues/90

[klueska]

This can / should be done using device taints https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5055-dra-device-taints-and-tolerations

[guptaNswati]

Looks like we never extended the healthchecks in device-plugin. See: https://jirasw.nvidia.com/browse/CNT-2044

This is all the XID errors we look at to mark GPUs unhealthy
https://github.com/NVIDIA/k8s-device-plugin/blob/main/internal/rm/health.go#L39

[guptaNswati]

Based on some investigation into following:

PRs of dranet based on KEP-4817

• Net status google/dranet#78
• Fix status updates only adding one device per claim google/dranet#138
• https://dranet.dev/docs/concepts/interface-status/

which implements device status reporting after network interfaces are allocated and configured by adding status updates to ResourceClaim.Status.Devices[]. Right now, they are batch processing all devices status update in one device per claim as Server-Side Apply (SSA) ownership prevent updating multiple devices.

An example

devices:
  - conditions:
    - lastTransitionTime: "2025-08-22T19:58:22Z"
      message: ""
      reason: NetworkDeviceReady
      status: "True"
      type: Ready
    - lastTransitionTime: "2025-08-22T19:58:22Z"
      message: ""
      reason: NetworkReady
      status: "True"
      type: NetworkReady
    device: dummy0
    driver: dra.net
    networkData:
      hardwareAddress: aa:bb:cc:dd:ee:ff
      interfaceName: dummy-renamed
      ips:
      - 10.0.0.1

And

• KEP-5283 is proposing to add
  .status section to ResourceSlice so DRA drivers can report per-device health data to act (alert, taint, remediate) without prescribing what healthy means. The healthy or unhealthy is decided by the driver but action is external.
• DM chats with @klueska

The way to do this and related follow up tasks:

Health checks

1. first port GPU health checks from device-plugin
2. allow to do additional checks and make it configurable via a flag. need to see if this can be extended to additional metrics like GPU power, temp, ecc errors etc

ResourceClaim Device status update showing post-allocation status

3. enable DRAResourceClaimDeviceStatus FG to add the additional fields in resource claim status to show device condition
4. use above health checks to show GPU status, Ready or not

ResourceSlice Device Health Data pre-allocation (which GPUs to avoid)

5. Store GPU/device health in ResourceSlice
6. add a controller that taints unhealthy GPUs to mark them unallocatable

I will start with 1 but i don't think i can finish it for the August release which is supposed to be due on 08-29-25

[guptaNswati]

I need to check again what we do in device_state.go before creating the list of allocatable devices. But both status updates are dependent on having health checks in the dra driver where

ResourceSlice.status = which GPUs are allocatable on each node. A cluster-wide inventory of healthy and unhealthy GPUs to make scheduling decisions. This list is continuously updated by the driver. For this, health check needs to be more exhaustive.

ResourceClaim.status = how is the GPU doing after getting assigned to a pod of a claim. This is kind of similar (not exactly) to collecting per pod metrics with dcgm-exporter but instead of scarping any metrics, here the dra driver is baking a minimal health view by collecting and publishing some selective metrics for the lifecycle of a given claim.

[klueska]

To start with, I think we should just try and have feature parity with what the device plugin does. Meaning port over its health loop (as you mentioned) and then use the information provided by it to report the GPU health somewhere / taint the devices so they become unschedulable.

We obviously want to do better in the long run, but this simple port will already go a long way, since this is essentially what the device plugin has been doing for years.

[guptaNswati]

Yes agree. @ArangoGutierrez is working on a doc to drive this whole initiative and break into milestones. And this is the first task to copy device-plugin logic in dra driver.