feat(nvidia-fabricmanager): support Blackwell baseboards (DGX/HGX B100/B200/B300)

[npdgm]

Quoting: https://docs.nvidia.com/datacenter/tesla/pdf/fabric-manager-user-guide.pdf

On NVIDIA DGX-B200, HGX-B200, and HGX-B100 systems and later, the FabricManager package needs an additional NVLSM dependency for proper operation.
NVLink Subnet manager (NVLSM) originated from the InfiniBand networking and contains additional logic to program NVSwitches and NVLinks.

Not running the NVLSM on Blackwell and newer fabrics with NVLink 5.0+ will result in FabricManager failing to start with error NV_WARN_NOTHING_TO_DO. NVSwitches will remain uninitialized and applications will fail with the CUDA_ERROR_SYSTEM_NOT_READY or cudaErrorSystemNotReady error. The CUDA initialization process can only begin after the GPUs complete their registration process with the NVLink fabric.
A GPU fabric registration status can be verified with the command: nvidia-smi -q -i 0 | grep -i -A 2 Fabric. An In Progress state indicates that the GPU is being registered and FabricManager is likely not running or missing the NVLSM dependency. A Completed state is shown when the GPU is successfully registered with the NVLink fabric.

The FabricManager package includes the script nvidia-fabricmanager-start.sh, which is used to selectively start FabricManager and NVLSM processes depending on the underlying platform.
A key aspect of determining whether an NVLink 5.0+ fabric is present is to look for a Limited Physical Function (LPF) port in InfiniBand devices. To differentiate LPFs, the Vital Product Data (VPD) information includes a vendor-specific field called SMDL, with a non-zero value defined as SW_MNG. The first device is then selected, and it's port GUID is extracted and passed to NVLSM and FabricManager.
So both services share a configuration key that result from a common initialization process. Additionally, they communicate with each other over a Unix socket.

This patch introduces the following changes:

• Adds NVLSM to the nvidia-fabricmanager extension.
• Introduces a new nvidia-fabricmanager-wrapper program to replicate the initialization process from nvidia-fabricmanager-start.sh:
  • Detects NVLink 5.0+ fabrics and extracts an NVSwitch LPF port GUID. This is done by calling libibumad directly with CGO instead of parsing the output of the ibstat command as in the upstream script.
  • Starts FabricManager, and NVLSM only when needed.
  • Keeps both process lifecycles synchronized and ensures the Talos container will restart if either process crashes.
• Refactors the nvidia-fabricmanager container to be self-contained, as this service does not share files with other nvidia-gpu extentions.

[npdgm]

@frezbo I'm building rdma-core here, so perhaps #127 could be used instead?

Edit: on second thought, it seemed too complex to have a decision tree determining whether an rdma-core extension dependency is necessary for nvidia-fabricmanager to run, depending on the platform. This was especially true considering it's only for pulling a 50 KB library file. So instead, I switched to building only libibumad.a and linking the archive.

[frezbo]

@frezbo I'm building rdma-core here, so perhaps #127 could be used instead?

I guess we can split this PR out, rdma-core in a different one, feel free to work on #127

[npdgm]

• not initializing (loading ?) the GRPC plugin
• no unix socket file created for FM

The issue was due to the configuration not being found because of an incorrect path.

The PR has been tested on a DGX-B200 system, and I expect all new baseboards to work on Talos now.
It depends on siderolabs/pkgs#1245 and siderolabs/talos#11211 to be effective. However, the lack of the IB kernel module won't pose any problems for starting nvidia-fabricmanager on older GPU architectures.

It will be required for Blackwell baseboard users to load the kernel module ib_umad so that the extension can detect the NVSwitch and start NVLSM:

machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset
      - name: ib_umad

[dsseng]

Thanks, this looks like a really complex task you accomplished. Please see some of the questions I had, maybe those would help us clarify some questions

[dsseng]

Should this change be done for all models (e.g. A100 only needs nvfm, not nvlsm)? Has this been regression-tested against a system not requiring nvlsm?

This could perhaps be done in the integration-aws-nvidia-nonfree CI workflow perhaps, if you cannot test, so that's not a big deal I guess

[npdgm]

Has this been regression-tested against a system not requiring nvlsm?

No, unfortunately I can't at this time and it might not even happen. So using the CI workflow sounds like the best option.

What was tested on Blackwell is altering the IB lookup function so that no NVSwitch configuration port is found. In this case NVLSM was skipped as expected and NVFM started like it used to be. But being a Blackwell system it failed to initialize the baseboard.

[dsseng]

No, unfortunately I can't at this time and it might not even happen. So using the CI workflow sounds like the best option.

Okay, that should be possible. Keeping this thread open so Noel or someone else keeps need for CI test in mind as I recall that test being occasionally disabled due to pricey GPU and often failures due to no resources available in the cloud

[npdgm]

Is this likely to merge for 1.11.0 or are you in a freeze window already? I guess it wouldn't hit a patchlevel later.
No pressure, I just need a little insight to adjust ops planning with clients. Thanks!

[frezbo]

Is this likely to merge for 1.11.0 or are you in a freeze window already? I guess it wouldn't hit a patchlevel later. No pressure, I just need a little insight to adjust ops planning with clients. Thanks!

if we can sort out everything before siderolabs/talos#10907 beta.0 it we can pull this in

[frezbo]

@npdgm is this good for another round of review?

[npdgm]

@npdgm is this good for another round of review?

Yes! It's been in use with multi-GPU inference in production for 3 months, no problem. Lately built on 1.10.4.
I still can't validate this on pre-Blackwell baseboards myself, sorry. Also any new changes would take some time to test on B200 as I would need to schedule a maintenance. Resources are scarce...

[frezbo]

@npdgm is this good for another round of review?

Yes! It's been in use with multi-GPU inference in production for 3 months, no problem. Lately built on 1.10.4. I still can't validate this on pre-Blackwell baseboards myself, sorry. Also any new changes would take some time to test on B200 as I would need to schedule a maintenance. Resources are scarce...

Cool, I'll get this worked on and run our tests and see if this works for existing stuff

[frezbo]

i think we don't need to do this hack anymore? since this stage doesn't seem to use anything from wolfi anymore?

[npdgm]

Fixed.

Sorry I had it fixed in production but not lts. Both are the same now, except for vars.
So the base stage is back to /, and Wolfi is pulled to /wolfi-base. We still need Wolfi because of the libgcc_s.so.1 file we copy to the extension rootfs, for NVLSM. No package is installed so that file is locked down by WOLFI_BASE_REF. Anyway, GCC's runtime library is quite small and portable, we could pick one from any distro or version and that would work.

[dsseng]

Agreed, this shouldn't need Wolfi IIUC, since no Glibc build happens here

[npdgm]

Thanks. See my answer to frezbo above, Wolfi is moved to it's own root but still needed, as a source for GCC's runtime library that's not present in Talos.

[frezbo]

Let's create a meta pkg and copy the file to the rootfs in there, then use that as depends on here, so the intent is clear and obvious

[npdgm]

It looks much better indeed. Let me know what you think I wasn't sure how to name it.
I'm satisfied with how it builds so far. Once this round of review is cleared I'll schedule an upgrade for validation.

[frezbo]

this looks good, could you validate from your side? Then will see about getting this merged and run through our NVIDIA tests

[frezbo]

so is this good to go?

[npdgm]

Can't tell yet, sorry. Last maintenance window was wasted on the issue with containerd file permission.
I've got a new slot from my client, we'll upgrade nodes on Tueday at 11 GMT.

[npdgm]

It failed again. I don't get it, looks like I can't make working builds anymore.

Previously I never had to call the imager with --system-extension-image ghcr.io/siderolabs/glibc:x. Now if I don't, then the service container fails to bind mount /usr/local/glibc because it doesn't exist.
But then, when explicitely imaging with the glibc extention, then I still get errors I had never see before about libraries such as :

nv-fabricmanager: /usr/bin/nv-fabricmanager: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

[frezbo]

if glibc is added as a dependency it should work, maybe it's looking for libz from the alpine base

[npdgm]

Sorry about the noise, I figured it out. Silly problem on my side.

So it's good to go, I validated the current PR head.

• rebased on 1.10.6 : OK and now used in production
• rebased on release-1.11 : OK too

[frezbo]

how is the production and lts versions determined?