Fix issue #382 (multiple publish output files with same relative path)

[LucaMaccarini]

Fixes #382: On Windows, runtime files from backend packages are currently flattened during copy. This means that all files from subfolders end up in the same output directory, causing conflicts when multiple files share the same name.

Proposed fix

1. Correct the file LLamaSharpBackend.props as shown in the changes included in this PR.

2. When building for Windows, and after installing the Windows-specific backend packages, make sure to add ExcludeAssets="Native" to the PackageReference in the .csproj, as shown below:

<PackageReference Include="LLamaSharp.Backend.Cpu" Version="0.24.0" ExcludeAssets="Native"/>

Although ExcludeAssets="Native" is strictly necessary only for Windows backends when building on Windows, it shouldn’t cause issues for other backends. To be safe, it can be applied to all imported backend packages. I have tested this with both LLamaSharp.Backend.Cpu and LLamaSharp.Backend.Android installed in the same MAUI project. This attribute tells MSBuild not to copy the runtime files from the NuGet package to the output directory, because copying is already handled correctly in the LLamaSharpBackend.props file updated in step 1.

How to Apply this fix before it is merged

If you urgently need to fix this issue, you can temporarily apply this change by copying the corrected LLamaSharpBackend.props content (from step 1) into each backend package directory, e.g.:

C:\Users\<your user>\.nuget\packages\<llamasharp backend>\<version>\build\netstandard2.0\<llamasharp backend>.props

and then import the package as indicated in step 2.

Note: this modification will be lost if the package is updated.

Future improvements:

Avoid using the default Runtimes folder, since MSBuild always flattens its contents building for windows causing conflicts. Using a custom folder for runtime files would eliminate the need to set ExcludeAssets="Native" in the package reference, and then change the LLamaSharpBackend.props file to correctly fetch runtime files from the custom folder.

Tests

I tested this fix on a limited scenario with a MAUI project having both LLamaSharp.Backend.Cpu and LLamaSharp.Backend.Android installed. The goal was to ensure the app works on both types of devices. Given the large heterogeneity of the backends, I hope this fix will work for other backends as well.

[martindevans]

Thanks for looking into this. #382 has been a long standing issue that a few people (including me) have had a look at but failed to solve, I'm happy to see the fix turns out to be so simple!

I'll find some time to test this out soon, I'd really like to get it into the next update if possible.

Would you be interested in developing a followup PR with your suggested improvements? Removing the extra installation step and making it just work would be fantastic.

[LucaMaccarini]

Glad I could help, this issue had been around for a while, let’s see if the tests confirm the fix.
At the moment, with my fix, the runtimes are packaged into the NuGet packages only under the LLamaSharpRuntimes directory, but then imported into the project under the runtimes folder. This way, nothing should change on the code side and the runtimes end up where they’re expected to be.

One note

I’m not sure why, but the last time I worked on this bug I was able to pack the NuGets with the original paths. This time, however, I had to adjust them. In all the .nuspec files the src attribute of the <file> elements no longer starts with runtimes/ but with ../.

For example:
<file src="../deps/noavx/ggml.dll" target="LLamaSharpRuntimes\win-x64\native\noavx\ggml.dll" />

I hope this doesn’t break the way you build the packages, if it does, please let me know and I’ll restore the src paths to their previous state.

Testing

As before, I tested with a MAUI project using LLamaSharp.Backend.Cpu and LLamaSharp.Backend.Android. This time, I added the packages using the usual NuGet Package Manager GUI, and it was not necessary to use ExcludeAssets="Native".

[m0nsky]

Hi @LucaMaccarini , does this PR need any additional changes? I'm not sure what you mean by that last note (have you found out if we need to start with runtimes/ or ../ ?). I'm currently running into the same issue on linux (but have not yet tested this PR).

Edit
Ok, I had a chance to test this PR. I built the nuget packages locally and added the local source to my project. The PR it's current state, it will result in a bunch of errors like this while packing the nuget files:

Could not find a part of the path 'C:\Users\m0nsky\Desktop\nuget_publish_binaries\LLamaSharp\deps\cu12.4.0'.
"Packing LLamaSharp.Backend.Vulkan.Linux.nuspec"
Attempting to build package from 'LLamaSharp.Backend.Vulkan.Linux.nuspec'.
Could not find a part of the path 'C:\Users\m0nsky\Desktop\nuget_publish_binaries\LLamaSharp\deps\vulkan'.

After replacing all ../deps/ with runtimes/deps/ in the .nuspec files, the .nupkg files build correctly. I installed the packages in my project, and published a linux x64 release. The initial issue is fixed, however, the runtimes folder only seems to contain CPU backend folders (even though I have the CUDA12 backend installed). I think we're close, any ideas?

[LucaMaccarini]

Hi @m0nsky, Thanks a lot for testing and for the detailed feedback

I actually suspected that the path part should have remained runtimes/deps/.
In my local setup, I was using nuget.exe directly, passing the .nuspec file to create the package. Since the .nuspec files are located one level deeper respect to deps folder, I assumed I needed to prepend ../deps.

Example of the command I used locally:

.\nuget.exe pack "C:\Users\luca.maccarini\Desktop\luca\myProjects\LLamaSharp\LLamaSharp\LLama\runtimes\build\LLamaSharp.Backend.Cuda12.Linux.nuspec" -Version 0.25.0

I’ll fix the initial path in this PR so that it uses runtimes/deps/.

Could you kindly share how you usually generate the NuGet packages on your side? (what script or command do you run). That would help me align with your workflow and avoid mismatches like this.

Regarding the point you mentioned about the LLamaSharp.Backend.Cuda12.Linux package also including the CPU backend: this is expected, because in its .nuspec there is an explicit dependency on LLamaSharp.Backend.Cpu (introduced by @martindevans in commit 02eedd94).
Here’s the relevant metadata:

<dependencies>
    <dependency id="LLamaSharp.Backend.Cpu" version="$version$" />
</dependencies>

That dependency is the reason why the CPU backend is included when packing the CUDA12 backend.

[m0nsky]

Yes, absolutely, here are my steps:

• Download nuget.exe and add to PATH (environment variable, Windows 11)
• Clone your WIP branch, build the project
• Pack the nuget packages, using my method described here

Basically, I put that nuget_pack.bat inside the LLamaSharp folder, and run it. After packing the nuget packages, the .nupkg files are located in LLamaSharp/temp.

What I meant by the CPU/CUDA12 issue, is that the CUDA12 runtimes seem to be completely missing when building my application.

When I extract LLamaSharp.Backend.Cuda12.Linux.3.0.1.nupkg using 7zip, I can see that LLamaSharpRuntimes\linux-x64\native\cuda12 exists in the nuget package. However, when building my application which has the LLamaSharp and LLamaSharp.Backend.Cuda12 nuget packages installed (from the local source, which we just built), the runtimes folder only contains the CPU backends.

I'm using the following command to build/publish my application:
dotnet publish -c Release -r linux-x64 --self-contained