Allow sideloading backends on filesystem

[arenekosreal]

Is your feature request related to a problem? Please describe.

I am trying to package LocalAI to archlinux, so LocalAI can be installed and run directly on archlinux with its files managed by pacman. But I find it is hard to package those backends like llama-cpp, piper, bark, etc. LocalAI supports installing custom backends by using ocifile protocol, but this requires us using tools like docker/podman/buildah, etc to create an archive, and run command manually to install. And libraries dependencies of tarred backends cannot be tracked easily because backends files are in a tar file, they cannot be checked directly by packaging checking tools.

Describe the solution you'd like

Allow us sideloading backends existed on filesystem automatically seems to be fine to solve this. For example, we can load llama-cpp backend at /path/to/local-ai/backend/cpp/llama-cpp/package/ if required files exist, /path/to/local-ai is the directory contains local-ai binary itself. As you can see, if /path/to/local-ai is where the git repository is, this path is where built backends files are before sending to docker. This means that we can even run built local-ai and those backends directly in LocalAI's git repository, it may be easier and quicker for us to check changes to those backends when developing. But you cannot assume the sideloaded backend directory is writable, because they may be the path like /usr/lib/local-ai/backend/cpp/llama-cpp/package/, only root has permission to modify this path, and this path should be managed by package manager on normal Linux. So the metadata.json may need to be writen to other writable place. My current thought is that we still write metadata.json to ./backends/llama-cpp@sideloaded, but there is a new key named sideload_path with value /path/to/local-ai/backends/cpp/llama-cpp/package/.

Describe alternatives you've considered

N/A

Additional context

This does not mean we should remove ocifile protocol install method. Actually, it behaves very well when we need to install backend in special scenario like without network.

[mudler]

Having an arch package would be great!

good point here - would it make sense to you if we have something among the lines of:

• you build the backends, e.g. manually
• you move the backend to the backends folder (e.g. could just be a system location, e.g. /var/localai/backends)
• you run local-ai backends install --sideload /var/localai/backends/<backend> OR we have something like local-ai backends sideload-install that scans the backends path and configures them.

The problem is not really the location of the metadata, but that, backends aliases won't be supported and I don't like that disconnection. The metadata file is generated out of an installation, and tells additional backend configuration, such as which backend it aliases to.

I'm thinking that we could expose that to the CLI installation command (albeit questionable as users shouldn't be exposed to that, it might just add confusion), or in some other form (e.g. providing the metadata.json file during installation).

@arenekosreal if you are willing to package it, you can actually generate the metadata.json file inside the backend and that would be enough for LocalAI to recognize it. Point is, it would make still more sense from a packaging perspective because the metadata file tells to LocalAI also how to use the backend

[mudler]

@arenekosreal would #5962 solve your issue?

[arenekosreal]

@arenekosreal would #5962 solve your issue?

Well, I think it solves partically, at least we can load backends without creating container archive. But I still prefer loading those backends directly instead installing and running them. Loading directly does not only avoid copying files to save space, but also lets us not need to update those installed backends manually after local-ai is upgraded.

What's more, the packaged local-ai is usually wrapped in a systemd service, we do apply some tweaks in service file, like specifying data directory, use a dynamic user to run process, or even do some isolation work, to keep everything working as safely as possible. This means that running local-ai manually from user may not update files of local-ai instance managed by systemd. So I prefer scanning existing backends and loading them automatically instead letting user install them manually.

If we consider the previous comment, I think generating metadata during packaging is acceptable. local-ai can scan specified directory, which may be given by user or even do not allow customizing, and load scanned backends automatically.

[mudler]

OK I see. Let me come back at this, I think we could find a middle ground here and have LocalAI infer the metadata.json if not present, or automatically generating it. Of course it would miss some metadata if coming from e.g. a gallery, but would still be functional.

[mudler]

Ok, with this #5963 it will be possible to have directly backends without a metadata file.

This allows backends to be recognized, however, it won't support additional things like aliases which are defined during installation. This basically means that when you are packaging the various backends, you have to take care about which one has the proper acceleration and make sure these can't be installed all together to not conflict (for the same backend).

For example, if you are going to build the llama-cpp one for CPU it has to be in the backends directory as llama-cpp, and if you plan to have a version for, let's say, Nvidia, it will still have to be named llama-cpp (if you want to comply with the models in the gallery). I don't know of arch internals, but typically (e.g. in Gentoo) if you plan to have different packages that ships the same file/folders, these shouldn't be installed together (as would overlap) and mark these appropriately with the package manager.

[arenekosreal]

Package manager can set relationship among packages, like conflicting, providing and/or replacing. So this is not a big problem. Anyway, the current solution seems to be a balance between our plans and it seems good enough.

[arenekosreal]

Sorry to bother you again, but I find current implementation may have a point to improve after I tested deeper on newly released local-ai: I have to set LOCALAI_BACKENDS_PATH environment variable to let local-ai load packaged backends, but the path is also used by installing new backends. Which results that if I want to install some backends while keep loading packaged backends, I have to make sure the path is writable by the local-ai process. I suggest using a new environment variable like LOCALAI_UNMANAGED_BACKENDS_PATH to set the path to those packaged backends, and local-ai should only read this path.

P.S.: It seems that local-ai itself will try setting run.sh's permission to make it executable when launching backend, this is also not available on backends managed by package manager. Because those files can be modified by root only.

P.P.S.: I thought a bit more, and I suggest we use packaged backends as a fallback when no installed backend found. This will also implement a "overlay" mechanism. User can install their own backend with same name to override packaged one. You can see firefox's extension loading mechanism as a reference, firefox's extension can be installed by package manager, while packaged extension can also be overriden by user installed one. Also, the Uninstall button should be disabled when the backend is packaged one and a tooltip to explain why should be displayed to user.

[arenekosreal]

@mudler The problems mentioned in previous comment still exist in 3.4.0, please consider it so we can install extra backends while using packaged backends?

[mudler]

@arenekosreal This sounds more like a new feature request, can you open up a new issue please?

I need to give it more thought - generally I would like to keep the code very simple around that area, that's why I'm not really fancying having two paths for the backends but we can find a middle ground here.

have to make sure the path is writable by the local-ai process.

Why this is a problem? Ideally LocalAI should have its own user in the system, where it have access to the models and backends folders as it is meant to manage these two. I understand there are some models that shouldn't be delete-able, this could be treated as well in the code