|
1 | | -# Common commands about conda environment |
| 1 | +# Python Backends for LocalAI |
2 | 2 |
|
3 | | -## Create a new empty conda environment |
| 3 | +This directory contains Python-based AI backends for LocalAI, providing support for various AI models and hardware acceleration targets. |
4 | 4 |
|
5 | | -``` |
6 | | -conda create --name <env-name> python=<your version> -y |
| 5 | +## Overview |
7 | 6 |
|
8 | | -conda create --name autogptq python=3.11 -y |
9 | | -``` |
| 7 | +The Python backends use a unified build system based on `libbackend.sh` that provides: |
| 8 | +- **Automatic virtual environment management** with support for both `uv` and `pip` |
| 9 | +- **Hardware-specific dependency installation** (CPU, CUDA, Intel, MLX, etc.) |
| 10 | +- **Portable Python support** for standalone deployments |
| 11 | +- **Consistent backend execution** across different environments |
| 12 | + |
| 13 | +## Available Backends |
| 14 | + |
| 15 | +### Core AI Models |
| 16 | +- **transformers** - Hugging Face Transformers framework (PyTorch-based) |
| 17 | +- **vllm** - High-performance LLM inference engine |
| 18 | +- **mlx** - Apple Silicon optimized ML framework |
| 19 | +- **exllama2** - ExLlama2 quantized models |
| 20 | + |
| 21 | +### Audio & Speech |
| 22 | +- **bark** - Text-to-speech synthesis |
| 23 | +- **coqui** - Coqui TTS models |
| 24 | +- **faster-whisper** - Fast Whisper speech recognition |
| 25 | +- **kitten-tts** - Lightweight TTS |
| 26 | +- **mlx-audio** - Apple Silicon audio processing |
| 27 | +- **chatterbox** - TTS model |
| 28 | +- **kokoro** - TTS models |
| 29 | + |
| 30 | +### Computer Vision |
| 31 | +- **diffusers** - Stable Diffusion and image generation |
| 32 | +- **mlx-vlm** - Vision-language models for Apple Silicon |
| 33 | +- **rfdetr** - Object detection models |
| 34 | + |
| 35 | +### Specialized |
| 36 | + |
| 37 | +- **rerankers** - Text reranking models |
| 38 | + |
| 39 | +## Quick Start |
| 40 | + |
| 41 | +### Prerequisites |
| 42 | +- Python 3.10+ (default: 3.10.18) |
| 43 | +- `uv` package manager (recommended) or `pip` |
| 44 | +- Appropriate hardware drivers for your target (CUDA, Intel, etc.) |
10 | 45 |
|
11 | | -## To activate the environment |
| 46 | +### Installation |
12 | 47 |
|
13 | | -As of conda 4.4 |
| 48 | +Each backend can be installed individually: |
| 49 | + |
| 50 | +```bash |
| 51 | +# Navigate to a specific backend |
| 52 | +cd backend/python/transformers |
| 53 | + |
| 54 | +# Install dependencies |
| 55 | +make transformers |
| 56 | +# or |
| 57 | +bash install.sh |
| 58 | + |
| 59 | +# Run the backend |
| 60 | +make run |
| 61 | +# or |
| 62 | +bash run.sh |
14 | 63 | ``` |
15 | | -conda activate autogptq |
| 64 | + |
| 65 | +### Using the Unified Build System |
| 66 | + |
| 67 | +The `libbackend.sh` script provides consistent commands across all backends: |
| 68 | + |
| 69 | +```bash |
| 70 | +# Source the library in your backend script |
| 71 | +source $(dirname $0)/../common/libbackend.sh |
| 72 | + |
| 73 | +# Install requirements (automatically handles hardware detection) |
| 74 | +installRequirements |
| 75 | + |
| 76 | +# Start the backend server |
| 77 | +startBackend $@ |
| 78 | + |
| 79 | +# Run tests |
| 80 | +runUnittests |
16 | 81 | ``` |
17 | 82 |
|
18 | | -The conda version older than 4.4 |
| 83 | +## Hardware Targets |
| 84 | + |
| 85 | +The build system automatically detects and configures for different hardware: |
19 | 86 |
|
| 87 | +- **CPU** - Standard CPU-only builds |
| 88 | +- **CUDA** - NVIDIA GPU acceleration (supports CUDA 11/12) |
| 89 | +- **Intel** - Intel XPU/GPU optimization |
| 90 | +- **MLX** - Apple Silicon (M1/M2/M3) optimization |
| 91 | +- **HIP** - AMD GPU acceleration |
| 92 | + |
| 93 | +### Target-Specific Requirements |
| 94 | + |
| 95 | +Backends can specify hardware-specific dependencies: |
| 96 | +- `requirements.txt` - Base requirements |
| 97 | +- `requirements-cpu.txt` - CPU-specific packages |
| 98 | +- `requirements-cublas11.txt` - CUDA 11 packages |
| 99 | +- `requirements-cublas12.txt` - CUDA 12 packages |
| 100 | +- `requirements-intel.txt` - Intel-optimized packages |
| 101 | +- `requirements-mps.txt` - Apple Silicon packages |
| 102 | + |
| 103 | +## Configuration Options |
| 104 | + |
| 105 | +### Environment Variables |
| 106 | + |
| 107 | +- `PYTHON_VERSION` - Python version (default: 3.10) |
| 108 | +- `PYTHON_PATCH` - Python patch version (default: 18) |
| 109 | +- `BUILD_TYPE` - Force specific build target |
| 110 | +- `USE_PIP` - Use pip instead of uv (default: false) |
| 111 | +- `PORTABLE_PYTHON` - Enable portable Python builds |
| 112 | +- `LIMIT_TARGETS` - Restrict backend to specific targets |
| 113 | + |
| 114 | +### Example: CUDA 12 Only Backend |
| 115 | + |
| 116 | +```bash |
| 117 | +# In your backend script |
| 118 | +LIMIT_TARGETS="cublas12" |
| 119 | +source $(dirname $0)/../common/libbackend.sh |
20 | 120 | ``` |
21 | | -source activate autogptq |
| 121 | + |
| 122 | +### Example: Intel-Optimized Backend |
| 123 | + |
| 124 | +```bash |
| 125 | +# In your backend script |
| 126 | +LIMIT_TARGETS="intel" |
| 127 | +source $(dirname $0)/../common/libbackend.sh |
22 | 128 | ``` |
23 | 129 |
|
24 | | -## Install the packages to your environment |
| 130 | +## Development |
| 131 | + |
| 132 | +### Adding a New Backend |
25 | 133 |
|
26 | | -Sometimes you need to install the packages from the conda-forge channel |
| 134 | +1. Create a new directory in `backend/python/` |
| 135 | +2. Copy the template structure from `common/template/` |
| 136 | +3. Implement your `backend.py` with the required gRPC interface |
| 137 | +4. Add appropriate requirements files for your target hardware |
| 138 | +5. Use `libbackend.sh` for consistent build and execution |
27 | 139 |
|
28 | | -By using `conda` |
| 140 | +### Testing |
| 141 | + |
| 142 | +```bash |
| 143 | +# Run backend tests |
| 144 | +make test |
| 145 | +# or |
| 146 | +bash test.sh |
29 | 147 | ``` |
30 | | -conda install <your-package-name> |
31 | 148 |
|
32 | | -conda install -c conda-forge <your package-name> |
| 149 | +### Building |
| 150 | + |
| 151 | +```bash |
| 152 | +# Install dependencies |
| 153 | +make <backend-name> |
| 154 | + |
| 155 | +# Clean build artifacts |
| 156 | +make clean |
33 | 157 | ``` |
34 | 158 |
|
35 | | -Or by using `pip` |
| 159 | +## Architecture |
| 160 | + |
| 161 | +Each backend follows a consistent structure: |
36 | 162 | ``` |
37 | | -pip install <your-package-name> |
| 163 | +backend-name/ |
| 164 | +├── backend.py # Main backend implementation |
| 165 | +├── requirements.txt # Base dependencies |
| 166 | +├── requirements-*.txt # Hardware-specific dependencies |
| 167 | +├── install.sh # Installation script |
| 168 | +├── run.sh # Execution script |
| 169 | +├── test.sh # Test script |
| 170 | +├── Makefile # Build targets |
| 171 | +└── test.py # Unit tests |
38 | 172 | ``` |
| 173 | + |
| 174 | +## Troubleshooting |
| 175 | + |
| 176 | +### Common Issues |
| 177 | + |
| 178 | +1. **Missing dependencies**: Ensure all requirements files are properly configured |
| 179 | +2. **Hardware detection**: Check that `BUILD_TYPE` matches your system |
| 180 | +3. **Python version**: Verify Python 3.10+ is available |
| 181 | +4. **Virtual environment**: Use `ensureVenv` to create/activate environments |
| 182 | + |
| 183 | +## Contributing |
| 184 | + |
| 185 | +When adding new backends or modifying existing ones: |
| 186 | +1. Follow the established directory structure |
| 187 | +2. Use `libbackend.sh` for consistent behavior |
| 188 | +3. Include appropriate requirements files for all target hardware |
| 189 | +4. Add comprehensive tests |
| 190 | +5. Update this README if adding new backend types |
0 commit comments