Multiple endpoints roadmap #492
Description
🚀 Support for Multiple Endpoints in LitServe
We are introducing support for multiple endpoints in LitServe, which will require minor architectural adjustments to better separate concerns between the core components.
✨ Proposed Usage
Here’s an example of what the new API could look like:
api1 = VisionEncoderAPI(batch_size=4, api_path="/ap1")
api2 = TextEncoderAPI(batch_size=8, api_path="/path2)
def resize(...):
...
server = LitServer([
api1,
api2,
{"/random": resize}
])
server.run()🔍 Design Goals
To cleanly support this functionality, we need to ensure that responsibilities are clearly divided between the two main entities:
LitAPI
- Manages properties that pertain to the logic of individual APIs, such as:
batch_sizeloopstreamapi_path
LitServer
- Handles server-level configuration, such as:
- Endpoint routing
Shared Responsibility (LitAPI + LitServer)
- Callback handling
- Streaming functionality
📌 Action Items
In PR #468, we successfully moved the batch_size parameter from LitServer to LitAPI to align with this separation of concerns. We need to apply a similar refactoring for other attributes:
- Move
loopfromLitServertoLitAPI- move stream, endpoint path, loop to LitAPI initialization #512 - Move
streamargument from LitServer to LitAPI - move stream, endpoint path, loop to LitAPI initialization #512 - Move
api_path- move stream, endpoint path, loop to LitAPI initialization #512 - Remove
LitServerargument fromLitSpec.setup. Connect the queues and response buffer with a connector instead. - Add multiple LitAPI support - Support multiple LitAPIs for inference process and endpoints #513
- Allow streaming and non-streaming together. - Support stream with non-stream LitAPIs #518
- Scale multiple APIs separately as described here.
- Multiple endpoints with a single LitAPI (shared setup)