-
Notifications
You must be signed in to change notification settings - Fork 3.6k
[App] Introduce auto scaler #15769
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
[App] Introduce auto scaler #15769
Changes from all commits
Commits
Show all changes
127 commits
Select commit
Hold shift + click to select a range
e0cb13e
Exlucde __pycache__ in setuptools
akihironitta 6e5a030
Merge branch 'master' into bugfix/dont-collect-pycache
akihironitta f162e96
Add load balancer example
akihironitta 324d41d
Merge branch 'master' into bugfix/dont-collect-pycache
akihironitta 0725439
wip
akihironitta 0d67fd2
Merge branch 'bugfix/dont-collect-pycache' into feat/load-balancer-co…
akihironitta 9cc237f
Update example
akihironitta 9594371
rename
akihironitta fc64ceb
remove prints
akihironitta c4b5ac0
_LoadBalancer -> LoadBalancer
akihironitta 4c61501
AutoScaler(work)
akihironitta dc72f1a
change var name
akihironitta 57943ac
remove locust
akihironitta 8aa68d9
Merge branch 'master' into feat/load-balancer-component
akihironitta b6a9918
Update docs
akihironitta 744ddbe
Merge branch 'docs/add-missing-components' into feat/load-balancer-co…
akihironitta 16f7333
include autoscaler in api ref
akihironitta cd9929c
docs typo
akihironitta f33874e
docs typo
akihironitta 12d12b4
docs typo
akihironitta 656b0b6
docs typo
akihironitta a5859f7
remove unused loadtest
akihironitta 1bdf1bc
remove unused device_type
akihironitta 1cb366f
clean up
akihironitta fb4d2e5
clean up
akihironitta c0ba351
clean up
akihironitta 666918b
Add docstring
akihironitta 6f0f43f
type
akihironitta 24cb9c0
Merge branch 'master' into feat/load-balancer-component
akihironitta 9cda544
env vars to args
akihironitta 609eb10
expose an API for users to override to customise autoscaling logic
akihironitta 4e779a1
update example
akihironitta 92737e4
comment
akihironitta 04c3a72
udpate var name
akihironitta 52d240e
fix scale mechanism and clean up
akihironitta 6d18f24
Update exampl
akihironitta 5c5197e
ignore mypy
akihironitta 98d56ad
Add test file
akihironitta b34076e
.
akihironitta dfc5dff
Merge branch 'master' into docs/add-missing-components
akihironitta c139ede
Merge branch 'docs/add-missing-components' into feat/load-balancer-co…
akihironitta 26ca77d
update impl and update tests
akihironitta 5082d44
Merge branch 'master' into feat/load-balancer-component
akihironitta c230254
Update changlog
akihironitta 80e6b7d
.
akihironitta 2aeec1c
revert docs
akihironitta 8091ca9
update test
akihironitta a2bfaed
update state to keep calling 'flow.run()'
akihironitta eb784fe
Add aiohttp to base requirements
akihironitta 9208959
Update docs
akihironitta 15dc3ae
Use deserializer utility
akihironitta 7ffd45a
fake trigger
akihironitta 10627e9
wip: protect /system/* with basic auth
akihironitta f79d16b
read password at runtime
akihironitta ae7f300
Change env var name
akihironitta 8ea25d1
import torch as optional
akihironitta a0fb484
Merge branch 'master' into feat/load-balancer-component
akihironitta 9e66136
Don't overcreate works
akihironitta 94600d7
simplify imports
akihironitta 15dca21
Update example
akihironitta 8d65628
aiohttp
Borda fe5c0f4
Add work_args work_kwargs
akihironitta 77faca5
More docs
akihironitta cf32733
Merge remote-tracking branch 'origin/feat/load-balancer-component' in…
akihironitta 39de0ba
remove FIXME
akihironitta a49766d
Apply Jirka's suggestions
akihironitta 6861444
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 2b95111
clean example device
akihironitta 5af2569
add comment on init threshold value
akihironitta 69ec4c3
bad merge
akihironitta 64d99e4
Merge remote-tracking branch 'origin/feat/load-balancer-component' in…
akihironitta 61433b3
nit: logging format
akihironitta f9debb6
{in,out}put_schema -> {in,out}put_type
akihironitta e534e09
lowercase
akihironitta 87e2882
docs on seconds
akihironitta 7f2731d
process_time -> processing_time
akihironitta a933247
Dont modify work state from flow
akihironitta 5d7d1c3
Update tests
akihironitta 73cf389
worker_url -> endpoint
akihironitta 8840b85
fix exampl
akihironitta b7301e6
Fix default scale logic
akihironitta c8d0c86
Fix default scale logic
akihironitta ad5b8e5
Fix num_pending_works
akihironitta 9a43d31
Update num_pending_works
akihironitta ce4d257
Fix bug creating too many works
akihironitta 9bebb86
Remove up/downscale_threshold args
akihironitta 8b14154
Update example
akihironitta 611077e
Add typing
akihironitta 7a627ac
Merge branch 'master' into feat/load-balancer-component
akihironitta 4e93898
Fix example in docstring
akihironitta 1e42f55
Fix default scale logic
akihironitta 0b6153a
Update src/lightning_app/components/auto_scaler.py
akihironitta 6d677c1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 67bbe49
rename method
akihironitta e524702
rename locvar
akihironitta 2f24422
Add todo
akihironitta a20797d
docs ci
akihironitta a8a8aaa
docs ci
akihironitta 09dfda5
asdfafsdasdf pls docs
akihironitta 11842b0
Apply suggestions from code review
akihironitta 29059a0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] a35b4c8
Merge branch 'master' into feat/load-balancer-component
4285506
.
akihironitta 72a6f13
doc
akihironitta 56ea78b
Update src/lightning_app/components/auto_scaler.py
akihironitta 24983a0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 27431f4
Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"
akihironitta b7dd2c1
Revert "Update src/lightning_app/components/auto_scaler.py"
akihironitta ebcfc51
Merge branch 'master' into feat/load-balancer-component
akihironitta a634446
Remove redefinition
akihironitta 64a1960
Remove load balancer run blocker
akihironitta fba7a3c
raise RuntimeError
akihironitta 4ccc38c
remove has_sent
akihironitta aa1785c
lower the default timeout_batching from 10 to 1
akihironitta 7c09716
remove debug
akihironitta ff2009a
update the default timeout_batching
akihironitta 839734d
.
akihironitta 6a553b9
tighten condition
akihironitta 506e192
fix endpoint
akihironitta 205c0af
typo in runtimeerror cond
akihironitta 6d76b0d
async lock update severs
akihironitta 2233098
add a test
akihironitta 4526496
{in,out}put_type typing
akihironitta 00fed69
Merge branch 'master' into feat/load-balancer-component
tchaton 468b626
Update examples/app_server_with_auto_scaler/app.py
akihironitta 5b2b69f
Update .actions/setup_tools.py
akihironitta 2c41e38
Merge branch 'master' into feat/load-balancer-component
akihironitta File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
from typing import Any, List | ||
|
||
import torch | ||
import torchvision | ||
from pydantic import BaseModel | ||
|
||
import lightning as L | ||
|
||
|
||
class RequestModel(BaseModel): | ||
image: str # bytecode | ||
|
||
|
||
class BatchRequestModel(BaseModel): | ||
inputs: List[RequestModel] | ||
|
||
|
||
class BatchResponse(BaseModel): | ||
outputs: List[Any] | ||
|
||
|
||
class PyTorchServer(L.app.components.PythonServer): | ||
def __init__(self, *args, **kwargs): | ||
super().__init__( | ||
port=L.app.utilities.network.find_free_network_port(), | ||
input_type=BatchRequestModel, | ||
output_type=BatchResponse, | ||
cloud_compute=L.CloudCompute("gpu"), | ||
) | ||
|
||
def setup(self): | ||
self._device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") | ||
self._model = torchvision.models.resnet18(pretrained=True).to(self._device) | ||
|
||
def predict(self, requests: BatchRequestModel): | ||
transforms = torchvision.transforms.Compose( | ||
[ | ||
torchvision.transforms.Resize(224), | ||
torchvision.transforms.ToTensor(), | ||
torchvision.transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), | ||
] | ||
) | ||
images = [] | ||
for request in requests.inputs: | ||
image = L.app.components.serve.types.image.Image.deserialize(request.image) | ||
image = transforms(image).unsqueeze(0) | ||
images.append(image) | ||
images = torch.cat(images) | ||
images = images.to(self._device) | ||
predictions = self._model(images) | ||
results = predictions.argmax(1).cpu().numpy().tolist() | ||
return BatchResponse(outputs=[{"prediction": pred} for pred in results]) | ||
|
||
|
||
class MyAutoScaler(L.app.components.AutoScaler): | ||
def scale(self, replicas: int, metrics: dict) -> int: | ||
"""The default scaling logic that users can override.""" | ||
# scale out if the number of pending requests exceeds max batch size. | ||
max_requests_per_work = self.max_batch_size | ||
pending_requests_per_running_or_pending_work = metrics["pending_requests"] / ( | ||
replicas + metrics["pending_works"] | ||
) | ||
if pending_requests_per_running_or_pending_work >= max_requests_per_work: | ||
return replicas + 1 | ||
|
||
# scale in if the number of pending requests is below 25% of max_requests_per_work | ||
min_requests_per_work = max_requests_per_work * 0.25 | ||
pending_requests_per_running_work = metrics["pending_requests"] / replicas | ||
if pending_requests_per_running_work < min_requests_per_work: | ||
return replicas - 1 | ||
|
||
return replicas | ||
|
||
|
||
app = L.LightningApp( | ||
MyAutoScaler( | ||
PyTorchServer, | ||
min_replicas=2, | ||
max_replicas=4, | ||
autoscale_interval=10, | ||
endpoint="predict", | ||
input_type=RequestModel, | ||
output_type=Any, | ||
timeout_batching=1, | ||
) | ||
) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.