Trainer: auto default #16847

carmocca · 2023-02-23T00:34:38Z

What does this PR do?

Fixes #10606

Follow-up questions:

CUDA CI only runs the "min_cuda_gpus=N" tests. This helped greatly reduce the CI load in the past. Since we default to "auto" now, should we have CUDA CI run all tests?
The same question above applies to HPU, IPU, TPU, ...

cc @justusschock @awaelchli @Borda @carmocca

github-actions · 2023-02-23T00:35:06Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py, src/lightning/pytorch/trainer/trainer.py, tests/tests_pytorch/accelerators/test_gpu.py, tests/tests_pytorch/accelerators/test_tpu.py, tests/tests_pytorch/callbacks/test_early_stopping.py, tests/tests_pytorch/callbacks/test_pruning.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/models/test_amp.py, tests/tests_pytorch/strategies/test_ddp_spawn_strategy.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/logging_/test_train_loop_logging.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py, src/lightning/pytorch/trainer/trainer.py, tests/tests_pytorch/accelerators/test_gpu.py, tests/tests_pytorch/accelerators/test_tpu.py, tests/tests_pytorch/callbacks/test_early_stopping.py, tests/tests_pytorch/callbacks/test_pruning.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/models/test_amp.py, tests/tests_pytorch/strategies/test_ddp_spawn_strategy.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/logging_/test_train_loop_logging.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py, src/lightning/pytorch/trainer/trainer.py, tests/tests_pytorch/accelerators/test_gpu.py, tests/tests_pytorch/accelerators/test_tpu.py, tests/tests_pytorch/callbacks/test_early_stopping.py, tests/tests_pytorch/callbacks/test_pruning.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/models/test_amp.py, tests/tests_pytorch/strategies/test_ddp_spawn_strategy.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/logging_/test_train_loop_logging.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py, src/lightning/pytorch/trainer/trainer.py, tests/tests_pytorch/accelerators/test_gpu.py, tests/tests_pytorch/accelerators/test_tpu.py, tests/tests_pytorch/callbacks/test_early_stopping.py, tests/tests_pytorch/callbacks/test_pruning.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/models/test_amp.py, tests/tests_pytorch/strategies/test_ddp_spawn_strategy.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/logging_/test_train_loop_logging.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py, src/lightning/pytorch/trainer/trainer.py, docs/source-pytorch/accelerators/gpu_basic.rst, docs/source-pytorch/accelerators/hpu_basic.rst, docs/source-pytorch/accelerators/ipu_basic.rst, docs/source-pytorch/accelerators/tpu_basic.rst, docs/source-pytorch/common/trainer.rst.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py, src/lightning/pytorch/trainer/trainer.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/trainer/connectors/accelerator_connector.py, src/lightning/pytorch/trainer/trainer.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

docs/source-pytorch/accelerators/hpu_basic.rst

This reverts commit c3dd282.

justusschock · 2023-02-23T14:20:21Z

CUDA CI only runs the "min_cuda_gpus=N" tests. This helped greatly reduce the CI load in the past. Since we default to "auto" now, should we have CUDA CI run all tests?

I wouldn't do this. I think the current way would be preferred as other wise CUDA CI might become a bottleneck.

The same question above applies to HPU, IPU, TPU, ...

They are already fragile enough so I'd prefer to run as little tests as possible there.

docs/source-pytorch/accelerators/hpu_basic.rst

docs/source-pytorch/accelerators/tpu_basic.rst

tests/tests_pytorch/strategies/test_ddp_strategy.py

tests/tests_pytorch/trainer/test_trainer.py

carmocca · 2023-02-23T16:10:41Z

I added one conenctor test that asserts "auto" under each accelerator availability. Just as the one in #16842

for more information, see https://pre-commit.ci

commit autogluon@dd96a19 forces a minimum version bump of pytorch-lightning to 2.0.0 ref Lightning-AI/pytorch-lightning#16847

commit dd96a19 forces a minimum version bump of pytorch-lightning to 2.0.0 ref Lightning-AI/pytorch-lightning#16847

Trainer: auto default

d190fca

carmocca added refactor breaking change Includes a breaking change trainer: argument pl Generic label for PyTorch Lightning package labels Feb 23, 2023

carmocca added this to the 2.0 milestone Feb 23, 2023

carmocca requested review from williamFalcon and tchaton as code owners February 23, 2023 00:34

carmocca self-assigned this Feb 23, 2023

carmocca requested review from awaelchli and justusschock as code owners February 23, 2023 00:34

carmocca added 4 commits February 23, 2023 01:48

Fixes

2dcf3ed

Fix tests

ee971eb

Docs

ab69880

Hang

5d2a0e2

carmocca requested review from edenlightning, lantiga and Borda as code owners February 23, 2023 01:33

carmocca commented Feb 23, 2023

View reviewed changes

docs/source-pytorch/accelerators/hpu_basic.rst Show resolved Hide resolved

Debug standalone

c3dd282

carmocca requested a review from akihironitta as a code owner February 23, 2023 11:53

github-actions bot added the ci Continuous Integration label Feb 23, 2023

carmocca added 2 commits February 23, 2023 13:43

Fix deepspeed test

59d243f

Revert "Debug standalone"

f0a64b4

This reverts commit c3dd282.

github-actions bot removed the ci Continuous Integration label Feb 23, 2023

CHANGELOG

ff4d15d

carmocca changed the title ~~[WIP] Trainer: auto default~~ Trainer: auto default Feb 23, 2023

justusschock approved these changes Feb 23, 2023

View reviewed changes

awaelchli reviewed Feb 23, 2023

View reviewed changes

carmocca added 4 commits February 23, 2023 16:09

Revamp docs

bd1dd97

Explicit

5c7b227

Mps back

0555fd3

Deepspeed test

da57da2

awaelchli approved these changes Feb 23, 2023

View reviewed changes

mergify bot added the ready PRs ready to be merged label Feb 23, 2023

justusschock and others added 2 commits February 23, 2023 16:34

Merge branch 'master' into feat/auto-default

f8454a7

Super test and docs fixes

f8589bd

carmocca and others added 3 commits February 23, 2023 17:45

Fix tests

385f64f

Update test_accelerator_connector.py

4faec22

[pre-commit.ci] auto fixes from pre-commit.com hooks

a498524

for more information, see https://pre-commit.ci

carmocca merged commit 0130273 into master Feb 23, 2023

carmocca deleted the feat/auto-default branch February 23, 2023 17:42

ddelange added a commit to ddelange/autogluon that referenced this pull request May 4, 2023

Bump pytorch-lightning

6a0387e

commit autogluon@dd96a19 forces a minimum version bump of pytorch-lightning to 2.0.0 ref Lightning-AI/pytorch-lightning#16847

yinweisu pushed a commit to autogluon/autogluon that referenced this pull request May 8, 2023

Bump pytorch-lightning

ab00f18

commit dd96a19 forces a minimum version bump of pytorch-lightning to 2.0.0 ref Lightning-AI/pytorch-lightning#16847

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Trainer: auto default #16847

Trainer: auto default #16847

Uh oh!

carmocca commented Feb 23, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Feb 23, 2023 •

edited

Loading

Uh oh!

Uh oh!

justusschock commented Feb 23, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carmocca commented Feb 23, 2023

Uh oh!

Uh oh!

Trainer: auto default #16847

Trainer: auto default #16847

Uh oh!

Conversation

carmocca commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

github-actions bot commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚡ Required checks status: All passing 🟢

Groups summary

Uh oh!

Uh oh!

justusschock commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carmocca commented Feb 23, 2023

Uh oh!

Uh oh!

carmocca commented Feb 23, 2023 •

edited

Loading

github-actions bot commented Feb 23, 2023 •

edited

Loading

justusschock commented Feb 23, 2023 •

edited

Loading