Remove DP #16748

justusschock · 2023-02-14T10:11:25Z

What does this PR do?

Removes DataParallel as it is discouraged by PyTorch and requires a lot of workarounds due to a different data-flow.

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @justusschock @awaelchli @Borda

github-actions · 2023-02-14T10:11:59Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/data_parallel.py, src/lightning/pytorch/strategies/__init__.py, src/lightning/pytorch/strategies/dp.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/models/test_amp.py, tests/tests_pytorch/models/test_gpu.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/overrides/test_data_parallel.py, tests/tests_pytorch/strategies/test_common.py, tests/tests_pytorch/strategies/test_dp.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/logging_/test_logger_connector.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_config_validator.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_compile.py, tests/tests_pytorch/utilities/test_dtype_device_mixin.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/data_parallel.py, src/lightning/pytorch/strategies/__init__.py, src/lightning/pytorch/strategies/dp.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/models/test_amp.py, tests/tests_pytorch/models/test_gpu.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/overrides/test_data_parallel.py, tests/tests_pytorch/strategies/test_common.py, tests/tests_pytorch/strategies/test_dp.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/logging_/test_logger_connector.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_config_validator.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_compile.py, tests/tests_pytorch/utilities/test_dtype_device_mixin.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/data_parallel.py, src/lightning/pytorch/strategies/__init__.py, src/lightning/pytorch/strategies/dp.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/models/test_amp.py, tests/tests_pytorch/models/test_gpu.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/overrides/test_data_parallel.py, tests/tests_pytorch/strategies/test_common.py, tests/tests_pytorch/strategies/test_dp.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/logging_/test_logger_connector.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_config_validator.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_compile.py, tests/tests_pytorch/utilities/test_dtype_device_mixin.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/data_parallel.py, src/lightning/pytorch/strategies/__init__.py, src/lightning/pytorch/strategies/dp.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/models/test_amp.py, tests/tests_pytorch/models/test_gpu.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/overrides/test_data_parallel.py, tests/tests_pytorch/strategies/test_common.py, tests/tests_pytorch/strategies/test_dp.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/logging_/test_logger_connector.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_config_validator.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_compile.py, tests/tests_pytorch/utilities/test_dtype_device_mixin.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/data_parallel.py, src/lightning/pytorch/strategies/__init__.py, src/lightning/pytorch/strategies/dp.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, docs/source-pytorch/accelerators/gpu_faq.rst, docs/source-pytorch/accelerators/gpu_intermediate.rst, docs/source-pytorch/api_references.rst, docs/source-pytorch/common/lightning_module.rst, docs/source-pytorch/extensions/strategy.rst, docs/source-pytorch/guides/speed.rst.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/data_parallel.py, src/lightning/pytorch/strategies/__init__.py, src/lightning/pytorch/strategies/dp.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/data_parallel.py, src/lightning/pytorch/strategies/__init__.py, src/lightning/pytorch/strategies/dp.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

carmocca

Pushed some missed removals

docs/source-pytorch/guides/speed.rst

src/lightning/pytorch/CHANGELOG.md

tests/tests_pytorch/utilities/test_compile.py

for more information, see https://pre-commit.ci

docs/source-pytorch/accelerators/gpu_intermediate.rst

justusschock added 4 commits February 14, 2023 10:50

update docs

5c861c1

update tests

0bb56f4

remove usage from source code

f64bfde

remove dp tests

046cc76

justusschock requested review from awaelchli, carmocca, williamFalcon, edenlightning, lantiga and Borda as code owners February 14, 2023 10:11

github-actions bot added the pl Generic label for PyTorch Lightning package label Feb 14, 2023

pre-commit

36ce35b

justusschock changed the title ~~update docs~~ Remove DP Feb 14, 2023

justusschock added the strategy: dp (removed in pl) DataParallel label Feb 14, 2023

justusschock added this to the 2.0 milestone Feb 14, 2023

justusschock and others added 4 commits February 14, 2023 11:17

add changelog

57f3e5a

update test to use normal clf model

fe53376

remaining test occurences

23a203c

[pre-commit.ci] auto fixes from pre-commit.com hooks

4c7d9d8

for more information, see https://pre-commit.ci

awaelchli added the breaking change Includes a breaking change label Feb 14, 2023

awaelchli approved these changes Feb 14, 2023

View reviewed changes

mergify bot added the has conflicts label Feb 14, 2023

carmocca added 2 commits February 14, 2023 14:14

more removals

c185eb2

More removals

8893d31

carmocca approved these changes Feb 14, 2023

View reviewed changes

docs/source-pytorch/guides/speed.rst Outdated Show resolved Hide resolved

src/lightning/pytorch/CHANGELOG.md Outdated Show resolved Hide resolved

tests/tests_pytorch/utilities/test_compile.py Show resolved Hide resolved

justusschock and others added 2 commits February 15, 2023 11:33

address reviewer requests

c6c7623

Merge branch 'master' into 2.0/dp

b4b90e5

mergify bot added ready PRs ready to be merged and removed has conflicts labels Feb 15, 2023

mergify bot added ready PRs ready to be merged and removed ready PRs ready to be merged labels Feb 15, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

afa1c0c

for more information, see https://pre-commit.ci

awaelchli approved these changes Feb 16, 2023

View reviewed changes

docs/source-pytorch/accelerators/gpu_intermediate.rst Outdated Show resolved Hide resolved

Update docs/source-pytorch/accelerators/gpu_intermediate.rst

0c05a99

awaelchli enabled auto-merge (squash) February 16, 2023 04:50

awaelchli merged commit 47c69cd into master Feb 16, 2023

awaelchli deleted the 2.0/dp branch February 16, 2023 05:12

This was referenced Feb 17, 2023

Remove the *_step_end hooks #16791

Merged

AMP not used for training_step_end() #8387

Closed

zhiqiangdon mentioned this pull request Aug 8, 2023

[AutoMM] Support muti-GPU inference with ddp autogluon/autogluon#3445

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Remove DP #16748

Remove DP #16748

Uh oh!

justusschock commented Feb 14, 2023 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Feb 14, 2023 •

edited

Loading

Uh oh!

carmocca left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Remove DP #16748

Remove DP #16748

Uh oh!

Conversation

justusschock commented Feb 14, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

PR review

Uh oh!

github-actions bot commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚡ Required checks status: All passing 🟢

Groups summary

Uh oh!

carmocca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

justusschock commented Feb 14, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Feb 14, 2023 •

edited

Loading