Separate the Gradient Accumulation Scheduler from Trainer #16729

awaelchli · 2023-02-11T13:38:18Z

What does this PR do?

Removes support for passing a scheduling dictionary to Trainer(accumulate_grad_batches=...). The same functionality can still be achieved by simply passing the callback in. This simplifies the Trainer and the overall validation logic. Furthermore, it makes the callback self-contained and leaves open the possibility to move it out of the lightning package (not decided).

Before:

trainer = Trainer(accumulate_grad_batches={"1": 5, "10": 3})

Now:

from lightning.pytorch.callbacks import GradientAccumulationScheduler
trainer = Trainer(callbacks=GradientAccumulationScheduler({"1": 5, "10": 3}))

This refactor and breaking change to the Trainer API is part of the Trainer 2.0 simplification effort.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @justusschock @awaelchli @Borda

for more information, see https://pre-commit.ci

src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py

tests/tests_pytorch/accelerators/test_ipu.py

tests/tests_pytorch/callbacks/test_gradient_accumulation_scheduler.py

for more information, see https://pre-commit.ci

…into removal/accumulation-scheduler

github-actions · 2023-02-11T14:41:35Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py, src/lightning/pytorch/callbacks/stochastic_weight_avg.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/strategies/colossalai.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/ipu.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/callback_connector.py, src/lightning/pytorch/trainer/trainer.py, tests/tests_pytorch/accelerators/test_ipu.py, tests/tests_pytorch/callbacks/test_gradient_accumulation_scheduler.py, tests/tests_pytorch/strategies/test_colossalai.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/trainer/connectors/test_callback_connector.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py, src/lightning/pytorch/callbacks/stochastic_weight_avg.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/strategies/colossalai.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/ipu.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/callback_connector.py, src/lightning/pytorch/trainer/trainer.py, tests/tests_pytorch/accelerators/test_ipu.py, tests/tests_pytorch/callbacks/test_gradient_accumulation_scheduler.py, tests/tests_pytorch/strategies/test_colossalai.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/trainer/connectors/test_callback_connector.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py, src/lightning/pytorch/callbacks/stochastic_weight_avg.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/strategies/colossalai.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/ipu.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/callback_connector.py, src/lightning/pytorch/trainer/trainer.py, tests/tests_pytorch/accelerators/test_ipu.py, tests/tests_pytorch/callbacks/test_gradient_accumulation_scheduler.py, tests/tests_pytorch/strategies/test_colossalai.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/trainer/connectors/test_callback_connector.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py, src/lightning/pytorch/callbacks/stochastic_weight_avg.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/strategies/colossalai.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/ipu.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/callback_connector.py, src/lightning/pytorch/trainer/trainer.py, tests/tests_pytorch/accelerators/test_ipu.py, tests/tests_pytorch/callbacks/test_gradient_accumulation_scheduler.py, tests/tests_pytorch/strategies/test_colossalai.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/trainer/connectors/test_callback_connector.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py, src/lightning/pytorch/callbacks/stochastic_weight_avg.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/strategies/colossalai.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/ipu.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/callback_connector.py, src/lightning/pytorch/trainer/trainer.py, docs/source-pytorch/common/gradient_accumulation.rst, docs/source-pytorch/common/optimization.rst, docs/source-pytorch/common/trainer.rst.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py, src/lightning/pytorch/callbacks/stochastic_weight_avg.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/strategies/colossalai.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/ipu.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/callback_connector.py, src/lightning/pytorch/trainer/trainer.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py, src/lightning/pytorch/callbacks/stochastic_weight_avg.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/strategies/colossalai.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/ipu.py, src/lightning/pytorch/trainer/configuration_validator.py, src/lightning/pytorch/trainer/connectors/callback_connector.py, src/lightning/pytorch/trainer/trainer.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

docs/source-pytorch/common/trainer.rst

src/lightning/pytorch/loops/fit_loop.py

src/lightning/pytorch/trainer/connectors/callback_connector.py

src/lightning/pytorch/trainer/trainer.py

Co-authored-by: Carlos Mocholí <[email protected]>

…into removal/accumulation-scheduler

for more information, see https://pre-commit.ci

awaelchli added 4 commits February 11, 2023 14:02

removal

78871f2

simplify

670a460

tests

73752b2

test

6c14648

github-actions bot added the pl Generic label for PyTorch Lightning package label Feb 11, 2023

pre-commit-ci bot and others added 2 commits February 11, 2023 13:39

[pre-commit.ci] auto fixes from pre-commit.com hooks

e036545

for more information, see https://pre-commit.ci

must be int

aebecd9

awaelchli added refactor breaking change Includes a breaking change callback: gradient accumulation fun Staff contributions outside working hours - to differentiate from the "community" label labels Feb 11, 2023

awaelchli added this to the 2.0 milestone Feb 11, 2023

awaelchli self-assigned this Feb 11, 2023

awaelchli commented Feb 11, 2023

View reviewed changes

src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py Show resolved Hide resolved

awaelchli commented Feb 11, 2023

View reviewed changes

tests/tests_pytorch/accelerators/test_ipu.py Show resolved Hide resolved

awaelchli commented Feb 11, 2023

View reviewed changes

tests/tests_pytorch/callbacks/test_gradient_accumulation_scheduler.py Show resolved Hide resolved

awaelchli and others added 5 commits February 11, 2023 15:21

test for untested warning

7388e93

changelog

5fe8be1

[pre-commit.ci] auto fixes from pre-commit.com hooks

a5165ff

for more information, see https://pre-commit.ci

update docs

2937982

Merge remote-tracking branch 'origin/removal/accumulation-scheduler' …

5eb9856

…into removal/accumulation-scheduler

awaelchli marked this pull request as ready for review February 11, 2023 14:41

awaelchli requested review from williamFalcon, tchaton, carmocca, justusschock, edenlightning, lantiga and Borda as code owners February 11, 2023 14:41

awaelchli and others added 2 commits February 11, 2023 15:44

update tests

1100d0a

[pre-commit.ci] auto fixes from pre-commit.com hooks

a8f46e3

for more information, see https://pre-commit.ci

mergify bot added the has conflicts label Feb 11, 2023

carmocca reviewed Feb 12, 2023

View reviewed changes

awaelchli and others added 4 commits February 12, 2023 19:07

Update docs/source-pytorch/common/trainer.rst

d13c2b2

Co-authored-by: Carlos Mocholí <[email protected]>

add test for arg + callback conflict

23752b0

Merge remote-tracking branch 'origin/removal/accumulation-scheduler' …

3d16a97

…into removal/accumulation-scheduler

remove redundant test

7141ef6

carmocca approved these changes Feb 13, 2023

View reviewed changes

Borda approved these changes Feb 13, 2023

View reviewed changes

Merge branch 'master' into removal/accumulation-scheduler

a723e93

Borda enabled auto-merge (squash) February 13, 2023 17:58

mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Feb 13, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

2211fac

for more information, see https://pre-commit.ci

Borda merged commit 67c09e3 into master Feb 13, 2023

Borda deleted the removal/accumulation-scheduler branch February 13, 2023 20:15

awaelchli mentioned this pull request Feb 15, 2023

Replace ColossalAIStrategy with external implementation #16757

Merged

3 tasks

carmocca mentioned this pull request Mar 14, 2023

Remove outdated trainer argument videos #17071

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Separate the Gradient Accumulation Scheduler from Trainer #16729

Separate the Gradient Accumulation Scheduler from Trainer #16729

Uh oh!

awaelchli commented Feb 11, 2023 •

edited by github-actions bot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 11, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Separate the Gradient Accumulation Scheduler from Trainer #16729

Separate the Gradient Accumulation Scheduler from Trainer #16729

Uh oh!

Conversation

awaelchli commented Feb 11, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

PR review

Did you have fun?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚡ Required checks status: All passing 🟢

Groups summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

awaelchli commented Feb 11, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Feb 11, 2023 •

edited

Loading