Separate training/validation progress bars

[ZhaofengWu]

🚀 Feature

Currently, there is a global progress bar for training + validation, and a validation progress bar during validation. It would be very helpful to have a separate training-only progress bar.

Motivation

1. A global progress bar does not accurately estimate the time remaining because training and validation have different speed. A training-only progress bar at least gives us an unbiased estimate of the time remaining for the training stage.
2. The n_total in the global training bar considers both training and validation batches. Having a training-only progress bar that displays the number of training batches is helpful for calculating gradient accumulation, distributed training nodes, etc.

Pitch

An option to either replace the global progress bar with a training-only progress bar, or to add a training-only progress bar.

cc @Borda @kaushikb11 @rohitgr7 @awaelchli @carmocca

[rohitgr7]

this behavior has been there since the beginning, although I have seen several issues where users are confused around the total shown in the main progress bar.

A global progress bar does not accurately estimate the time remaining because training and validation have different speed.

very valid point. But one thing to note is that we show the version, metrics, etc only on the main progress bar.

Also since validation is shown on its own progress bar, the validation progress on the main progress bar is somewhat irrelevant.

A user can do this by slightly modifying the tqdm progress bar too, but not super convenient.

[ZhaofengWu]

But one thing to note is that we show the version, metrics, etc only on the main progress bar.

Some of these could be moved to the training progress bar, while some might make more sense to be displayed alongside the validation bar (e.g. metrics).

Also since validation is shown on its own progress bar, the validation progress on the main progress bar is somewhat irrelevant.

Suppose a user knows how long the validation would take (not a crazy assumption, since it's typically pretty fast), a training progress bar would allow an easy estimation of the time remaining in the epoch (by adding the ETA on the progress bar and the validation time). It's a lot more difficult to calculate the time remaining with the current setup.

A user can do this by slightly modifying the tqdm progress bar too, but not super convenient.

This is exactly my point. It's ok if lightning wants to keep the current behavior as the default, but it'd be great if there's an easy way to achieve what I described. Many people that I know have been confused by the behavior of the main progress bar.

[carmocca]

It's a highly requested feature. I myself wrote one when I found the project

[arunpatro-meesho]

+1 really need this feature.

[weicheng113]

+1 really need this feature too. It is confusing at the moment. Each time I spend too much time to twist it.