Auto_select_gpus always choose gpus [0,1] when devices=2, strategy = dp, accelarator = 'auto'

[allanchan339]

🐛 Bug

I have tried to use auto_select_gpus in the server to train multiple models concurrently. The first model already occupied 13GB VRAM and 6GB VRAM in GPUs [0,1] respectively. However, the expected behaviors that select different GPUs didn't happen.

Environment before launching BoringModel

To Reproduce

The following BoringModel is used to test the auto_select_gpus. Launch the following codes twice in separate terminals. When the second model is launched, we may observe that the GPUs [0,1] is still being selected.

The first line of output indicate the GPUs [0,1] is still being selected.

import os

import torch
from torch.utils.data import DataLoader, Dataset

from pytorch_lightning import LightningModule, Trainer


class RandomDataset(Dataset):
    def __init__(self, size, num_samples):
        self.len = num_samples
        self.data = torch.randn(num_samples, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


num_samples = 10000


class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)


def run():
    train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    val_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    test_data = DataLoader(RandomDataset(32, 64), batch_size=2)

    model = BoringModel()
    trainer = Trainer(
        default_root_dir=os.getcwd(),
        min_epochs=10000,
        max_epochs=20000,
        enable_model_summary=False,
        devices=2,
        auto_select_gpus=True,
        strategy='dp',
        accelerator='auto'
    )
    trainer.fit(model, train_dataloaders=train_data, val_dataloaders=val_data)
    trainer.test(model, dataloaders=test_data)


run()

The BoringModel used

Expected behavior

The second training code, BoringModel, will select other unused GPUs instead of occupied GPUs [0,1]

Environment

• CUDA:
  - GPU:
  - NVIDIA GeForce RTX 3090
  - NVIDIA GeForce RTX 3090
  - NVIDIA GeForce RTX 3090
  - NVIDIA GeForce RTX 3090
  - NVIDIA GeForce RTX 3090
  - NVIDIA GeForce RTX 3090
  - available: True
  - version: 11.2
• Packages:
  - numpy: 1.22.3
  - pyTorch_debug: False
  - pyTorch_version: 1.10.2
  - pytorch-lightning: 1.6.3
  - tqdm: 4.62.3
• System:
  - OS: Linux
  - architecture:
  - 64bit
  - ELF
  - processor: x86_64
  - python: 3.9.10
  - version: Extend CI #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022

Additional context

cc @tchaton @rohitgr7 @justusschock @kaushikb11 @awaelchli @akihironitta

[awaelchli]

@allanchan339

The second training code, BoringModel, will select other unused GPUs instead of occupied GPUs [0,1]

The current behavior is the opposite, intentionally.

Auto-select is only really useful if GPUs are in exclusive mode. If your GPUs are not in exclusive mode, it means multiple processes are allowed to occupy one GPU which is why Lightning will select them even if there is already a boring model running. If your GPU supports it, you can set it to exclusive mode like so: nvidia-smi -i [index of gpu] -c 1. Then, only one process is allowed to occupy. With auto_select_gpus=True Lightning will then find the one where it can run.

Personal note: I don't like that we have this flag on the trainer. The functionality only makes sense if you set the GPU total count and it should be turned on by default anyway. There is no reason to have it False. I think it only confuses users.

[allanchan339]

@awaelchli
Thank you for your reply. I am confused about the selection criteria.
The current selection criteria are decided by exclusive mode + scanning.
Why not select a GPU by the lowest VRAM occupied instead?

[justusschock]

@allanchan339 The lowest VRAM occupation does not always correlate with the lowest load for the CUDA-/Tensor-Cores.

Meaning that you could have a GPU using only 1 GB of VRAM but the cores being 100% utilised, while another GPU using 10GB VRAM is on average 20% utilised meaning that using the GPU with lower utilisation would slow done the training unnecessarily.

The issue with measuring utilisation is, that this can't really be done beforehand as this can really fluctuate a lot (depending of data loading bottlenecks etc.) and may also drastically change once your model is running on said GPU (to the good and the bad).

This is why we said, we will use the first GPUs available, since that one is at least consistent behaviour.

[allanchan339]

@justusschock
Thank you for pointing out my blind spot in GPU selection and the criteria to select a GPU. Unfortunately, due to policy, I cannot set a GPU to exclusive mode. the least VARM selection method is still attractive (for my case with 1 node multiple GPU).

Finally, I wrote a naive code to select the least VRAM GPU and input as parameter devices=find_gpus(2). Thank you for your help and explanation.

[FutureWithoutEnding]

What strategy do you use to choose the GPU? @allanchan339