[BUG] A bug while fine-tuning the model by iteratively training and evaluating using a sliding time window #783
Description
Bug description
I find out that there is no label in valid.parquet.
Steps/Code to reproduce bug
While I m running this code:
start_time_window_index = 1
final_time_window_index = 4
for time_index in range(start_time_window_index, final_time_window_index):
# Set data
time_index_train = time_index
time_index_eval = time_index + 1
train_paths = glob.glob(os.path.join(OUTPUT_DIR, f"{time_index_train}/train.parquet"))
eval_paths = glob.glob(os.path.join(OUTPUT_DIR, f"{time_index_eval}/valid.parquet"))
# Train on day related to time_index
print('*'20)
print("Launch training for day %s are:" %time_index)
print(''20 + '\n')
trainer.train_dataset_or_path = train_paths
trainer.reset_lr_scheduler()
trainer.train()
trainer.state.global_step +=1
# Evaluate on the following day
trainer.eval_dataset_or_path = eval_paths
train_metrics = trainer.evaluate(metric_key_prefix='eval')
print(''20)
print("Eval results for day %s are:\t" %time_index_eval)
print('\n' + ''*20 + '\n')
for key in sorted(train_metrics.keys()):
print(" %s = %s" % (key, str(train_metrics[key])))
wipe_memory()
the error appear:
Launch training for day 1 are:
/usr/local/lib/python3.10/dist-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set no_deprecation_warning=True to disable this warning
warnings.warn(
{'train_runtime': 4.0234, 'train_samples_per_second': 3817.691, 'train_steps_per_second': 14.913, 'train_loss': 10.525657145182292, 'epoch': 60.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:04<00:00, 14.92it/s]
TrainOutput(global_step=60, training_loss=10.525657145182292, metrics={'train_runtime': 4.0234, 'train_samples_per_second': 3817.691, 'train_steps_per_second': 14.913, 'total_flos': 0.0, 'train_loss': 10.525657145182292})
Traceback (most recent call last):
File "", line 17, in
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2932, in evaluate
output = eval_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/trainer.py", line 515, in evaluation_loop
metrics_results_detailed = model.calculate_metrics(preds, labels)
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/base.py", line 616, in calculate_metrics
head.calculate_metrics(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/base.py", line 453, in calculate_metrics
task.calculate_metrics(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/prediction_task.py", line 489, in calculate_metrics
result = metric(predictions, targets)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 301, in forward
self._forward_cache = self._forward_full_state_update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 316, in _forward_full_state_update
self.update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 465, in wrapped_func
update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/ranking_metric.py", line 56, in update
metric = self._metric(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/ranking_metric.py", line 137, in _metric
if rel_indices.shape[0] > 0:
IndexError: tuple index out of range
Expected behavior
I expected there have label for evaluation
Environment details
- Transformers4Rec version: 23.12
- Platform:Docker
- Python version:3.10
- Huggingface Transformers version:4.27.1
- PyTorch version (GPU?):2.1.0a0+4136153
- Tensorflow version (GPU?):