-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Closed
Labels
AutoML.NETAutomating various steps of the machine learning processAutomating various steps of the machine learning process
Milestone
Description
System Information (please complete the following information):
- OS & Version: Windows 11 [Version 10.0.22621.675]
- ML.NET Version: ML.NT v2.0.0-preview.22551.1
- .NET Version: .NET 6.0.9
Describe the bug
LightGBM
trials fail with "bad allocation" error message and they cause OutOfMemoryException
even if SetMaximumMemoryUsageInMegaByte()
is used.
To Reproduce
Steps to reproduce the behavior:
- Create a pipeline that includes LightGBM, for example:
var pipeline =
mlContext.Auto().Featurizer(trainTestData.TrainSet, numericColumns: new[] { "Features" })
.Append(mlContext.Auto().Regression(useFastTree: false, useLbfgs: true, useSdca: true, useFastForest: false, useLgbm: true));
- Create an ML.NET experiment and set its memory limit. I set it to 20 GB, because I usually have ~24 GB available when I start the training.
var experiment = mlContext.Auto().CreateExperiment();
experiment
.SetPipeline(pipeline)
.SetTrainingTimeInSeconds(trainingTimeInSeconds)
.SetRegressionMetric(RegressionMetric.RSquared, labelColumn: "Label")
.SetDataset(trainTestData.TrainSet, trainTestData.TestSet)
.SetMonitor(monitor)
.SetMaximumMemoryUsageInMegaByte(20 * 1024);
- Run the experiment for an hour
- The experiment will end with
OutOfMemoryException
Expected behavior
I would expect the AutoML stops the trial if it exceeds the memory limit and/or there is an exception during its run. It should not about the whole experiment.
Screenshots, Code, Sample Projects
Here are the logs of three runs I had today:
15:23:11.3 info: Bio Balance Detector Body Monitor v0.9.1.3
15:23:11.3 info: (The current UTC time is 2022-11-02 14:23:11)
15:23:11.6 info: Now listening on: https://0.0.0.0:7061
15:23:11.6 info: Now listening on: http://localhost:5061
15:23:11.6 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
15:23:11.6 info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development
15:23:11.6 info: Microsoft.Hosting.Lifetime[0] Content root path: C:\Work\BioBalanceDetector\Software\Source\BBDProto08\BBD.BodyMonitor\BBD.BodyMonitor.API\
15:23:38.1 info: Starting model training for 3600 seconds, using 'BBD_20221102_2022-11-02_MLP14_0p25Hz-6250Hz_IsSession.SegmentedData.Sleep.Level_1098rows.csv' as data source with the 'MLP14' profile.
15:23:51.5 info: Completed Trial # 0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,15 - Duration: 13 seconds
15:23:51.5 info: New Best Trial # 0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,15
15:24:01.2 info: Completed Trial # 1 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression - Metric: 0,05 - Duration: 10 seconds
15:24:11.1 info: Completed Trial # 2 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric: -2,20 - Duration: 10 seconds
15:50:15.5 info: Completed Trial # 3 - Pipeline: ReplaceMissingValues=>Concatenate=>SdcaRegression - Metric: 0,03 - Duration: 1564 seconds
15:50:32.2 info: Completed Trial # 4 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41 - Duration: 17 seconds
15:50:32.2 info: New Best Trial # 4 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41
15:50:44.3 info: Completed Trial # 5 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,16 - Duration: 12 seconds
15:50:50.0 info: Completed Trial # 6 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,40 - Duration: 6 seconds
15:51:15.3 info: Completed Trial # 7 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43 - Duration: 25 seconds
15:51:15.3 info: New Best Trial # 7 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43
15:51:43.3 info: Completed Trial # 8 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43 - Duration: 28 seconds
15:51:43.3 info: New Best Trial # 8 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43
15:51:54.4 info: Completed Trial # 9 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,14 - Duration: 11 seconds
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
17:49:52.6 info: BBD.BodyMonitor.API[0] Bio Balance Detector Body Monitor v0.9.1.3
17:49:52.6 info: BBD.BodyMonitor.API[0] (The current UTC time is 2022-11-02 16:49:52)
17:49:52.9 info: Microsoft.Hosting.Lifetime[14] Now listening on: https://0.0.0.0:7061
17:49:52.9 info: Microsoft.Hosting.Lifetime[14] Now listening on: http://localhost:5061
17:49:52.9 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
17:49:52.9 info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development
17:49:52.9 info: Microsoft.Hosting.Lifetime[0] Content root path: C:\Work\BioBalanceDetector\Software\Source\BBDProto08\BBD.BodyMonitor\BBD.BodyMonitor.API\
17:50:07.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Starting model training for 3600 seconds, using 'BBD_20221102_2022-11-02_MLP14_0p25Hz-6250Hz_IsSession.SegmentedData.Sleep.Level_1098rows.csv' as data source with the 'MLP14' profile.
17:50:08.0 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression
17:50:20.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,15 - Duration: 13 seconds
17:50:20.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] New Best Trial # 0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,15
17:50:20.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 1 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
17:50:30.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 1 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric: -2,20 - Duration: 10 seconds
17:50:30.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 2 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression
17:50:40.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 2 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression - Metric: 0,05 - Duration: 10 seconds
17:50:40.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 3 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression
18:16:49.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 3 - Pipeline: ReplaceMissingValues=>Concatenate=>SdcaRegression - Metric: 0,03 - Duration: 1569 seconds
18:16:49.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 4 - Pipeline: ReplaceMissingValues=>Concatenate=>SdcaRegression
18:17:05.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 4 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41 - Duration: 17 seconds
18:17:05.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] New Best Trial # 4 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41
18:17:05.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 5 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:17:17.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 5 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,21 - Duration: 12 seconds
18:17:17.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 6 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
18:17:22.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 6 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41 - Duration: 5 seconds
18:17:22.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 7 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:17:34.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 7 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,23 - Duration: 12 seconds
18:17:34.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 8 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
18:17:50.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 8 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,36 - Duration: 16 seconds
18:17:50.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 9 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:19:58.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 9 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,28 - Duration: 128 seconds
18:19:58.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 10 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
18:21:04.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 10 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,26 - Duration: 67 seconds
18:21:04.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 11 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
18:21:18.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 11 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,22 - Duration: 14 seconds
18:21:18.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 12 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:21:23.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 12 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,36 - Duration: 5 seconds
18:21:23.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 13 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:21:35.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 13 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,40 - Duration: 12 seconds
18:21:35.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 14 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:21:58.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 14 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43 - Duration: 23 seconds
18:21:58.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] New Best Trial # 14 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43
18:21:58.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 15 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:22:14.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 15 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,27 - Duration: 16 seconds
18:22:14.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 16 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:25:06.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 16 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,28 - Duration: 172 seconds
18:25:06.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 17 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
18:25:11.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 17 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,39 - Duration: 5 seconds
18:25:11.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 18 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:25:16.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 18 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,38 - Duration: 5 seconds
18:25:16.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 19 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:25:22.0 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 19 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,03 - Duration: 6 seconds
18:25:22.0 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 20 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:25:32.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 20 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression - Metric: 0,02 - Duration: 10 seconds
18:25:32.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 21 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression
18:25:37.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 21 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,37 - Duration: 5 seconds
18:25:37.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 22 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
18:26:03.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 22 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,29 - Duration: 26 seconds
18:26:03.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 23 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
18:26:07.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 23 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,04 - Duration: 4 seconds
18:26:07.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 24 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
18:53:28.1 info: BBD.BodyMonitor.API[0] Bio Balance Detector Body Monitor v0.9.1.3
18:53:28.1 info: BBD.BodyMonitor.API[0] (The current UTC time is 2022-11-02 17:53:28)
18:53:28.6 info: Microsoft.Hosting.Lifetime[14] Now listening on: https://0.0.0.0:7061
18:53:28.6 info: Microsoft.Hosting.Lifetime[14] Now listening on: http://localhost:5061
18:53:28.6 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
18:53:28.6 info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development
18:53:28.6 info: Microsoft.Hosting.Lifetime[0] Content root path: C:\Work\BioBalanceDetector\Software\Source\BBDProto08\BBD.BodyMonitor\BBD.BodyMonitor.API\
18:53:44.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Starting model training for 3600 seconds, using 'BBD_20221102_2022-11-02_MLP14_0p25Hz-6250Hz_IsSession.SegmentedData.Sleep.Level_1098rows.csv' as data source with the 'MLP14' profile.
18:53:44.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression
18:53:58.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,15 - Duration: 14 seconds
18:53:58.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] New Best Trial # 0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,15
18:53:58.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 1 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
18:54:08.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 1 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression - Metric: 0,05 - Duration: 10 seconds
18:54:08.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 2 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression
19:19:50.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 2 - Pipeline: ReplaceMissingValues=>Concatenate=>SdcaRegression - Metric: 0,03 - Duration: 1542 seconds
19:19:50.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 3 - Pipeline: ReplaceMissingValues=>Concatenate=>SdcaRegression
19:20:00.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 3 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric: -2,20 - Duration: 10 seconds
19:20:00.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 4 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression
19:20:17.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 4 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41 - Duration: 17 seconds
19:20:17.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] New Best Trial # 4 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41
19:20:17.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 5 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:20:32.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 5 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,25 - Duration: 15 seconds
19:20:32.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 6 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
19:21:19.0 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 6 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,26 - Duration: 47 seconds
19:21:19.0 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 7 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
19:21:41.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 7 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,39 - Duration: 23 seconds
19:21:41.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 8 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:21:46.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 8 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,37 - Duration: 5 seconds
19:21:46.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 9 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:22:04.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 9 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,23 - Duration: 18 seconds
19:22:04.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 10 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
19:22:08.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 10 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41 - Duration: 5 seconds
19:22:08.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 11 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:22:26.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 11 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,31 - Duration: 18 seconds
19:22:26.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 12 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:22:31.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 12 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,36 - Duration: 5 seconds
19:22:31.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 13 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:22:58.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 13 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,35 - Duration: 27 seconds
19:22:58.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 14 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:23:04.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 14 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,40 - Duration: 6 seconds
19:23:04.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 15 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:23:20.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 15 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,24 - Duration: 16 seconds
19:23:20.4 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 16 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:23:37.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 16 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41 - Duration: 17 seconds
19:23:37.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 17 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:23:46.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 17 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,39 - Duration: 9 seconds
19:23:46.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 18 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:24:14.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 18 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43 - Duration: 28 seconds
19:24:14.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] New Best Trial # 18 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43
19:24:14.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 19 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:24:46.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 19 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,43 - Duration: 32 seconds
19:24:46.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 20 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:24:57.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 20 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,41 - Duration: 10 seconds
19:24:57.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 21 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:25:14.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 21 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression - Metric: 0,29 - Duration: 17 seconds
19:25:14.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 22 - Pipeline: ReplaceMissingValues=>Concatenate=>LbfgsPoissonRegressionRegression
19:26:55.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 22 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression - Metric: 0,27 - Duration: 101 seconds
19:26:55.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 23 - Pipeline: ReplaceMissingValues=>Concatenate=>FastForestRegression
19:27:05.0 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial # 23 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression - Metric: 0,13 - Duration: 10 seconds
19:27:05.0 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Running Trial # 24 - Pipeline: ReplaceMissingValues=>Concatenate=>LightGbmRegression
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
Additional context
My project is open-source so both the source and the data are available. Let me know if you need them for testing.
Metadata
Metadata
Assignees
Labels
AutoML.NETAutomating various steps of the machine learning processAutomating various steps of the machine learning process