Skip to content

Commit c0fc15c

Browse files
author
Github Actions
committed
Ravin Kohli: Embedding layer (#91)
1 parent 2aa06dd commit c0fc15c

18 files changed

+458
-88
lines changed

‎refactor_development/.buildinfo‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
# Sphinx build info version 1
22
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
3-
config: 16955af63d37db79a6e43b67a0b42045
3+
config: d4f3d04cddab6e3e314b10c7fdfafae2
44
tags: 645f666f9bcd5a90fca523b33c5a78b7

‎refactor_development/_downloads/306036486863b5329c4111d8adbaac63/example_tabular_regression.ipynb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
"name": "python",
4747
"nbconvert_exporter": "python",
4848
"pygments_lexer": "ipython3",
49-
"version": "3.8.7"
49+
"version": "3.8.8"
5050
}
5151
},
5252
"nbformat": 4,

‎refactor_development/_downloads/a39c0378d911b81ecec47ff0a116e6bf/example_image_classification.ipynb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
"name": "python",
4747
"nbconvert_exporter": "python",
4848
"pygments_lexer": "ipython3",
49-
"version": "3.8.7"
49+
"version": "3.8.8"
5050
}
5151
},
5252
"nbformat": 4,

‎refactor_development/_downloads/bc82bea3a5dd7bdba60b65220891d9e5/examples_python.zip‎

0 Bytes
Binary file not shown.

‎refactor_development/_downloads/c62547d844fd82c936bc377a1da9a504/example_tabular_classification.ipynb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
"name": "python",
4747
"nbconvert_exporter": "python",
4848
"pygments_lexer": "ipython3",
49-
"version": "3.8.7"
49+
"version": "3.8.8"
5050
}
5151
},
5252
"nbformat": 4,

‎refactor_development/_downloads/fb625db3c50d423b1b7881136ffdeec8/examples_jupyter.zip‎

0 Bytes
Binary file not shown.

‎refactor_development/_sources/examples/example_image_classification.rst.txt‎

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Image Classification
4242
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to ../datasets/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz
4343
0.0% 159.1%Extracting ../datasets/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz to ../datasets/FashionMNIST/raw
4444
Processing...
45-
/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/torchvision/datasets/mnist.py:480: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
45+
/opt/hostedtoolcache/Python/3.8.8/x64/lib/python3.8/site-packages/torchvision/datasets/mnist.py:480: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
4646
return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)
4747
Done!
4848
Pipeline CS:
@@ -77,23 +77,22 @@ Image Classification
7777
Pipeline Random Config:
7878
________________________________________
7979
Configuration:
80-
image_augmenter:GaussianBlur:sigma_min, Value: 0.8028082276383658
81-
image_augmenter:GaussianBlur:sigma_offset, Value: 0.9134580927714017
82-
image_augmenter:GaussianBlur:use_augmenter, Value: True
83-
image_augmenter:GaussianNoise:sigma_offset, Value: 2.7052914035288045
80+
image_augmenter:GaussianBlur:use_augmenter, Value: False
81+
image_augmenter:GaussianNoise:sigma_offset, Value: 0.0912963521330118
8482
image_augmenter:GaussianNoise:use_augmenter, Value: True
8583
image_augmenter:RandomAffine:use_augmenter, Value: False
86-
image_augmenter:RandomCutout:use_augmenter, Value: False
87-
image_augmenter:Resize:use_augmenter, Value: False
88-
image_augmenter:ZeroPadAndCrop:percent, Value: 0.31028229980870564
89-
normalizer:__choice__, Value: 'ImageNormalizer'
84+
image_augmenter:RandomCutout:p, Value: 0.26803834833676243
85+
image_augmenter:RandomCutout:use_augmenter, Value: True
86+
image_augmenter:Resize:use_augmenter, Value: True
87+
image_augmenter:ZeroPadAndCrop:percent, Value: 0.3212184051746134
88+
normalizer:__choice__, Value: 'NoNormalizer'
9089

9190
Fitting the pipeline...
9291
________________________________________
9392
ImageClassificationPipeline
9493
________________________________________
9594
0-) normalizer:
96-
ImageNormalizer
95+
NoNormalizer
9796

9897
1-) preprocessing:
9998
EarlyPreprocessing
@@ -165,7 +164,7 @@ Image Classification
165164
166165
.. rst-class:: sphx-glr-timing
167166

168-
**Total running time of the script:** ( 0 minutes 8.327 seconds)
167+
**Total running time of the script:** ( 0 minutes 6.992 seconds)
169168

170169

171170
.. _sphx_glr_download_examples_example_image_classification.py:

‎refactor_development/_sources/examples/example_tabular_classification.rst.txt‎

Lines changed: 140 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ with AutoPyTorch
3636

3737
.. code-block:: none
3838
39-
<smac.runhistory.runhistory.RunHistory object at 0x7fa1d40e1f70> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
39+
<smac.runhistory.runhistory.RunHistory object at 0x7fb8b3596490> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
4040
data_loader:batch_size, Value: 32
4141
encoder:__choice__, Value: 'OneHotEncoder'
4242
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -53,6 +53,16 @@ with AutoPyTorch
5353
network_backbone:ShapedMLPBackbone:output_dim, Value: 200
5454
network_backbone:ShapedMLPBackbone:use_dropout, Value: False
5555
network_backbone:__choice__, Value: 'ShapedMLPBackbone'
56+
network_embedding:LearnedEntityEmbedding:dimension_reduction_0, Value: 0.5
57+
network_embedding:LearnedEntityEmbedding:dimension_reduction_1, Value: 0.5
58+
network_embedding:LearnedEntityEmbedding:dimension_reduction_2, Value: 0.5
59+
network_embedding:LearnedEntityEmbedding:dimension_reduction_3, Value: 0.5
60+
network_embedding:LearnedEntityEmbedding:dimension_reduction_4, Value: 0.5
61+
network_embedding:LearnedEntityEmbedding:dimension_reduction_5, Value: 0.5
62+
network_embedding:LearnedEntityEmbedding:dimension_reduction_6, Value: 0.5
63+
network_embedding:LearnedEntityEmbedding:dimension_reduction_7, Value: 0.5
64+
network_embedding:LearnedEntityEmbedding:min_unique_values_for_embedding, Value: 5
65+
network_embedding:__choice__, Value: 'LearnedEntityEmbedding'
5666
network_head:__choice__, Value: 'fully_connected'
5767
network_head:fully_connected:activation, Value: 'relu'
5868
network_head:fully_connected:num_layers, Value: 2
@@ -67,7 +77,7 @@ with AutoPyTorch
6777
scaler:__choice__, Value: 'StandardScaler'
6878
trainer:StandardTrainer:weighted_loss, Value: True
6979
trainer:__choice__, Value: 'StandardTrainer'
70-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0025670528411865234, budget=0), TrajEntry(train_perf=0.15204678362573099, incumbent_id=1, incumbent=Configuration:
80+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0018329620361328125, budget=0), TrajEntry(train_perf=0.1871345029239766, incumbent_id=1, incumbent=Configuration:
7181
data_loader:batch_size, Value: 32
7282
encoder:__choice__, Value: 'OneHotEncoder'
7383
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -84,6 +94,16 @@ with AutoPyTorch
8494
network_backbone:ShapedMLPBackbone:output_dim, Value: 200
8595
network_backbone:ShapedMLPBackbone:use_dropout, Value: False
8696
network_backbone:__choice__, Value: 'ShapedMLPBackbone'
97+
network_embedding:LearnedEntityEmbedding:dimension_reduction_0, Value: 0.5
98+
network_embedding:LearnedEntityEmbedding:dimension_reduction_1, Value: 0.5
99+
network_embedding:LearnedEntityEmbedding:dimension_reduction_2, Value: 0.5
100+
network_embedding:LearnedEntityEmbedding:dimension_reduction_3, Value: 0.5
101+
network_embedding:LearnedEntityEmbedding:dimension_reduction_4, Value: 0.5
102+
network_embedding:LearnedEntityEmbedding:dimension_reduction_5, Value: 0.5
103+
network_embedding:LearnedEntityEmbedding:dimension_reduction_6, Value: 0.5
104+
network_embedding:LearnedEntityEmbedding:dimension_reduction_7, Value: 0.5
105+
network_embedding:LearnedEntityEmbedding:min_unique_values_for_embedding, Value: 5
106+
network_embedding:__choice__, Value: 'LearnedEntityEmbedding'
87107
network_head:__choice__, Value: 'fully_connected'
88108
network_head:fully_connected:activation, Value: 'relu'
89109
network_head:fully_connected:num_layers, Value: 2
@@ -98,23 +118,123 @@ with AutoPyTorch
98118
scaler:__choice__, Value: 'StandardScaler'
99119
trainer:StandardTrainer:weighted_loss, Value: True
100120
trainer:__choice__, Value: 'StandardTrainer'
101-
, ta_runs=1, ta_time_used=4.820392370223999, wallclock_time=6.193603515625, budget=5.555555555555555)]
102-
{'accuracy': 0.8670520231213873}
103-
| | Preprocessing | Estimator | Weight |
104-
|---:|:------------------------------------------------------------------|:---------------------------------------------------|---------:|
105-
| 0 | None | ExtraTreesClassifier | 0.2 |
106-
| 1 | None | RFClassifier | 0.16 |
107-
| 2 | SimpleImputer,NoEncoder,StandardScaler,KitchenSink | ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.14 |
108-
| 3 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.12 |
109-
| 4 | None | KNNClassifier | 0.1 |
110-
| 5 | SimpleImputer,OrdinalEncoder,Normalizer,PowerTransformer | MLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
111-
| 6 | SimpleImputer,OneHotEncoder,MinMaxScaler,PolynomialFeatures | ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
112-
| 7 | None | LGBMClassifier | 0.04 |
113-
| 8 | None | SVC | 0.04 |
114-
| 9 | SimpleImputer,OneHotEncoder,Normalizer,KitchenSink | ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
115-
| 10 | SimpleImputer,OneHotEncoder,StandardScaler,PolynomialFeatures | ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
116-
| 11 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
117-
| 12 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
121+
, ta_runs=1, ta_time_used=4.976077318191528, wallclock_time=6.364607572555542, budget=5.555555555555555), TrajEntry(train_perf=0.16959064327485385, incumbent_id=2, incumbent=Configuration:
122+
data_loader:batch_size, Value: 224
123+
encoder:__choice__, Value: 'OneHotEncoder'
124+
feature_preprocessor:KernelPCA:gamma, Value: 0.6217858094449208
125+
feature_preprocessor:KernelPCA:kernel, Value: 'rbf'
126+
feature_preprocessor:KernelPCA:n_components, Value: 4
127+
feature_preprocessor:__choice__, Value: 'KernelPCA'
128+
imputer:categorical_strategy, Value: 'constant_!missing!'
129+
imputer:numerical_strategy, Value: 'most_frequent'
130+
lr_scheduler:CosineAnnealingWarmRestarts:T_0, Value: 10
131+
lr_scheduler:CosineAnnealingWarmRestarts:T_mult, Value: 1.9483254217071713
132+
lr_scheduler:__choice__, Value: 'CosineAnnealingWarmRestarts'
133+
network_backbone:ShapedMLPBackbone:activation, Value: 'relu'
134+
network_backbone:ShapedMLPBackbone:max_units, Value: 948
135+
network_backbone:ShapedMLPBackbone:mlp_shape, Value: 'long_funnel'
136+
network_backbone:ShapedMLPBackbone:num_groups, Value: 13
137+
network_backbone:ShapedMLPBackbone:output_dim, Value: 761
138+
network_backbone:ShapedMLPBackbone:use_dropout, Value: False
139+
network_backbone:__choice__, Value: 'ShapedMLPBackbone'
140+
network_embedding:__choice__, Value: 'NoEmbedding'
141+
network_head:__choice__, Value: 'fully_connected'
142+
network_head:fully_connected:num_layers, Value: 1
143+
network_init:KaimingInit:bias_strategy, Value: 'Zero'
144+
network_init:__choice__, Value: 'KaimingInit'
145+
optimizer:RMSpropOptimizer:alpha, Value: 0.19641480830908647
146+
optimizer:RMSpropOptimizer:lr, Value: 5.575047339285285e-05
147+
optimizer:RMSpropOptimizer:momentum, Value: 0.9188318520804722
148+
optimizer:RMSpropOptimizer:weight_decay, Value: 0.03663295762981204
149+
optimizer:__choice__, Value: 'RMSpropOptimizer'
150+
scaler:__choice__, Value: 'StandardScaler'
151+
trainer:MixUpTrainer:alpha, Value: 0.7490557199071863
152+
trainer:MixUpTrainer:weighted_loss, Value: False
153+
trainer:__choice__, Value: 'MixUpTrainer'
154+
, ta_runs=4, ta_time_used=49.824679136276245, wallclock_time=57.541908502578735, budget=5.555555555555555), TrajEntry(train_perf=0.1578947368421053, incumbent_id=3, incumbent=Configuration:
155+
data_loader:batch_size, Value: 32
156+
encoder:__choice__, Value: 'OneHotEncoder'
157+
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
158+
imputer:categorical_strategy, Value: 'most_frequent'
159+
imputer:numerical_strategy, Value: 'mean'
160+
lr_scheduler:ReduceLROnPlateau:factor, Value: 0.1
161+
lr_scheduler:ReduceLROnPlateau:mode, Value: 'min'
162+
lr_scheduler:ReduceLROnPlateau:patience, Value: 10
163+
lr_scheduler:__choice__, Value: 'ReduceLROnPlateau'
164+
network_backbone:ShapedMLPBackbone:activation, Value: 'relu'
165+
network_backbone:ShapedMLPBackbone:max_units, Value: 200
166+
network_backbone:ShapedMLPBackbone:mlp_shape, Value: 'funnel'
167+
network_backbone:ShapedMLPBackbone:num_groups, Value: 5
168+
network_backbone:ShapedMLPBackbone:output_dim, Value: 200
169+
network_backbone:ShapedMLPBackbone:use_dropout, Value: False
170+
network_backbone:__choice__, Value: 'ShapedMLPBackbone'
171+
network_embedding:LearnedEntityEmbedding:dimension_reduction_0, Value: 0.5
172+
network_embedding:LearnedEntityEmbedding:dimension_reduction_1, Value: 0.5
173+
network_embedding:LearnedEntityEmbedding:dimension_reduction_2, Value: 0.5
174+
network_embedding:LearnedEntityEmbedding:dimension_reduction_3, Value: 0.5
175+
network_embedding:LearnedEntityEmbedding:dimension_reduction_4, Value: 0.5
176+
network_embedding:LearnedEntityEmbedding:dimension_reduction_5, Value: 0.5
177+
network_embedding:LearnedEntityEmbedding:dimension_reduction_6, Value: 0.5
178+
network_embedding:LearnedEntityEmbedding:dimension_reduction_7, Value: 0.5
179+
network_embedding:LearnedEntityEmbedding:min_unique_values_for_embedding, Value: 5
180+
network_embedding:__choice__, Value: 'LearnedEntityEmbedding'
181+
network_head:__choice__, Value: 'fully_connected'
182+
network_head:fully_connected:activation, Value: 'relu'
183+
network_head:fully_connected:num_layers, Value: 2
184+
network_head:fully_connected:units_layer_1, Value: 128
185+
network_init:XavierInit:bias_strategy, Value: 'Normal'
186+
network_init:__choice__, Value: 'XavierInit'
187+
optimizer:AdamOptimizer:beta1, Value: 0.9
188+
optimizer:AdamOptimizer:beta2, Value: 0.9
189+
optimizer:AdamOptimizer:lr, Value: 0.01
190+
optimizer:AdamOptimizer:weight_decay, Value: 0.0
191+
optimizer:__choice__, Value: 'AdamOptimizer'
192+
scaler:__choice__, Value: 'StandardScaler'
193+
trainer:StandardTrainer:weighted_loss, Value: True
194+
trainer:__choice__, Value: 'StandardTrainer'
195+
, ta_runs=11, ta_time_used=115.39660000801086, wallclock_time=140.8079354763031, budget=16.666666666666664), TrajEntry(train_perf=0.1578947368421053, incumbent_id=4, incumbent=Configuration:
196+
data_loader:batch_size, Value: 295
197+
encoder:__choice__, Value: 'OneHotEncoder'
198+
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
199+
imputer:categorical_strategy, Value: 'constant_!missing!'
200+
imputer:numerical_strategy, Value: 'median'
201+
lr_scheduler:ReduceLROnPlateau:factor, Value: 0.7683488018951772
202+
lr_scheduler:ReduceLROnPlateau:mode, Value: 'min'
203+
lr_scheduler:ReduceLROnPlateau:patience, Value: 7
204+
lr_scheduler:__choice__, Value: 'ReduceLROnPlateau'
205+
network_backbone:ShapedMLPBackbone:activation, Value: 'tanh'
206+
network_backbone:ShapedMLPBackbone:max_units, Value: 316
207+
network_backbone:ShapedMLPBackbone:mlp_shape, Value: 'long_funnel'
208+
network_backbone:ShapedMLPBackbone:num_groups, Value: 6
209+
network_backbone:ShapedMLPBackbone:output_dim, Value: 425
210+
network_backbone:ShapedMLPBackbone:use_dropout, Value: False
211+
network_backbone:__choice__, Value: 'ShapedMLPBackbone'
212+
network_embedding:__choice__, Value: 'NoEmbedding'
213+
network_head:__choice__, Value: 'fully_connected'
214+
network_head:fully_connected:activation, Value: 'relu'
215+
network_head:fully_connected:num_layers, Value: 2
216+
network_head:fully_connected:units_layer_1, Value: 424
217+
network_init:OrthogonalInit:bias_strategy, Value: 'Zero'
218+
network_init:__choice__, Value: 'OrthogonalInit'
219+
optimizer:RMSpropOptimizer:alpha, Value: 0.6699215268945383
220+
optimizer:RMSpropOptimizer:lr, Value: 0.0009911973694107326
221+
optimizer:RMSpropOptimizer:momentum, Value: 0.11786464509318967
222+
optimizer:RMSpropOptimizer:weight_decay, Value: 0.04607537154099883
223+
optimizer:__choice__, Value: 'RMSpropOptimizer'
224+
scaler:Normalizer:norm, Value: 'max'
225+
scaler:__choice__, Value: 'Normalizer'
226+
trainer:StandardTrainer:weighted_loss, Value: False
227+
trainer:__choice__, Value: 'StandardTrainer'
228+
, ta_runs=20, ta_time_used=267.25257754325867, wallclock_time=329.0171344280243, budget=50.0)]
229+
{'accuracy': 0.8554913294797688}
230+
| | Preprocessing | Estimator | Weight |
231+
|---:|:------------------------------------------------------------|:-------------------------------------------------------------------|---------:|
232+
| 0 | SimpleImputer,OneHotEncoder,Normalizer,KitchenSink | no embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.6 |
233+
| 1 | SimpleImputer,OneHotEncoder,Normalizer,KitchenSink | no embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.3 |
234+
| 2 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
235+
| 3 | SimpleImputer,NoEncoder,MinMaxScaler,TruncSVD | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
236+
| 4 | SimpleImputer,OneHotEncoder,NoScaler,NoFeaturePreprocessing | embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
237+
| 5 | None | ExtraTreesClassifier | 0.02 |
118238
119239
120240
@@ -206,7 +326,7 @@ with AutoPyTorch
206326
207327
.. rst-class:: sphx-glr-timing
208328

209-
**Total running time of the script:** ( 9 minutes 9.851 seconds)
329+
**Total running time of the script:** ( 9 minutes 15.502 seconds)
210330

211331

212332
.. _sphx_glr_download_examples_example_tabular_classification.py:

0 commit comments

Comments
 (0)