2025-05-08 13:37:14,714 [INFO ] [logger.py:dumpArgs:73]	 Starting with arguments
{"name": "celeba", "dataset": {"path": "../../datasets/CelebAMask/256/img", "lists_root": "datasets/celeba", "targets": ["Blond_Hair"], "attributes_path": "../../datasets/CelebAMask/1024/CelebAMask-HQ-attribute-anno.txt"}, "output": "./models/celeba/Blond_Hair/", "seed": 1337, "verbose": 1, "cuda_index": 0, "learning_rate": 0.0001, "training_iterations": 1000, "batch_size": 250, "validate_every": 10, "patience": 5, "classification": false, "evaluate_training": false, "loss": "BCELogits", "input": null, "architecture": {"name": "resnet", "momentum": 0.01, "track_running_stats": false, "activation": "identity"}, "layers": []}
2025-05-08 13:37:14,714 [INFO ] [learn_neural_network.py:<module>:60]	 Starting to train celeba
2025-05-08 13:37:14,714 [INFO ] [loader.py:getDatasetSplits:23]	 Loading CelebA dataset from ../../datasets/CelebAMask/256/img
2025-05-08 13:37:19,754 [INFO ] [celeba.py:createCelebADataset:139]	 Found 1 CelebA attributes for 30000 images: ['Blond_Hair']
2025-05-08 13:37:19,807 [INFO ] [celeba.py:createCelebADataset:149]	 Loading 26000 training images, 2000 validation images, and 2000 testing images
2025-05-08 13:37:19,807 [INFO ] [learn_neural_network.py:<module>:71]	 Constructing neural network
2025-05-08 13:37:19,807 [INFO ] [loader.py:_ensureNumClassesSet:22]	 Constructing ResNet with 1 targets
2025-05-08 13:37:19,954 [INFO ] [learn_neural_network.py:<module>:84]	 Network has 11177025 parameters
2025-05-08 13:37:19,992 [INFO ] [util.py:selectDevice:85]	 Using cuda:0 for tensor calculations
2025-05-08 13:37:23,128 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.90947 seconds
2025-05-08 13:37:23,129 [INFO ] [learn_neural_network.py:<module>:108]	 Initial validation error 1.005651e+00
2025-05-08 13:37:23,129 [INFO ] [learn_neural_network.py:<module>:111]	 Starting network learning
2025-05-08 13:38:24,613 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 1/1000 in 61.47909 seconds - error: 2.152835e-01
2025-05-08 13:38:24,613 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 13:38:24,613 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 1 to ./models/celeba/Blond_Hair/celeba-20250508-133714-1.pklz
2025-05-08 13:38:29,431 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 3.21491 seconds
2025-05-08 13:38:29,432 [INFO ] [learn_neural_network.py:<module>:166]	 Found new best model with error 2.817101e-01 < prev best 1.005651e+00
2025-05-08 13:39:30,103 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 2/1000 in 60.66596 seconds - error: 8.274613e-02
2025-05-08 13:40:27,014 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 3/1000 in 56.91095 seconds - error: 4.338879e-02
2025-05-08 13:41:22,721 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 4/1000 in 55.70663 seconds - error: 1.874595e-02
2025-05-08 13:42:19,411 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 5/1000 in 56.68955 seconds - error: 9.514232e-03
2025-05-08 13:43:15,292 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 6/1000 in 55.88079 seconds - error: 5.842340e-03
2025-05-08 13:44:11,058 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 7/1000 in 55.76601 seconds - error: 2.080215e-03
2025-05-08 13:45:08,074 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 8/1000 in 57.01536 seconds - error: 1.488828e-03
2025-05-08 13:46:04,223 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 9/1000 in 56.14908 seconds - error: 1.066520e-03
2025-05-08 13:47:00,013 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 10/1000 in 55.78954 seconds - error: 8.412985e-04
2025-05-08 13:47:55,531 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 11/1000 in 55.51849 seconds - error: 7.818644e-04
2025-05-08 13:47:55,531 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 13:47:55,532 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 11 to ./models/celeba/Blond_Hair/celeba-20250508-133714-11.pklz
2025-05-08 13:47:59,976 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.83350 seconds
2025-05-08 13:47:59,976 [INFO ] [learn_neural_network.py:<module>:166]	 Found new best model with error 2.413640e-01 < prev best 2.817101e-01
2025-05-08 13:48:55,709 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 12/1000 in 55.72772 seconds - error: 1.268338e-03
2025-05-08 13:49:51,417 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 13/1000 in 55.70770 seconds - error: 1.414376e-03
2025-05-08 13:50:47,732 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 14/1000 in 56.31484 seconds - error: 2.551470e-02
2025-05-08 13:51:43,253 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 15/1000 in 55.52080 seconds - error: 2.429660e-02
2025-05-08 13:52:38,494 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 16/1000 in 55.24055 seconds - error: 6.916561e-03
2025-05-08 13:53:33,917 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 17/1000 in 55.42344 seconds - error: 2.168730e-03
2025-05-08 13:54:29,637 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 18/1000 in 55.71981 seconds - error: 8.952891e-04
2025-05-08 13:55:25,267 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 19/1000 in 55.62969 seconds - error: 6.162268e-04
2025-05-08 13:56:21,163 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 20/1000 in 55.89571 seconds - error: 1.914817e-03
2025-05-08 13:57:16,680 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 21/1000 in 55.51694 seconds - error: 4.706299e-04
2025-05-08 13:57:16,680 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 13:57:16,680 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 21 to ./models/celeba/Blond_Hair/celeba-20250508-133714-21.pklz
2025-05-08 13:57:21,161 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.89245 seconds
2025-05-08 13:57:21,162 [INFO ] [learn_neural_network.py:<module>:166]	 Found new best model with error 2.153704e-01 < prev best 2.413640e-01
2025-05-08 13:58:16,750 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 22/1000 in 55.58327 seconds - error: 4.270888e-04
2025-05-08 13:59:13,483 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 23/1000 in 56.73245 seconds - error: 3.431569e-04
2025-05-08 14:00:10,534 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 24/1000 in 57.05101 seconds - error: 2.874090e-04
2025-05-08 14:01:06,191 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 25/1000 in 55.65681 seconds - error: 2.576886e-04
2025-05-08 14:02:02,372 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 26/1000 in 56.18053 seconds - error: 3.047823e-04
2025-05-08 14:02:58,446 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 27/1000 in 56.07388 seconds - error: 2.392658e-04
2025-05-08 14:03:56,131 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 28/1000 in 57.68496 seconds - error: 2.209859e-04
2025-05-08 14:04:52,465 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 29/1000 in 56.33328 seconds - error: 2.359297e-04
2025-05-08 14:05:49,174 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 30/1000 in 56.70861 seconds - error: 1.939861e-04
2025-05-08 14:06:44,773 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 31/1000 in 55.59901 seconds - error: 2.039846e-04
2025-05-08 14:06:44,773 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 14:06:44,773 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 31 to ./models/celeba/Blond_Hair/celeba-20250508-133714-31.pklz
2025-05-08 14:06:49,338 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.90339 seconds
2025-05-08 14:06:49,338 [INFO ] [learn_neural_network.py:<module>:159]	 Worsening on valid 2.718651e-01 > prev best 2.153704e-01
2025-05-08 14:07:44,770 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 32/1000 in 55.43204 seconds - error: 1.782401e-04
2025-05-08 14:08:40,403 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 33/1000 in 55.63239 seconds - error: 1.771811e-04
2025-05-08 14:09:36,578 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 34/1000 in 56.17487 seconds - error: 1.800044e-04
2025-05-08 14:10:33,099 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 35/1000 in 56.52170 seconds - error: 1.602005e-04
2025-05-08 14:11:29,125 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 36/1000 in 56.02574 seconds - error: 1.511047e-04
2025-05-08 14:12:24,792 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 37/1000 in 55.66669 seconds - error: 1.636510e-04
2025-05-08 14:13:20,129 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 38/1000 in 55.33679 seconds - error: 1.559673e-04
2025-05-08 14:14:16,172 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 39/1000 in 56.04241 seconds - error: 1.342288e-04
2025-05-08 14:15:11,980 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 40/1000 in 55.80769 seconds - error: 1.331375e-04
2025-05-08 14:16:08,306 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 41/1000 in 56.32614 seconds - error: 1.462207e-04
2025-05-08 14:16:08,306 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 14:16:08,306 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 41 to ./models/celeba/Blond_Hair/celeba-20250508-133714-41.pklz
2025-05-08 14:16:12,851 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.96967 seconds
2025-05-08 14:16:12,851 [INFO ] [learn_neural_network.py:<module>:159]	 Worsening on valid 3.446800e-01 > prev best 2.153704e-01
2025-05-08 14:17:08,715 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 42/1000 in 55.86429 seconds - error: 1.350905e-04
2025-05-08 14:18:04,916 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 43/1000 in 56.20047 seconds - error: 1.271959e-04
2025-05-08 14:19:01,233 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 44/1000 in 56.31636 seconds - error: 1.257696e-04
2025-05-08 14:19:58,067 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 45/1000 in 56.83425 seconds - error: 1.186934e-04
2025-05-08 14:20:54,599 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 46/1000 in 56.53196 seconds - error: 1.123772e-04
2025-05-08 14:21:50,612 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 47/1000 in 56.01213 seconds - error: 1.010047e-04
2025-05-08 14:22:47,413 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 48/1000 in 56.80083 seconds - error: 1.065062e-04
2025-05-08 14:23:43,191 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 49/1000 in 55.77786 seconds - error: 1.054526e-04
2025-05-08 14:24:39,877 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 50/1000 in 56.68675 seconds - error: 9.502100e-05
2025-05-08 14:25:35,616 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 51/1000 in 55.73808 seconds - error: 9.132259e-05
2025-05-08 14:25:35,616 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 14:25:35,616 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 51 to ./models/celeba/Blond_Hair/celeba-20250508-133714-51.pklz
2025-05-08 14:25:40,077 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.88275 seconds
2025-05-08 14:25:40,078 [INFO ] [learn_neural_network.py:<module>:159]	 Worsening on valid 4.131075e-01 > prev best 2.153704e-01
2025-05-08 14:26:35,945 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 52/1000 in 55.86629 seconds - error: 9.959866e-05
2025-05-08 14:27:31,620 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 53/1000 in 55.67454 seconds - error: 9.032708e-05
2025-05-08 14:28:26,949 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 54/1000 in 55.32938 seconds - error: 9.226859e-05
2025-05-08 14:29:22,630 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 55/1000 in 55.68019 seconds - error: 8.522054e-05
2025-05-08 14:30:17,928 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 56/1000 in 55.29827 seconds - error: 9.017858e-05
2025-05-08 14:31:14,566 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 57/1000 in 56.63746 seconds - error: 9.146812e-05
2025-05-08 14:32:11,420 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 58/1000 in 56.85458 seconds - error: 8.493148e-05
2025-05-08 14:33:07,560 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 59/1000 in 56.13986 seconds - error: 1.412792e-02
2025-05-08 14:34:04,136 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 60/1000 in 56.57558 seconds - error: 6.406239e-02
2025-05-08 14:35:00,305 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 61/1000 in 56.16862 seconds - error: 1.365339e-02
2025-05-08 14:35:00,305 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 14:35:00,305 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 61 to ./models/celeba/Blond_Hair/celeba-20250508-133714-61.pklz
2025-05-08 14:35:04,847 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.95407 seconds
2025-05-08 14:35:04,847 [INFO ] [learn_neural_network.py:<module>:159]	 Worsening on valid 9.690395e+00 > prev best 2.153704e-01
2025-05-08 14:36:00,678 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 62/1000 in 55.83104 seconds - error: 8.033586e-03
2025-05-08 14:36:57,371 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 63/1000 in 56.69205 seconds - error: 2.179330e-03
2025-05-08 14:37:53,573 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 64/1000 in 56.20217 seconds - error: 8.589389e-04
2025-05-08 14:39:02,740 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 65/1000 in 69.16688 seconds - error: 3.417928e-04
2025-05-08 14:40:00,078 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 66/1000 in 57.33745 seconds - error: 2.154012e-04
2025-05-08 14:40:53,764 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 67/1000 in 53.68645 seconds - error: 2.000780e-04
2025-05-08 14:41:47,383 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 68/1000 in 53.61883 seconds - error: 1.812446e-04
2025-05-08 14:42:40,614 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 69/1000 in 53.23071 seconds - error: 1.340487e-04
2025-05-08 14:43:33,746 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 70/1000 in 53.13193 seconds - error: 1.420688e-04
2025-05-08 14:44:27,036 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 71/1000 in 53.28997 seconds - error: 1.332068e-04
2025-05-08 14:44:27,036 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 14:44:27,037 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 71 to ./models/celeba/Blond_Hair/celeba-20250508-133714-71.pklz
2025-05-08 14:44:31,109 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.60468 seconds
2025-05-08 14:44:31,109 [INFO ] [learn_neural_network.py:<module>:159]	 Worsening on valid 5.930558e+00 > prev best 2.153704e-01
2025-05-08 14:45:24,423 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 72/1000 in 53.31310 seconds - error: 1.093737e-04
2025-05-08 14:46:17,687 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 73/1000 in 53.26429 seconds - error: 1.110697e-04
2025-05-08 14:47:11,052 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 74/1000 in 53.36467 seconds - error: 1.044517e-04
2025-05-08 14:48:04,383 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 75/1000 in 53.33083 seconds - error: 1.027544e-04
2025-05-08 14:48:57,999 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 76/1000 in 53.61611 seconds - error: 1.070776e-04
2025-05-08 14:49:51,186 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 77/1000 in 53.18722 seconds - error: 8.135801e-05
2025-05-08 14:50:44,442 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 78/1000 in 53.25572 seconds - error: 9.216016e-05
2025-05-08 14:51:37,692 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 79/1000 in 53.24934 seconds - error: 1.032843e-04
2025-05-08 14:52:31,080 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 80/1000 in 53.38855 seconds - error: 9.229759e-05
2025-05-08 14:53:24,328 [INFO ] [learn_neural_network.py:<module>:138]	 celeba iteration 81/1000 in 53.24772 seconds - error: 1.001670e-04
2025-05-08 14:53:24,328 [INFO ] [learn_neural_network.py:<module>:143]	 Evaluating the model via validation data
2025-05-08 14:53:24,329 [INFO ] [learn_neural_network.py:<module>:148]	 Saving model at iteration 81 to ./models/celeba/Blond_Hair/celeba-20250508-133714-81.pklz
2025-05-08 14:53:28,432 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.63562 seconds
2025-05-08 14:53:28,433 [INFO ] [learn_neural_network.py:<module>:159]	 Worsening on valid 4.462364e+00 > prev best 2.153704e-01
2025-05-08 14:53:28,433 [INFO ] [learn_neural_network.py:<module>:161]	 Exceeding patience 5, stopping training
2025-05-08 14:53:28,433 [INFO ] [learn_neural_network.py:<module>:173]	 Network learning done in 4565.30387 secs
2025-05-08 14:53:28,448 [INFO ] [learn_neural_network.py:<module>:197]	 Saving model to ./models/celeba/Blond_Hair/celeba-20250508-133714.pklz
2025-05-08 14:54:04,635 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 34.66157 seconds
2025-05-08 14:54:04,636 [INFO ] [regressor.py:evaluateDataLoaders:119]	 MSE for train is 0.14112283289432526:
tensor([0.1411])
2025-05-08 14:54:07,283 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.64727 seconds
2025-05-08 14:54:07,284 [INFO ] [regressor.py:evaluateDataLoaders:119]	 MSE for validate is 0.21537038683891296:
tensor([0.2154])
2025-05-08 14:54:09,915 [INFO ] [regressor.py:evaluateData:67]	 Evaluated regressor with data loader in 2.63070 seconds
2025-05-08 14:54:09,923 [INFO ] [regressor.py:evaluateDataLoaders:119]	 MSE for test is 0.21070760488510132:
tensor([0.2107])
