FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881585         	|  1491.3         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  71.111         	|  20             	|  1422.2         	|  95.369         	|
|  run_training_batch                                                                                                                                                                                         	|  0.061852       	|  20000          	|  1237.0         	|  82.951         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.061423       	|  20000          	|  1228.5         	|  82.375         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.020862       	|  20000          	|  417.23         	|  27.978         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0061361      	|  20000          	|  122.72         	|  8.2293         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0047642      	|  20000          	|  95.284         	|  6.3894         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  57.369         	|  1              	|  57.369         	|  3.8469         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.46562        	|  70             	|  32.594         	|  2.1856         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00055314     	|  20000          	|  11.063         	|  0.74183        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.22081        	|  20             	|  4.4162         	|  0.29614        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00017904     	|  20070          	|  3.5934         	|  0.24096        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00011699     	|  20070          	|  2.348          	|  0.15745        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.4327e-05     	|  20000          	|  0.68653        	|  0.046036       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.2427e-05     	|  20000          	|  0.44854        	|  0.030077       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.8808e-05     	|  20000          	|  0.37616        	|  0.025224       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0038846      	|  70             	|  0.27192        	|  0.018234       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  5.2436e-06     	|  20000          	|  0.10487        	|  0.0070323      	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.049766       	|  2              	|  0.099531       	|  0.0066742      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  4.467e-06      	|  20000          	|  0.089339       	|  0.0059907      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  4.1144e-06     	|  20000          	|  0.082287       	|  0.0055179      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.7934e-06     	|  20000          	|  0.075868       	|  0.0050874      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.5867e-06     	|  20000          	|  0.071734       	|  0.0048102      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.4413e-06     	|  20000          	|  0.068825       	|  0.0046151      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.2011e-06     	|  20000          	|  0.064022       	|  0.0042931      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.1197e-06     	|  20000          	|  0.062395       	|  0.004184       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  3.0451e-06     	|  20000          	|  0.060902       	|  0.0040839      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.9156e-06     	|  20000          	|  0.058311       	|  0.0039101      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.7299e-06     	|  20000          	|  0.054597       	|  0.0036611      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.6196e-06     	|  20000          	|  0.052393       	|  0.0035133      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.4999e-06     	|  20000          	|  0.049997       	|  0.0033526      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.4603e-06     	|  20000          	|  0.049206       	|  0.0032996      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.4299e-06     	|  20000          	|  0.048598       	|  0.0032588      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.3937e-06     	|  20000          	|  0.047874       	|  0.0032102      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.3566e-06     	|  20000          	|  0.047132       	|  0.0031605      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.3363e-06     	|  20000          	|  0.046726       	|  0.0031332      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.2712e-06     	|  20000          	|  0.045423       	|  0.0030459      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.2204e-06     	|  20000          	|  0.044408       	|  0.0029778      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.1198e-06     	|  20000          	|  0.042396       	|  0.0028429      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.0959e-06     	|  20000          	|  0.041917       	|  0.0028108      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.7921e-06     	|  20070          	|  0.035967       	|  0.0024118      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.0004679      	|  70             	|  0.032753       	|  0.0021963      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.4946e-06     	|  20000          	|  0.029892       	|  0.0020044      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.4899e-06     	|  20000          	|  0.029799       	|  0.0019982      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.4227e-06     	|  20000          	|  0.028455       	|  0.0019081      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.4118e-06     	|  20000          	|  0.028237       	|  0.0018934      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.3859e-06     	|  20070          	|  0.027816       	|  0.0018652      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.2987e-06     	|  20000          	|  0.025974       	|  0.0017417      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.2964e-06     	|  20000          	|  0.025928       	|  0.0017386      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.1824e-06     	|  20000          	|  0.023648       	|  0.0015857      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.1703e-06     	|  20000          	|  0.023406       	|  0.0015695      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.1149e-06     	|  20000          	|  0.022298       	|  0.0014952      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0036704      	|  2              	|  0.0073409      	|  0.00049225     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00029468     	|  20             	|  0.0058936      	|  0.0003952      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.005663       	|  1              	|  0.005663       	|  0.00037974     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00027784     	|  20             	|  0.0055569      	|  0.00037262     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  3.1901e-05     	|  70             	|  0.002233       	|  0.00014974     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  9.8223e-05     	|  20             	|  0.0019645      	|  0.00013173     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00065287     	|  2              	|  0.0013057      	|  8.7558e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00045081     	|  2              	|  0.00090161     	|  6.0459e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00044164     	|  2              	|  0.00088328     	|  5.923e-05      	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00070643     	|  1              	|  0.00070643     	|  4.7371e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  9.6433e-06     	|  70             	|  0.00067503     	|  4.5265e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00022403     	|  2              	|  0.00044806     	|  3.0045e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00035769     	|  1              	|  0.00035769     	|  2.3985e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00031623     	|  1              	|  0.00031623     	|  2.1205e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.0003031      	|  1              	|  0.0003031      	|  2.0325e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.7143e-06     	|  70             	|  0.00026        	|  1.7435e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.4567e-06     	|  70             	|  0.00024197     	|  1.6225e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.056e-06      	|  70             	|  0.00021392     	|  1.4345e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.8234e-06     	|  70             	|  0.00019764     	|  1.3253e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.9918e-06     	|  70             	|  0.00013943     	|  9.3495e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.499e-06      	|  20             	|  0.00012998     	|  8.7159e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.5482e-06     	|  70             	|  0.00010837     	|  7.2671e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  5.3995e-06     	|  20             	|  0.00010799     	|  7.2414e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.3009e-06     	|  20             	|  0.00010602     	|  7.1092e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.3116e-06     	|  70             	|  9.1813e-05     	|  6.1566e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  4.1408e-06     	|  22             	|  9.1097e-05     	|  6.1086e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.7013e-06     	|  22             	|  8.1429e-05     	|  5.4603e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.8114e-05     	|  2              	|  7.6229e-05     	|  5.1116e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.4072e-06     	|  22             	|  5.2958e-05     	|  3.5511e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.3873e-06     	|  22             	|  5.2521e-05     	|  3.5219e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.3727e-06     	|  22             	|  5.2199e-05     	|  3.5002e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.3541e-06     	|  22             	|  5.179e-05      	|  3.4728e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  4.178e-05      	|  1              	|  4.178e-05      	|  2.8016e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.8035e-06     	|  20             	|  3.6069e-05     	|  2.4187e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.5518e-06     	|  22             	|  3.4139e-05     	|  2.2893e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.454e-06      	|  22             	|  3.1989e-05     	|  2.1451e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.4075e-06     	|  20             	|  2.8149e-05     	|  1.8876e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  7.3351e-06     	|  2              	|  1.467e-05      	|  9.8373e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  5.1502e-06     	|  2              	|  1.03e-05       	|  6.9071e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  5.1097e-06     	|  2              	|  1.0219e-05     	|  6.8527e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  9.8497e-06     	|  1              	|  9.8497e-06     	|  6.6048e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  9.6206e-06     	|  1              	|  9.6206e-06     	|  6.4512e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  4.6347e-06     	|  2              	|  9.2695e-06     	|  6.2157e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  4.2003e-06     	|  2              	|  8.4005e-06     	|  5.6331e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.79e-06       	|  1              	|  5.79e-06       	|  3.8826e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  5.75e-06       	|  1              	|  5.75e-06       	|  3.8557e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  5.7295e-06     	|  1              	|  5.7295e-06     	|  3.842e-07      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.7851e-06     	|  2              	|  5.5702e-06     	|  3.7352e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  5.3896e-06     	|  1              	|  5.3896e-06     	|  3.614e-07      	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  5.3495e-06     	|  1              	|  5.3495e-06     	|  3.5872e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  5.2806e-06     	|  1              	|  5.2806e-06     	|  3.541e-07      	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.9099e-06     	|  1              	|  4.9099e-06     	|  3.2924e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.7497e-06     	|  1              	|  4.7497e-06     	|  3.185e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.3399e-06     	|  2              	|  4.6799e-06     	|  3.1382e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.3101e-06     	|  2              	|  4.6203e-06     	|  3.0982e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.57e-06       	|  1              	|  4.57e-06       	|  3.0645e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  4.5104e-06     	|  1              	|  4.5104e-06     	|  3.0245e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.2999e-06     	|  1              	|  4.2999e-06     	|  2.8834e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.115e-06      	|  2              	|  4.2301e-06     	|  2.8365e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.1146e-06     	|  2              	|  4.2291e-06     	|  2.8359e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.0699e-06     	|  2              	|  4.1397e-06     	|  2.7759e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.0652e-06     	|  2              	|  4.1304e-06     	|  2.7697e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.1099e-06     	|  1              	|  4.1099e-06     	|  2.756e-07      	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  3.2494e-06     	|  1              	|  3.2494e-06     	|  2.1789e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.2401e-06     	|  1              	|  3.2401e-06     	|  2.1727e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  3.1805e-06     	|  1              	|  3.1805e-06     	|  2.1327e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  2.9802e-06     	|  1              	|  2.9802e-06     	|  1.9984e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.9495e-06     	|  1              	|  2.9495e-06     	|  1.9778e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  2.8796e-06     	|  1              	|  2.8796e-06     	|  1.931e-07      	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.7604e-06     	|  1              	|  2.7604e-06     	|  1.851e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.6803e-06     	|  1              	|  2.6803e-06     	|  1.7973e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.6003e-06     	|  1              	|  2.6003e-06     	|  1.7436e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.2503e-06     	|  2              	|  2.5006e-06     	|  1.6768e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.4205e-06     	|  1              	|  2.4205e-06     	|  1.6231e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.3702e-06     	|  1              	|  2.3702e-06     	|  1.5894e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.1702e-06     	|  2              	|  2.3404e-06     	|  1.5694e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.3302e-06     	|  1              	|  2.3302e-06     	|  1.5625e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  2.2696e-06     	|  1              	|  2.2696e-06     	|  1.5219e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.21e-06       	|  1              	|  2.21e-06       	|  1.482e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.1905e-06     	|  1              	|  2.1905e-06     	|  1.4688e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.1802e-06     	|  1              	|  2.1802e-06     	|  1.462e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.1001e-06     	|  1              	|  2.1001e-06     	|  1.4083e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.0806e-06     	|  1              	|  2.0806e-06     	|  1.3952e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.0601e-06     	|  1              	|  2.0601e-06     	|  1.3814e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  2.0098e-06     	|  1              	|  2.0098e-06     	|  1.3477e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.8096e-06     	|  1              	|  1.8096e-06     	|  1.2134e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.6699e-06     	|  1              	|  1.6699e-06     	|  1.1197e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.5199e-06     	|  1              	|  1.5199e-06     	|  1.0192e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.2899e-06     	|  1              	|  1.2899e-06     	|  8.6494e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.2796e-06     	|  1              	|  1.2796e-06     	|  8.5807e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  1.1502e-06     	|  1              	|  1.1502e-06     	|  7.7127e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
