FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881489         	|  326.72         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  15.088         	|  20             	|  301.76         	|  92.358         	|
|  run_training_batch                                                                                                                                                                                         	|  0.011837       	|  20000          	|  236.75         	|  72.461         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.011575       	|  20000          	|  231.49         	|  70.853         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0049927      	|  20000          	|  99.854         	|  30.562         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0025066      	|  20000          	|  50.132         	|  15.344         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.35474        	|  64             	|  22.703         	|  6.9488         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.00091804     	|  20000          	|  18.361         	|  5.6197         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  13.187         	|  1              	|  13.187         	|  4.0362         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.0004365      	|  20000          	|  8.73           	|  2.672          	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00012835     	|  20064          	|  2.5752         	|  0.78818        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.11273        	|  20             	|  2.2546         	|  0.69008        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  8.0417e-05     	|  20064          	|  1.6135         	|  0.49384        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.019764       	|  64             	|  1.2649         	|  0.38715        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.1163e-05     	|  20000          	|  0.42326        	|  0.12955        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.4828e-05     	|  20000          	|  0.29656        	|  0.090769       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.1836e-05     	|  20000          	|  0.23673        	|  0.072455       	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.080592       	|  2              	|  0.16118        	|  0.049333       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.9165e-06     	|  20000          	|  0.09833        	|  0.030096       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.038462       	|  2              	|  0.076924       	|  0.023544       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.6411e-06     	|  20000          	|  0.072823       	|  0.022289       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.6144e-06     	|  20000          	|  0.072288       	|  0.022125       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.5509e-06     	|  20000          	|  0.071018       	|  0.021736       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.4521e-06     	|  20000          	|  0.069042       	|  0.021132       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.1303e-06     	|  20000          	|  0.062606       	|  0.019162       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.9504e-06     	|  20000          	|  0.059008       	|  0.018061       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.9331e-06     	|  20000          	|  0.058662       	|  0.017955       	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.7954e-06     	|  20000          	|  0.055908       	|  0.017112       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.7368e-06     	|  20000          	|  0.054735       	|  0.016753       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.7203e-06     	|  20000          	|  0.054406       	|  0.016652       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.6846e-06     	|  20000          	|  0.053693       	|  0.016434       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.6702e-06     	|  20000          	|  0.053403       	|  0.016345       	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.6448e-06     	|  20000          	|  0.052895       	|  0.01619        	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.6168e-06     	|  20000          	|  0.052336       	|  0.016019       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.5635e-06     	|  20000          	|  0.05127        	|  0.015692       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.3561e-06     	|  20000          	|  0.047121       	|  0.014422       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.2808e-06     	|  20000          	|  0.045615       	|  0.013961       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.2726e-06     	|  20000          	|  0.045452       	|  0.013911       	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.2463e-06     	|  20000          	|  0.044926       	|  0.01375        	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.162e-06      	|  20000          	|  0.043239       	|  0.013234       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.148e-06      	|  20000          	|  0.04296        	|  0.013149       	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.3242e-06     	|  20064          	|  0.026569       	|  0.008132       	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.3109e-06     	|  20000          	|  0.026219       	|  0.0080247      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.2295e-06     	|  20000          	|  0.024591       	|  0.0075265      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.1058e-06     	|  20064          	|  0.022188       	|  0.006791       	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.0584e-06     	|  20000          	|  0.021168       	|  0.006479       	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.0204e-06     	|  20000          	|  0.020408       	|  0.0062463      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.020291       	|  1              	|  0.020291       	|  0.0062104      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  9.2358e-07     	|  20000          	|  0.018472       	|  0.0056536      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  9.1574e-07     	|  20000          	|  0.018315       	|  0.0056056      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00027057     	|  64             	|  0.017317       	|  0.0053001      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  8.3966e-07     	|  20000          	|  0.016793       	|  0.0051399      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  7.7969e-07     	|  20000          	|  0.015594       	|  0.0047728      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  7.6625e-07     	|  20000          	|  0.015325       	|  0.0046905      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00028867     	|  20             	|  0.0057734      	|  0.0017671      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00026217     	|  20             	|  0.0052433      	|  0.0016048      	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  5.9441e-05     	|  64             	|  0.0038042      	|  0.0011644      	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  8.825e-05      	|  20             	|  0.001765       	|  0.00054021     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00065805     	|  2              	|  0.0013161      	|  0.00040282     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.0004424      	|  2              	|  0.0008848      	|  0.00027081     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00036182     	|  2              	|  0.00072364     	|  0.00022149     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00033227     	|  2              	|  0.00066455     	|  0.0002034      	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00054108     	|  1              	|  0.00054108     	|  0.00016561     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00035401     	|  1              	|  0.00035401     	|  0.00010835     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  4.3163e-06     	|  64             	|  0.00027625     	|  8.455e-05      	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00026248     	|  1              	|  0.00026248     	|  8.0337e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00025572     	|  1              	|  0.00025572     	|  7.8268e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  2.6812e-06     	|  64             	|  0.0001716      	|  5.252e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.2638e-06     	|  64             	|  0.00014489     	|  4.4345e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.9235e-06     	|  20             	|  0.00013847     	|  4.2381e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.1142e-06     	|  64             	|  0.00013531     	|  4.1415e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.524e-06      	|  20             	|  0.00011048     	|  3.3815e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  1.5425e-06     	|  64             	|  9.8723e-05     	|  3.0216e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.8815e-06     	|  22             	|  8.5392e-05     	|  2.6136e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.4078e-06     	|  22             	|  7.4971e-05     	|  2.2946e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.3944e-06     	|  20             	|  6.7889e-05     	|  2.0779e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.1079e-05     	|  2              	|  6.2158e-05     	|  1.9025e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  9.5376e-07     	|  64             	|  6.1041e-05     	|  1.8683e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  8.8419e-07     	|  64             	|  5.6588e-05     	|  1.732e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.4996e-06     	|  22             	|  5.4991e-05     	|  1.6831e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.4973e-06     	|  22             	|  5.494e-05      	|  1.6815e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.4709e-06     	|  22             	|  5.436e-05      	|  1.6638e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.4168e-06     	|  22             	|  5.3169e-05     	|  1.6273e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  8.0754e-07     	|  64             	|  5.1683e-05     	|  1.5819e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  4.194e-05      	|  1              	|  4.194e-05      	|  1.2837e-05     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.736e-06      	|  20             	|  3.472e-05      	|  1.0627e-05     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.4596e-06     	|  22             	|  3.211e-05      	|  9.8279e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.3714e-06     	|  22             	|  3.0171e-05     	|  9.2345e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.2505e-06     	|  20             	|  2.501e-05      	|  7.6547e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  8.6445e-06     	|  2              	|  1.7289e-05     	|  5.2917e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  7.519e-06      	|  2              	|  1.5038e-05     	|  4.6027e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  9.5703e-06     	|  1              	|  9.5703e-06     	|  2.9292e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  9.3589e-06     	|  1              	|  9.3589e-06     	|  2.8645e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  4.5453e-06     	|  2              	|  9.0906e-06     	|  2.7824e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  8.6902e-06     	|  1              	|  8.6902e-06     	|  2.6598e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  7.6694e-06     	|  1              	|  7.6694e-06     	|  2.3474e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  3.7448e-06     	|  2              	|  7.4897e-06     	|  2.2924e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  3.5851e-06     	|  2              	|  7.1703e-06     	|  2.1946e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  6.9505e-06     	|  1              	|  6.9505e-06     	|  2.1273e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.39e-06       	|  2              	|  6.78e-06       	|  2.0752e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.5202e-06     	|  2              	|  5.0403e-06     	|  1.5427e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.4852e-06     	|  2              	|  4.9705e-06     	|  1.5213e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.4396e-06     	|  1              	|  4.4396e-06     	|  1.3588e-06     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  4.4294e-06     	|  1              	|  4.4294e-06     	|  1.3557e-06     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.2506e-06     	|  1              	|  4.2506e-06     	|  1.301e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.2096e-06     	|  1              	|  4.2096e-06     	|  1.2884e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  4.0801e-06     	|  1              	|  4.0801e-06     	|  1.2488e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  1.9949e-06     	|  2              	|  3.9898e-06     	|  1.2212e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  1.9749e-06     	|  2              	|  3.9497e-06     	|  1.2089e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.8999e-06     	|  2              	|  3.7998e-06     	|  1.163e-06      	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  1.8803e-06     	|  2              	|  3.7607e-06     	|  1.151e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  3.6499e-06     	|  1              	|  3.6499e-06     	|  1.1171e-06     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  3.3602e-06     	|  1              	|  3.3602e-06     	|  1.0285e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  3.2801e-06     	|  1              	|  3.2801e-06     	|  1.0039e-06     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  3.2298e-06     	|  1              	|  3.2298e-06     	|  9.8855e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  3.2298e-06     	|  1              	|  3.2298e-06     	|  9.8855e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  3.2205e-06     	|  1              	|  3.2205e-06     	|  9.857e-07      	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.0799e-06     	|  1              	|  3.0799e-06     	|  9.4266e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  2.9998e-06     	|  1              	|  2.9998e-06     	|  9.1815e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  2.9495e-06     	|  1              	|  2.9495e-06     	|  9.0275e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  2.9495e-06     	|  1              	|  2.9495e-06     	|  9.0275e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.5397e-06     	|  1              	|  2.5397e-06     	|  7.7733e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.3302e-06     	|  1              	|  2.3302e-06     	|  7.1319e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  2.2901e-06     	|  1              	|  2.2901e-06     	|  7.0094e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.1446e-06     	|  2              	|  2.2892e-06     	|  7.0065e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.2799e-06     	|  1              	|  2.2799e-06     	|  6.978e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.2706e-06     	|  1              	|  2.2706e-06     	|  6.9495e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  2.2491e-06     	|  1              	|  2.2491e-06     	|  6.884e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.2203e-06     	|  1              	|  2.2203e-06     	|  6.7956e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.17e-06       	|  1              	|  2.17e-06       	|  6.6417e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.1504e-06     	|  1              	|  2.1504e-06     	|  6.5818e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.1402e-06     	|  1              	|  2.1402e-06     	|  6.5504e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.1299e-06     	|  1              	|  2.1299e-06     	|  6.5191e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.1206e-06     	|  1              	|  2.1206e-06     	|  6.4906e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.0796e-06     	|  1              	|  2.0796e-06     	|  6.3652e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.03e-06       	|  2              	|  2.0601e-06     	|  6.3053e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.0107e-06     	|  1              	|  2.0107e-06     	|  6.1542e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9995e-06     	|  1              	|  1.9995e-06     	|  6.12e-07       	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.8599e-06     	|  1              	|  1.8599e-06     	|  5.6924e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.4594e-06     	|  1              	|  1.4594e-06     	|  4.4667e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  1.3197e-06     	|  1              	|  1.3197e-06     	|  4.0392e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.0794e-06     	|  1              	|  1.0794e-06     	|  3.3037e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  9.6951e-07     	|  1              	|  9.6951e-07     	|  2.9674e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881489         	|  688.0          	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  32.898         	|  20             	|  657.96         	|  95.633         	|
|  run_training_batch                                                                                                                                                                                         	|  0.014723       	|  20000          	|  294.46         	|  42.799         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.014347       	|  20000          	|  286.95         	|  41.707         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.012682       	|  20000          	|  253.65         	|  36.867         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0070156      	|  20000          	|  140.31         	|  20.394         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  1.1244         	|  64             	|  71.962         	|  10.46          	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.002454       	|  20000          	|  49.081         	|  7.1338         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  20.603         	|  1              	|  20.603         	|  2.9946         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00061508     	|  20000          	|  12.302         	|  1.788          	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00032532     	|  20064          	|  6.5271         	|  0.94871        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00019551     	|  20064          	|  3.9228         	|  0.57017        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.12966        	|  20             	|  2.5933         	|  0.37693        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.018259       	|  64             	|  1.1686         	|  0.16985        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.5355e-05     	|  20000          	|  0.70709        	|  0.10278        	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.1828e-05     	|  20000          	|  0.43655        	|  0.063453       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.1395e-05     	|  20000          	|  0.4279         	|  0.062194       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.067597       	|  2              	|  0.13519        	|  0.01965        	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  5.5499e-06     	|  20000          	|  0.111          	|  0.016133       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  5.53e-06       	|  20000          	|  0.1106         	|  0.016076       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  5.2062e-06     	|  20000          	|  0.10412        	|  0.015134       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.0773e-06     	|  20000          	|  0.10155        	|  0.01476        	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  4.8868e-06     	|  20000          	|  0.097737       	|  0.014206       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  4.7118e-06     	|  20000          	|  0.094236       	|  0.013697       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  4.7036e-06     	|  20000          	|  0.094071       	|  0.013673       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  4.5274e-06     	|  20000          	|  0.090548       	|  0.013161       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  3.9735e-06     	|  20000          	|  0.07947        	|  0.011551       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.8008e-06     	|  20000          	|  0.076017       	|  0.011049       	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  3.6307e-06     	|  20000          	|  0.072614       	|  0.010554       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.6267e-06     	|  20000          	|  0.072533       	|  0.010543       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  3.3877e-06     	|  20000          	|  0.067754       	|  0.0098479      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  3.35e-06       	|  20064          	|  0.067213       	|  0.0097694      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  3.2743e-06     	|  20000          	|  0.065486       	|  0.0095183      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  3.1689e-06     	|  20000          	|  0.063378       	|  0.0092118      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  3.0857e-06     	|  20000          	|  0.061715       	|  0.0089702      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.9074e-06     	|  20000          	|  0.058149       	|  0.0084518      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.8983e-06     	|  20000          	|  0.057965       	|  0.0084251      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.814e-06      	|  20000          	|  0.05628        	|  0.0081803      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.7394e-06     	|  20000          	|  0.054788       	|  0.0079634      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.5186e-06     	|  20000          	|  0.050373       	|  0.0073216      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.3483e-06     	|  20000          	|  0.046965       	|  0.0068264      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.000654       	|  64             	|  0.041856       	|  0.0060837      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  2.0694e-06     	|  20000          	|  0.041387       	|  0.0060156      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.8222e-06     	|  20064          	|  0.036561       	|  0.0053141      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.7441e-06     	|  20000          	|  0.034881       	|  0.00507        	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.7371e-06     	|  20000          	|  0.034742       	|  0.0050497      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.5672e-06     	|  20000          	|  0.031344       	|  0.0045558      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.5527e-06     	|  20000          	|  0.031053       	|  0.0045136      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.4441e-06     	|  20000          	|  0.028882       	|  0.0041979      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.2897e-06     	|  20000          	|  0.025794       	|  0.0037491      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.2477e-06     	|  20000          	|  0.024955       	|  0.0036271      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.2464e-06     	|  20000          	|  0.024928       	|  0.0036233      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0050889      	|  2              	|  0.010178       	|  0.0014793      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00047471     	|  20             	|  0.0094941      	|  0.00138        	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.00816        	|  1              	|  0.00816        	|  0.001186       	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00039413     	|  20             	|  0.0078826      	|  0.0011457      	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.4883e-05     	|  64             	|  0.0028725      	|  0.00041752     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.0012939      	|  2              	|  0.0025877      	|  0.00037612     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00011309     	|  20             	|  0.0022619      	|  0.00032876     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00069722     	|  2              	|  0.0013944      	|  0.00020268     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00050217     	|  2              	|  0.0010043      	|  0.00014598     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00046121     	|  2              	|  0.00092241     	|  0.00013407     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  0.00070636     	|  1              	|  0.00070636     	|  0.00010267     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00060468     	|  1              	|  0.00060468     	|  8.7889e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  7.1811e-06     	|  64             	|  0.00045959     	|  6.6801e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00036874     	|  1              	|  0.00036874     	|  5.3596e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  5.0042e-06     	|  64             	|  0.00032027     	|  4.6551e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.0003126      	|  1              	|  0.0003126      	|  4.5435e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00029698     	|  1              	|  0.00029698     	|  4.3165e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.6348e-06     	|  64             	|  0.00023263     	|  3.3812e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.5414e-06     	|  64             	|  0.00022665     	|  3.2943e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.8209e-06     	|  64             	|  0.00018054     	|  2.6241e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.1769e-06     	|  64             	|  0.00013932     	|  2.025e-05      	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.7524e-06     	|  20             	|  0.00013505     	|  1.9629e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  6.339e-06      	|  20             	|  0.00012678     	|  1.8427e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.5301e-06     	|  64             	|  9.7925e-05     	|  1.4233e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.9546e-06     	|  22             	|  8.7001e-05     	|  1.2646e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.3371e-06     	|  64             	|  8.5572e-05     	|  1.2438e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.6309e-06     	|  22             	|  7.988e-05      	|  1.1611e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.784e-05      	|  2              	|  7.5679e-05     	|  1.1e-05        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.4275e-06     	|  20             	|  6.8551e-05     	|  9.9638e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.5845e-06     	|  22             	|  5.686e-05      	|  8.2645e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.575e-06      	|  22             	|  5.665e-05      	|  8.2339e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.4362e-06     	|  22             	|  5.3596e-05     	|  7.7901e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.4341e-06     	|  22             	|  5.3551e-05     	|  7.7836e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.944e-06      	|  20             	|  3.8881e-05     	|  5.6513e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.7437e-06     	|  22             	|  3.8361e-05     	|  5.5757e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.8826e-06     	|  20             	|  3.7652e-05     	|  5.4726e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.5787e-06     	|  22             	|  3.4732e-05     	|  5.0482e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  8.3498e-06     	|  2              	|  1.67e-05       	|  2.4273e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  7.6052e-06     	|  2              	|  1.521e-05      	|  2.2108e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.213e-05      	|  1              	|  1.213e-05      	|  1.763e-06      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  5.2401e-06     	|  2              	|  1.048e-05      	|  1.5233e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  4.5048e-06     	|  2              	|  9.0096e-06     	|  1.3095e-06     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  4.4247e-06     	|  2              	|  8.8494e-06     	|  1.2863e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  4.305e-06      	|  2              	|  8.6101e-06     	|  1.2515e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.7001e-06     	|  2              	|  7.4003e-06     	|  1.0756e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  3.6797e-06     	|  2              	|  7.3593e-06     	|  1.0697e-06     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  7.2904e-06     	|  1              	|  7.2904e-06     	|  1.0596e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  6.9402e-06     	|  1              	|  6.9402e-06     	|  1.0088e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  3.4599e-06     	|  2              	|  6.9197e-06     	|  1.0058e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  5.9195e-06     	|  1              	|  5.9195e-06     	|  8.6039e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  5.4305e-06     	|  1              	|  5.4305e-06     	|  7.8932e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.7101e-06     	|  2              	|  5.4203e-06     	|  7.8783e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.6897e-06     	|  2              	|  5.3793e-06     	|  7.8188e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.9798e-06     	|  1              	|  4.9798e-06     	|  7.238e-07      	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.3055e-06     	|  2              	|  4.611e-06      	|  6.702e-07      	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  4.2999e-06     	|  1              	|  4.2999e-06     	|  6.2499e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.9749e-06     	|  2              	|  3.9497e-06     	|  5.7409e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  3.7998e-06     	|  1              	|  3.7998e-06     	|  5.523e-07      	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  3.6703e-06     	|  1              	|  3.6703e-06     	|  5.3348e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.3202e-06     	|  1              	|  3.3202e-06     	|  4.8258e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  1.6256e-06     	|  2              	|  3.2512e-06     	|  4.7256e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.2391e-06     	|  1              	|  3.2391e-06     	|  4.708e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.8992e-06     	|  1              	|  2.8992e-06     	|  4.214e-07      	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  2.7902e-06     	|  1              	|  2.7902e-06     	|  4.0556e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.7698e-06     	|  1              	|  2.7698e-06     	|  4.0258e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.6906e-06     	|  1              	|  2.6906e-06     	|  3.9107e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  2.6599e-06     	|  1              	|  2.6599e-06     	|  3.8661e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.4801e-06     	|  1              	|  2.4801e-06     	|  3.6048e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.4196e-06     	|  1              	|  2.4196e-06     	|  3.5168e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  2.3898e-06     	|  1              	|  2.3898e-06     	|  3.4735e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  2.3795e-06     	|  1              	|  2.3795e-06     	|  3.4586e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.3702e-06     	|  1              	|  2.3702e-06     	|  3.4451e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.3395e-06     	|  1              	|  2.3395e-06     	|  3.4004e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.2203e-06     	|  1              	|  2.2203e-06     	|  3.2271e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.17e-06       	|  1              	|  2.17e-06       	|  3.154e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.1197e-06     	|  1              	|  2.1197e-06     	|  3.0809e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.1001e-06     	|  1              	|  2.1001e-06     	|  3.0525e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.0796e-06     	|  1              	|  2.0796e-06     	|  3.0227e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.0005e-06     	|  1              	|  2.0005e-06     	|  2.9077e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  1.9995e-06     	|  1              	|  1.9995e-06     	|  2.9063e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.9697e-06     	|  1              	|  1.9697e-06     	|  2.863e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  1.9409e-06     	|  1              	|  1.9409e-06     	|  2.821e-07      	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.9399e-06     	|  1              	|  1.9399e-06     	|  2.8197e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9195e-06     	|  1              	|  1.9195e-06     	|  2.7899e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  1.6596e-06     	|  1              	|  1.6596e-06     	|  2.4122e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.6298e-06     	|  1              	|  1.6298e-06     	|  2.3689e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.6e-06        	|  1              	|  1.6e-06        	|  2.3256e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  1.5097e-06     	|  1              	|  1.5097e-06     	|  2.1943e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.2601e-06     	|  1              	|  1.2601e-06     	|  1.8315e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.1101e-06     	|  1              	|  1.1101e-06     	|  1.6136e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  9.6951e-07     	|  1              	|  9.6951e-07     	|  1.4092e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
