FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  882001         	|  1150.9         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  55.145         	|  20             	|  1102.9         	|  95.831         	|
|  run_training_batch                                                                                                                                                                                         	|  0.041903       	|  20000          	|  838.06         	|  72.818         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.041492       	|  20000          	|  829.84         	|  72.104         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.015461       	|  20000          	|  309.21         	|  26.868         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0057424      	|  20000          	|  114.85         	|  9.9791         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  1.1676         	|  96             	|  112.09         	|  9.739          	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0047348      	|  20000          	|  94.696         	|  8.2281         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  29.85          	|  1              	|  29.85          	|  2.5936         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00080026     	|  20000          	|  16.005         	|  1.3907         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00022661     	|  20096          	|  4.5539         	|  0.39569        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00014367     	|  20096          	|  2.8872         	|  0.25086        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.1412         	|  20             	|  2.8239         	|  0.24537        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.016995       	|  96             	|  1.6316         	|  0.14177        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.5152e-05     	|  20000          	|  0.70303        	|  0.061086       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.033e-05      	|  20000          	|  0.4066         	|  0.035329       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.8853e-05     	|  20000          	|  0.37706        	|  0.032763       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.078984       	|  2              	|  0.15797        	|  0.013726       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  5.3719e-06     	|  20000          	|  0.10744        	|  0.0093353      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  4.1554e-06     	|  20000          	|  0.083108       	|  0.0072212      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  4.1548e-06     	|  20000          	|  0.083095       	|  0.0072201      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.921e-06      	|  20000          	|  0.07842        	|  0.0068139      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.7131e-06     	|  20000          	|  0.074262       	|  0.0064526      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.4314e-06     	|  20000          	|  0.068627       	|  0.005963       	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  3.2212e-06     	|  20000          	|  0.064424       	|  0.0055977      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  3.159e-06      	|  20000          	|  0.06318        	|  0.0054897      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.155e-06      	|  20000          	|  0.0631         	|  0.0054827      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.0677e-06     	|  20000          	|  0.061353       	|  0.005331       	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.7617e-06     	|  20000          	|  0.055235       	|  0.0047993      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.7422e-06     	|  20000          	|  0.054844       	|  0.0047653      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.6504e-06     	|  20000          	|  0.053007       	|  0.0046058      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.5664e-06     	|  20000          	|  0.051327       	|  0.0044598      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.5215e-06     	|  20000          	|  0.05043        	|  0.0043819      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.5095e-06     	|  20000          	|  0.050191       	|  0.004361       	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00051323     	|  96             	|  0.04927        	|  0.004281       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.4618e-06     	|  20000          	|  0.049235       	|  0.004278       	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  2.3756e-06     	|  20096          	|  0.047739       	|  0.004148       	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  2.3434e-06     	|  20096          	|  0.047094       	|  0.004092       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.2726e-06     	|  20000          	|  0.045451       	|  0.0039493      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.2247e-06     	|  20000          	|  0.044495       	|  0.0038661      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.2046e-06     	|  20000          	|  0.044092       	|  0.0038311      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.1447e-06     	|  20000          	|  0.042895       	|  0.0037271      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.1261e-06     	|  20000          	|  0.042521       	|  0.0036947      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.3971e-06     	|  20000          	|  0.027942       	|  0.0024279      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.368e-06      	|  20000          	|  0.027359       	|  0.0023772      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.3637e-06     	|  20000          	|  0.027273       	|  0.0023697      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.1833e-06     	|  20000          	|  0.023666       	|  0.0020563      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.1659e-06     	|  20000          	|  0.023319       	|  0.0020262      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.1355e-06     	|  20000          	|  0.02271        	|  0.0019733      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.1219e-06     	|  20000          	|  0.022438       	|  0.0019496      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.1199e-06     	|  20000          	|  0.022397       	|  0.0019461      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.0154e-06     	|  20000          	|  0.020307       	|  0.0017645      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.014072       	|  1              	|  0.014072       	|  0.0012227      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.0006119      	|  20             	|  0.012238       	|  0.0010634      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0041764      	|  2              	|  0.0083527      	|  0.00072576     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00027668     	|  20             	|  0.0055336      	|  0.00048081     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  3.0955e-05     	|  96             	|  0.0029717      	|  0.00025821     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00011114     	|  20             	|  0.0022228      	|  0.00019314     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00071139     	|  2              	|  0.0014228      	|  0.00012363     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00062514     	|  2              	|  0.0012503      	|  0.00010864     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00052429     	|  2              	|  0.0010486      	|  9.111e-05      	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00068959     	|  1              	|  0.00068959     	|  5.9918e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00034023     	|  2              	|  0.00068046     	|  5.9125e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  7.0615e-06     	|  96             	|  0.00067791     	|  5.8903e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  4.0316e-06     	|  96             	|  0.00038703     	|  3.3629e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.9108e-06     	|  96             	|  0.00037544     	|  3.2622e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00036853     	|  1              	|  0.00036853     	|  3.2021e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00034887     	|  1              	|  0.00034887     	|  3.0313e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00032793     	|  1              	|  0.00032793     	|  2.8493e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.2948e-06     	|  96             	|  0.0003163      	|  2.7483e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.9968e-06     	|  96             	|  0.0002877      	|  2.4998e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.6608e-06     	|  96             	|  0.00025544     	|  2.2195e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.8558e-06     	|  96             	|  0.00017815     	|  1.548e-05      	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.3846e-06     	|  96             	|  0.00013292     	|  1.155e-05      	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.3299e-06     	|  20             	|  0.0001266      	|  1.1e-05        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  0.00012423     	|  1              	|  0.00012423     	|  1.0794e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.1525e-06     	|  20             	|  0.00010305     	|  8.9539e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  4.504e-05      	|  2              	|  9.0079e-05     	|  7.8269e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.6986e-06     	|  22             	|  8.1369e-05     	|  7.0701e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.6601e-06     	|  22             	|  8.0523e-05     	|  6.9966e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.1381e-06     	|  20             	|  6.2763e-05     	|  5.4534e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.5059e-06     	|  22             	|  5.513e-05      	|  4.7902e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.381e-06      	|  22             	|  5.2381e-05     	|  4.5514e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.3395e-06     	|  22             	|  5.147e-05      	|  4.4722e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.2773e-06     	|  22             	|  5.01e-05       	|  4.3531e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.7208e-06     	|  22             	|  3.7858e-05     	|  3.2895e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.8104e-05     	|  2              	|  3.6209e-05     	|  3.1462e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.4424e-06     	|  22             	|  3.1733e-05     	|  2.7573e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.573e-06      	|  20             	|  3.146e-05      	|  2.7335e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.4795e-06     	|  20             	|  2.9589e-05     	|  2.571e-06      	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  1.024e-05      	|  2              	|  2.048e-05      	|  1.7795e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  6.93e-06       	|  2              	|  1.386e-05      	|  1.2043e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.3609e-05     	|  1              	|  1.3609e-05     	|  1.1825e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  6.3647e-06     	|  2              	|  1.2729e-05     	|  1.106e-06      	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.12e-05       	|  1              	|  1.12e-05       	|  9.7317e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  5.5451e-06     	|  2              	|  1.109e-05      	|  9.6362e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  1.043e-05      	|  1              	|  1.043e-05      	|  9.0625e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  3.5143e-06     	|  2              	|  7.0287e-06     	|  6.1072e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  3.2554e-06     	|  2              	|  6.5109e-06     	|  5.6573e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  6.4e-06        	|  1              	|  6.4e-06        	|  5.561e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  3.1199e-06     	|  2              	|  6.2399e-06     	|  5.4218e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  5.8198e-06     	|  1              	|  5.8198e-06     	|  5.0568e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.6494e-06     	|  1              	|  5.6494e-06     	|  4.9087e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  5.4706e-06     	|  1              	|  5.4706e-06     	|  4.7534e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  5.3197e-06     	|  1              	|  5.3197e-06     	|  4.6223e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.6049e-06     	|  2              	|  5.2098e-06     	|  4.5268e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  5.0897e-06     	|  1              	|  5.0897e-06     	|  4.4224e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  5.0403e-06     	|  1              	|  5.0403e-06     	|  4.3795e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.8699e-06     	|  1              	|  4.8699e-06     	|  4.2314e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  4.7395e-06     	|  1              	|  4.7395e-06     	|  4.1181e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.6995e-06     	|  1              	|  4.6995e-06     	|  4.0833e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  4.6305e-06     	|  1              	|  4.6305e-06     	|  4.0234e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.4899e-06     	|  1              	|  4.4899e-06     	|  3.9013e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.4107e-06     	|  1              	|  4.4107e-06     	|  3.8325e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  4.4098e-06     	|  1              	|  4.4098e-06     	|  3.8317e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.0894e-06     	|  1              	|  4.0894e-06     	|  3.5533e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  1.9651e-06     	|  2              	|  3.9302e-06     	|  3.4149e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.9497e-06     	|  2              	|  3.8994e-06     	|  3.3882e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  3.8203e-06     	|  1              	|  3.8203e-06     	|  3.3194e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.5698e-06     	|  1              	|  3.5698e-06     	|  3.1017e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.77e-06       	|  2              	|  3.54e-06       	|  3.0758e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.5204e-06     	|  1              	|  3.5204e-06     	|  3.0589e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.7099e-06     	|  2              	|  3.4198e-06     	|  2.9715e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.5851e-06     	|  2              	|  3.1702e-06     	|  2.7546e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  3.0501e-06     	|  1              	|  3.0501e-06     	|  2.6502e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.9895e-06     	|  1              	|  2.9895e-06     	|  2.5976e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.9206e-06     	|  1              	|  2.9206e-06     	|  2.5377e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.9001e-06     	|  1              	|  2.9001e-06     	|  2.5199e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.8899e-06     	|  1              	|  2.8899e-06     	|  2.511e-07      	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.7902e-06     	|  1              	|  2.7902e-06     	|  2.4244e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.7493e-06     	|  1              	|  2.7493e-06     	|  2.3888e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.7297e-06     	|  1              	|  2.7297e-06     	|  2.3718e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.6794e-06     	|  1              	|  2.6794e-06     	|  2.3281e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.6496e-06     	|  1              	|  2.6496e-06     	|  2.3022e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.6096e-06     	|  1              	|  2.6096e-06     	|  2.2674e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.5602e-06     	|  1              	|  2.5602e-06     	|  2.2245e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.4596e-06     	|  1              	|  2.4596e-06     	|  2.1372e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.2901e-06     	|  1              	|  2.2901e-06     	|  1.9899e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  2.1094e-06     	|  1              	|  2.1094e-06     	|  1.8329e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.5199e-06     	|  1              	|  1.5199e-06     	|  1.3206e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.3998e-06     	|  1              	|  1.3998e-06     	|  1.2163e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.3998e-06     	|  1              	|  1.3998e-06     	|  1.2163e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.1595e-06     	|  1              	|  1.1595e-06     	|  1.0075e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
