FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881937         	|  336.54         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  15.166         	|  20             	|  303.32         	|  90.129         	|
|  run_training_batch                                                                                                                                                                                         	|  0.011635       	|  20000          	|  232.71         	|  69.148         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.011387       	|  20000          	|  227.75         	|  67.674         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0049006      	|  20000          	|  98.012         	|  29.124         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.002355       	|  20000          	|  47.101         	|  13.996         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.28519        	|  92             	|  26.238         	|  7.7964         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  20.557         	|  1              	|  20.557         	|  6.1083         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0010095      	|  20000          	|  20.189         	|  5.9992         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00044319     	|  20000          	|  8.8639         	|  2.6338         	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.15181        	|  20             	|  3.0361         	|  0.90217        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00012609     	|  20092          	|  2.5334         	|  0.7528         	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  7.8747e-05     	|  20092          	|  1.5822         	|  0.47014        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.017185       	|  92             	|  1.581          	|  0.4698         	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.1826e-05     	|  20000          	|  0.43652        	|  0.12971        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.5092e-05     	|  20000          	|  0.30185        	|  0.089692       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.2705e-05     	|  20000          	|  0.2541         	|  0.075503       	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.081504       	|  2              	|  0.16301        	|  0.048437       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.062967       	|  2              	|  0.12593        	|  0.037421       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.5586e-06     	|  20000          	|  0.091172       	|  0.027091       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.6253e-06     	|  20000          	|  0.072506       	|  0.021545       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.5654e-06     	|  20000          	|  0.071309       	|  0.021189       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.5583e-06     	|  20000          	|  0.071166       	|  0.021147       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.4891e-06     	|  20000          	|  0.069782       	|  0.020735       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.0545e-06     	|  20000          	|  0.06109        	|  0.018153       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.8875e-06     	|  20000          	|  0.05775        	|  0.01716        	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.8698e-06     	|  20000          	|  0.057397       	|  0.017055       	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.8051e-06     	|  20000          	|  0.056101       	|  0.01667        	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.7539e-06     	|  20000          	|  0.055078       	|  0.016366       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.7391e-06     	|  20000          	|  0.054782       	|  0.016278       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.7039e-06     	|  20000          	|  0.054078       	|  0.016069       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.701e-06      	|  20000          	|  0.05402        	|  0.016052       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.6458e-06     	|  20000          	|  0.052916       	|  0.015724       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.6338e-06     	|  20000          	|  0.052677       	|  0.015653       	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.4988e-06     	|  20000          	|  0.049976       	|  0.01485        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.3371e-06     	|  20000          	|  0.046742       	|  0.013889       	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.2724e-06     	|  20000          	|  0.045449       	|  0.013505       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.2347e-06     	|  20000          	|  0.044694       	|  0.01328        	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.2056e-06     	|  20000          	|  0.044111       	|  0.013107       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.1658e-06     	|  20000          	|  0.043316       	|  0.012871       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.0844e-06     	|  20000          	|  0.041687       	|  0.012387       	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.305e-06      	|  20092          	|  0.026221       	|  0.0077913      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.2942e-06     	|  20000          	|  0.025885       	|  0.0076915      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.2639e-06     	|  20000          	|  0.025279       	|  0.0075114      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00027095     	|  92             	|  0.024927       	|  0.007407       	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.1551e-06     	|  20092          	|  0.023208       	|  0.0068962      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  9.9165e-07     	|  20000          	|  0.019833       	|  0.0058933      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  9.8884e-07     	|  20000          	|  0.019777       	|  0.0058766      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  9.8035e-07     	|  20000          	|  0.019607       	|  0.0058261      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  9.3029e-07     	|  20000          	|  0.018606       	|  0.0055286      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  8.4254e-07     	|  20000          	|  0.016851       	|  0.0050071      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  8.1292e-07     	|  20000          	|  0.016258       	|  0.0048311      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  7.9343e-07     	|  20000          	|  0.015869       	|  0.0047153      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.011533       	|  1              	|  0.011533       	|  0.0034269      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00026118     	|  20             	|  0.0052236      	|  0.0015522      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00025666     	|  20             	|  0.0051332      	|  0.0015253      	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  3.6123e-05     	|  92             	|  0.0033233      	|  0.00098751     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00010284     	|  20             	|  0.0020568      	|  0.00061117     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00062131     	|  2              	|  0.0012426      	|  0.00036924     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00046497     	|  2              	|  0.00092994     	|  0.00027633     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00037146     	|  2              	|  0.00074292     	|  0.00022076     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00067156     	|  1              	|  0.00067156     	|  0.00019955     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00029916     	|  2              	|  0.00059832     	|  0.00017779     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.0004397      	|  1              	|  0.0004397      	|  0.00013065     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  4.2167e-06     	|  92             	|  0.00038793     	|  0.00011527     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00034754     	|  1              	|  0.00034754     	|  0.00010327     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  0.00029074     	|  1              	|  0.00029074     	|  8.6391e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00024626     	|  1              	|  0.00024626     	|  7.3174e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  2.5915e-06     	|  92             	|  0.00023842     	|  7.0844e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.2339e-06     	|  92             	|  0.00020552     	|  6.107e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.1005e-06     	|  92             	|  0.00019324     	|  5.7421e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  1.4245e-06     	|  92             	|  0.00013105     	|  3.8942e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.5184e-06     	|  20             	|  0.00013037     	|  3.8738e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  9.964e-05      	|  1              	|  9.964e-05      	|  2.9607e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.9378e-06     	|  20             	|  9.8757e-05     	|  2.9345e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  4.7918e-06     	|  20             	|  9.5836e-05     	|  2.8477e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  9.0448e-07     	|  92             	|  8.3212e-05     	|  2.4726e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.6907e-06     	|  22             	|  8.1196e-05     	|  2.4127e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  8.4157e-07     	|  92             	|  7.7425e-05     	|  2.3006e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.4723e-06     	|  22             	|  7.6391e-05     	|  2.2699e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  7.9457e-07     	|  92             	|  7.31e-05       	|  2.1721e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.4969e-05     	|  2              	|  6.9939e-05     	|  2.0782e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.4245e-06     	|  22             	|  5.3339e-05     	|  1.5849e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.3486e-06     	|  22             	|  5.1669e-05     	|  1.5353e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.3282e-06     	|  22             	|  5.122e-05      	|  1.522e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.2941e-06     	|  22             	|  5.0471e-05     	|  1.4997e-05     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  4.1189e-05     	|  1              	|  4.1189e-05     	|  1.2239e-05     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.869e-06      	|  20             	|  3.738e-05      	|  1.1107e-05     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.4206e-06     	|  22             	|  3.1253e-05     	|  9.2867e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.4137e-06     	|  22             	|  3.1102e-05     	|  9.2419e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.3466e-06     	|  20             	|  2.6933e-05     	|  8.0029e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  2.441e-05      	|  1              	|  2.441e-05      	|  7.2533e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.008e-05      	|  2              	|  2.0159e-05     	|  5.9902e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  8.0951e-06     	|  2              	|  1.619e-05      	|  4.8108e-06     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  1.447e-05      	|  1              	|  1.447e-05      	|  4.2997e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  7.1852e-06     	|  2              	|  1.437e-05      	|  4.27e-06       	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  9.8301e-06     	|  1              	|  9.8301e-06     	|  2.921e-06      	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  4.9099e-06     	|  2              	|  9.8199e-06     	|  2.9179e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  9.0003e-06     	|  1              	|  9.0003e-06     	|  2.6744e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  4.2301e-06     	|  2              	|  8.4601e-06     	|  2.5139e-06     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  7.7896e-06     	|  1              	|  7.7896e-06     	|  2.3146e-06     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  7.3006e-06     	|  1              	|  7.3006e-06     	|  2.1693e-06     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  6.8685e-06     	|  1              	|  6.8685e-06     	|  2.0409e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.3551e-06     	|  2              	|  6.7102e-06     	|  1.9939e-06     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.5805e-06     	|  1              	|  5.5805e-06     	|  1.6582e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.7302e-06     	|  2              	|  5.4603e-06     	|  1.6225e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  4.6901e-06     	|  1              	|  4.6901e-06     	|  1.3936e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.6389e-06     	|  1              	|  4.6389e-06     	|  1.3784e-06     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  4.6296e-06     	|  1              	|  4.6296e-06     	|  1.3757e-06     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  4.6203e-06     	|  1              	|  4.6203e-06     	|  1.3729e-06     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.61e-06       	|  1              	|  4.61e-06       	|  1.3698e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.5802e-06     	|  1              	|  4.5802e-06     	|  1.361e-06      	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.5402e-06     	|  1              	|  4.5402e-06     	|  1.3491e-06     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.4499e-06     	|  1              	|  4.4499e-06     	|  1.3222e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.2999e-06     	|  1              	|  4.2999e-06     	|  1.2777e-06     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  2.0997e-06     	|  2              	|  4.1993e-06     	|  1.2478e-06     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  4.1099e-06     	|  1              	|  4.1099e-06     	|  1.2212e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  4.1099e-06     	|  1              	|  4.1099e-06     	|  1.2212e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.0904e-06     	|  1              	|  4.0904e-06     	|  1.2154e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.0149e-06     	|  2              	|  4.0298e-06     	|  1.1974e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  1.9595e-06     	|  2              	|  3.919e-06      	|  1.1645e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  1.9199e-06     	|  2              	|  3.8398e-06     	|  1.141e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  3.77e-06       	|  1              	|  3.77e-06       	|  1.1202e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.8249e-06     	|  2              	|  3.6499e-06     	|  1.0845e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.8198e-06     	|  2              	|  3.6396e-06     	|  1.0815e-06     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  3.5902e-06     	|  1              	|  3.5902e-06     	|  1.0668e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.5795e-06     	|  2              	|  3.159e-06      	|  9.3869e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  3.1292e-06     	|  1              	|  3.1292e-06     	|  9.2984e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.7502e-06     	|  1              	|  2.7502e-06     	|  8.172e-07      	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  2.5602e-06     	|  1              	|  2.5602e-06     	|  7.6075e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.4894e-06     	|  1              	|  2.4894e-06     	|  7.3972e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.4708e-06     	|  1              	|  2.4708e-06     	|  7.3418e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.4606e-06     	|  1              	|  2.4606e-06     	|  7.3114e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.4e-06        	|  1              	|  2.4e-06        	|  7.1315e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.4e-06        	|  1              	|  2.4e-06        	|  7.1315e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.1998e-06     	|  1              	|  2.1998e-06     	|  6.5365e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.1905e-06     	|  1              	|  2.1905e-06     	|  6.5088e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  2.1905e-06     	|  1              	|  2.1905e-06     	|  6.5088e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.0098e-06     	|  1              	|  2.0098e-06     	|  5.972e-07      	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.9399e-06     	|  1              	|  1.9399e-06     	|  5.7644e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  1.8198e-06     	|  1              	|  1.8198e-06     	|  5.4074e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.1893e-06     	|  1              	|  1.1893e-06     	|  3.5339e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.0999e-06     	|  1              	|  1.0999e-06     	|  3.2683e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881937         	|  724.55         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  34.23          	|  20             	|  684.6          	|  94.486         	|
|  run_training_batch                                                                                                                                                                                         	|  0.014852       	|  20000          	|  297.04         	|  40.996         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.014456       	|  20000          	|  289.13         	|  39.904         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.012865       	|  20000          	|  257.3          	|  35.512         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0071057      	|  20000          	|  142.11         	|  19.614         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.99099        	|  92             	|  91.171         	|  12.583         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0025683      	|  20000          	|  51.365         	|  7.0893         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  30.255         	|  1              	|  30.255         	|  4.1756         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00064652     	|  20000          	|  12.93          	|  1.7846         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00030687     	|  20092          	|  6.1657         	|  0.85097        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00019592     	|  20092          	|  3.9364         	|  0.5433         	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.14104        	|  20             	|  2.8208         	|  0.38932        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.017245       	|  92             	|  1.5866         	|  0.21898        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  4.1189e-05     	|  20000          	|  0.82378        	|  0.1137         	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.2015e-05     	|  20000          	|  0.4403         	|  0.06077        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.1078e-05     	|  20000          	|  0.42157        	|  0.058184       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.094137       	|  2              	|  0.18827        	|  0.025985       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  6.1995e-06     	|  20000          	|  0.12399        	|  0.017113       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  6.0401e-06     	|  20000          	|  0.1208         	|  0.016673       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  5.2891e-06     	|  20000          	|  0.10578        	|  0.0146         	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.1936e-06     	|  20000          	|  0.10387        	|  0.014336       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  4.8802e-06     	|  20000          	|  0.097603       	|  0.013471       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  4.7919e-06     	|  20000          	|  0.095838       	|  0.013227       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  4.3692e-06     	|  20000          	|  0.087383       	|  0.01206        	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  4.3468e-06     	|  20092          	|  0.087336       	|  0.012054       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  4.2573e-06     	|  20000          	|  0.085146       	|  0.011752       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  4.0656e-06     	|  20000          	|  0.081311       	|  0.011222       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.8768e-06     	|  20000          	|  0.077535       	|  0.010701       	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  3.5318e-06     	|  20000          	|  0.070636       	|  0.009749       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.3944e-06     	|  20000          	|  0.067888       	|  0.0093697      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  3.2084e-06     	|  20000          	|  0.064167       	|  0.0088561      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  2.911e-06      	|  20092          	|  0.058488       	|  0.0080724      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.8573e-06     	|  20000          	|  0.057146       	|  0.0078872      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.8017e-06     	|  20000          	|  0.056033       	|  0.0077335      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.7612e-06     	|  20000          	|  0.055224       	|  0.0076219      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.7542e-06     	|  20000          	|  0.055085       	|  0.0076026      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.7489e-06     	|  20000          	|  0.054977       	|  0.0075878      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.6176e-06     	|  20000          	|  0.052351       	|  0.0072254      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.5589e-06     	|  20000          	|  0.051178       	|  0.0070635      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.4758e-06     	|  20000          	|  0.049515       	|  0.0068339      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  2.4631e-06     	|  20000          	|  0.049262       	|  0.006799       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.3181e-06     	|  20000          	|  0.046363       	|  0.0063989      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00050329     	|  92             	|  0.046302       	|  0.0063905      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.9465e-06     	|  20000          	|  0.03893        	|  0.005373       	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.9042e-06     	|  20000          	|  0.038085       	|  0.0052563      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.8913e-06     	|  20000          	|  0.037826       	|  0.0052206      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.6575e-06     	|  20000          	|  0.03315        	|  0.0045753      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.6177e-06     	|  20000          	|  0.032354       	|  0.0044655      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.582e-06      	|  20000          	|  0.03164        	|  0.0043669      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.5381e-06     	|  20000          	|  0.030762       	|  0.0042457      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.2273e-06     	|  20000          	|  0.024546       	|  0.0033877      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.020416       	|  1              	|  0.020416       	|  0.0028177      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0065358      	|  2              	|  0.013072       	|  0.0018041      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00029928     	|  20             	|  0.0059855      	|  0.0008261      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00029214     	|  20             	|  0.0058429      	|  0.00080642     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  3.2213e-05     	|  92             	|  0.0029636      	|  0.00040903     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00011359     	|  20             	|  0.0022719      	|  0.00031356     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00082455     	|  2              	|  0.0016491      	|  0.0002276      	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00073298     	|  2              	|  0.001466       	|  0.00020233     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00054815     	|  2              	|  0.0010963      	|  0.00015131     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00092124     	|  1              	|  0.00092124     	|  0.00012715     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  9.159e-06      	|  92             	|  0.00084263     	|  0.0001163      	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.0003693      	|  2              	|  0.0007386      	|  0.00010194     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00051887     	|  1              	|  0.00051887     	|  7.1612e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00036587     	|  1              	|  0.00036587     	|  5.0496e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  0.00036273     	|  1              	|  0.00036273     	|  5.0062e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00036106     	|  1              	|  0.00036106     	|  4.9832e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.498e-06      	|  92             	|  0.00032182     	|  4.4417e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.4749e-06     	|  92             	|  0.00031969     	|  4.4123e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  1.4898e-05     	|  20             	|  0.00029796     	|  4.1124e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.9913e-06     	|  92             	|  0.0002752      	|  3.7982e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.9313e-06     	|  92             	|  0.00026968     	|  3.722e-05      	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.7754e-06     	|  92             	|  0.00016334     	|  2.2543e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.5056e-06     	|  92             	|  0.00013852     	|  1.9118e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.8041e-06     	|  20             	|  0.00013608     	|  1.8782e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.3684e-06     	|  92             	|  0.0001259      	|  1.7376e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.3798e-06     	|  20             	|  0.0001076      	|  1.485e-05      	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  4.5867e-06     	|  22             	|  0.00010091     	|  1.3927e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  4.0947e-06     	|  22             	|  9.0084e-05     	|  1.2433e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.5534e-05     	|  2              	|  7.1068e-05     	|  9.8086e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.5041e-06     	|  22             	|  5.5091e-05     	|  7.6034e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.4905e-06     	|  22             	|  5.4791e-05     	|  7.562e-06      	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.4432e-06     	|  22             	|  5.375e-05      	|  7.4185e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.4206e-06     	|  22             	|  5.3253e-05     	|  7.3498e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.7974e-06     	|  20             	|  3.5947e-05     	|  4.9613e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.6251e-06     	|  22             	|  3.5752e-05     	|  4.9343e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.7349e-06     	|  20             	|  3.4698e-05     	|  4.789e-06      	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.2819e-06     	|  22             	|  2.8202e-05     	|  3.8924e-06     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.392e-05      	|  2              	|  2.7841e-05     	|  3.8425e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  9.8404e-06     	|  2              	|  1.9681e-05     	|  2.7163e-06     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  1.664e-05      	|  1              	|  1.664e-05      	|  2.2966e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  1.2469e-05     	|  1              	|  1.2469e-05     	|  1.721e-06      	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  1.0599e-05     	|  1              	|  1.0599e-05     	|  1.4629e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  5.1749e-06     	|  2              	|  1.035e-05      	|  1.4284e-06     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  9.4203e-06     	|  1              	|  9.4203e-06     	|  1.3002e-06     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  9.1791e-06     	|  1              	|  9.1791e-06     	|  1.2669e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  9.14e-06       	|  1              	|  9.14e-06       	|  1.2615e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  8.53e-06       	|  1              	|  8.53e-06       	|  1.1773e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  4.1053e-06     	|  2              	|  8.2105e-06     	|  1.1332e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.6196e-06     	|  2              	|  7.2392e-06     	|  9.9913e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  6.7595e-06     	|  1              	|  6.7595e-06     	|  9.3293e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  5.3896e-06     	|  1              	|  5.3896e-06     	|  7.4385e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  5.2899e-06     	|  1              	|  5.2899e-06     	|  7.301e-07      	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.5644e-06     	|  2              	|  5.1288e-06     	|  7.0786e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.5197e-06     	|  2              	|  5.0394e-06     	|  6.9552e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.5053e-06     	|  2              	|  5.0105e-06     	|  6.9154e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.5048e-06     	|  2              	|  5.0096e-06     	|  6.9141e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.5001e-06     	|  2              	|  5.0003e-06     	|  6.9012e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.4852e-06     	|  2              	|  4.9705e-06     	|  6.8601e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.2897e-06     	|  1              	|  4.2897e-06     	|  5.9205e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.0401e-06     	|  1              	|  4.0401e-06     	|  5.576e-07      	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.8408e-06     	|  1              	|  3.8408e-06     	|  5.3009e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  3.7001e-06     	|  1              	|  3.7001e-06     	|  5.1068e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.7299e-06     	|  2              	|  3.4599e-06     	|  4.7752e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  1.4999e-06     	|  2              	|  2.9998e-06     	|  4.1402e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.9793e-06     	|  1              	|  2.9793e-06     	|  4.1119e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  2.9001e-06     	|  1              	|  2.9001e-06     	|  4.0027e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.9001e-06     	|  1              	|  2.9001e-06     	|  4.0027e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.8107e-06     	|  1              	|  2.8107e-06     	|  3.8793e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  2.8005e-06     	|  1              	|  2.8005e-06     	|  3.8652e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  1.3597e-06     	|  2              	|  2.7195e-06     	|  3.7533e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.6505e-06     	|  1              	|  2.6505e-06     	|  3.6582e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  2.4894e-06     	|  1              	|  2.4894e-06     	|  3.4358e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  2.4308e-06     	|  1              	|  2.4308e-06     	|  3.3549e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.4103e-06     	|  1              	|  2.4103e-06     	|  3.3266e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.3395e-06     	|  1              	|  2.3395e-06     	|  3.2289e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.3004e-06     	|  1              	|  2.3004e-06     	|  3.1749e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.2901e-06     	|  1              	|  2.2901e-06     	|  3.1608e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.2305e-06     	|  1              	|  2.2305e-06     	|  3.0785e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  2.1802e-06     	|  1              	|  2.1802e-06     	|  3.0091e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.17e-06       	|  1              	|  2.17e-06       	|  2.9949e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  2.0796e-06     	|  1              	|  2.0796e-06     	|  2.8703e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.0303e-06     	|  1              	|  2.0303e-06     	|  2.8021e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9502e-06     	|  1              	|  1.9502e-06     	|  2.6916e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.9297e-06     	|  1              	|  1.9297e-06     	|  2.6633e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  1.9297e-06     	|  1              	|  1.9297e-06     	|  2.6633e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  1.709e-06      	|  1              	|  1.709e-06      	|  2.3587e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.6708e-06     	|  1              	|  1.6708e-06     	|  2.306e-07      	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.6401e-06     	|  1              	|  1.6401e-06     	|  2.2636e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.3802e-06     	|  1              	|  1.3802e-06     	|  1.9049e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.3104e-06     	|  1              	|  1.3104e-06     	|  1.8085e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.1306e-06     	|  1              	|  1.1306e-06     	|  1.5605e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
