FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881521         	|  710.41         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  32.703         	|  20             	|  654.06         	|  92.068         	|
|  run_training_batch                                                                                                                                                                                         	|  0.018747       	|  20000          	|  374.95         	|  52.778         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.01829        	|  20000          	|  365.8          	|  51.492         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0096586      	|  20000          	|  193.17         	|  27.191         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0094084      	|  20000          	|  188.17         	|  26.487         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0049762      	|  20000          	|  99.524         	|  14.009         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.81822        	|  66             	|  54.003         	|  7.6016         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  39.487         	|  1              	|  39.487         	|  5.5583         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.0007064      	|  20000          	|  14.128         	|  1.9887         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00022043     	|  20066          	|  4.4231         	|  0.62261        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00014002     	|  20066          	|  2.8096         	|  0.39549        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.13306        	|  20             	|  2.6611         	|  0.37459        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.018193       	|  66             	|  1.2007         	|  0.16902        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  4.1192e-05     	|  20000          	|  0.82385        	|  0.11597        	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.3935e-05     	|  20000          	|  0.47871        	|  0.067384       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.2109e-05     	|  20000          	|  0.44218        	|  0.062243       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  6.0117e-06     	|  20000          	|  0.12023        	|  0.016925       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.6831e-06     	|  20000          	|  0.11366        	|  0.015999       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  4.7762e-06     	|  20000          	|  0.095525       	|  0.013446       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  4.0793e-06     	|  20000          	|  0.081586       	|  0.011484       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  4.0542e-06     	|  20000          	|  0.081083       	|  0.011414       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.040234       	|  2              	|  0.080468       	|  0.011327       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.9517e-06     	|  20000          	|  0.079033       	|  0.011125       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.6707e-06     	|  20000          	|  0.073413       	|  0.010334       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  3.6429e-06     	|  20000          	|  0.072859       	|  0.010256       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.482e-06      	|  20000          	|  0.069641       	|  0.0098028      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.4577e-06     	|  20000          	|  0.069155       	|  0.0097344      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  3.2001e-06     	|  20000          	|  0.064001       	|  0.009009       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  3.1755e-06     	|  20000          	|  0.063511       	|  0.00894        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.9243e-06     	|  20000          	|  0.058487       	|  0.0082328      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.9147e-06     	|  20000          	|  0.058294       	|  0.0082057      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.8335e-06     	|  20000          	|  0.05667        	|  0.007977       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.8232e-06     	|  20000          	|  0.056463       	|  0.0079479      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.8215e-06     	|  20000          	|  0.056429       	|  0.0079432      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.7172e-06     	|  20000          	|  0.054344       	|  0.0076497      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.6472e-06     	|  20000          	|  0.052943       	|  0.0074524      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.6291e-06     	|  20000          	|  0.052582       	|  0.0074016      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  2.5871e-06     	|  20066          	|  0.051914       	|  0.0073075      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.5e-06        	|  20000          	|  0.05           	|  0.0070381      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.2503e-06     	|  20000          	|  0.045005       	|  0.006335       	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  2.1073e-06     	|  20066          	|  0.042285       	|  0.0059521      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.8496e-06     	|  20000          	|  0.036992       	|  0.0052071      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.8432e-06     	|  20000          	|  0.036863       	|  0.005189       	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.7989e-06     	|  20000          	|  0.035977       	|  0.0050642      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.7649e-06     	|  20000          	|  0.035298       	|  0.0049686      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00051392     	|  66             	|  0.033919       	|  0.0047745      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.659e-06      	|  20000          	|  0.03318        	|  0.0046705      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.5152e-06     	|  20000          	|  0.030304       	|  0.0042657      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.4142e-06     	|  20000          	|  0.028284       	|  0.0039813      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.3369e-06     	|  20000          	|  0.026737       	|  0.0037636      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.2204e-06     	|  20000          	|  0.024408       	|  0.0034358      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.012092       	|  1              	|  0.012092       	|  0.0017021      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00057146     	|  20             	|  0.011429       	|  0.0016088      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0039316      	|  2              	|  0.0078631      	|  0.0011068      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00030685     	|  20             	|  0.0061371      	|  0.00086387     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.4971e-05     	|  66             	|  0.0029681      	|  0.0004178      	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.0001305      	|  20             	|  0.0026101      	|  0.0003674      	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00060667     	|  2              	|  0.0012133      	|  0.00017079     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.0006         	|  2              	|  0.0012         	|  0.00016892     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.0005499      	|  2              	|  0.0010998      	|  0.00015481     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00039042     	|  2              	|  0.00078084     	|  0.00010991     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00074274     	|  1              	|  0.00074274     	|  0.00010455     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  6.6471e-06     	|  66             	|  0.00043871     	|  6.1754e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00037346     	|  1              	|  0.00037346     	|  5.2569e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00033397     	|  1              	|  0.00033397     	|  4.701e-05      	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00029223     	|  1              	|  0.00029223     	|  4.1135e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.9126e-06     	|  66             	|  0.00025823     	|  3.635e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.403e-06      	|  66             	|  0.0002246      	|  3.1615e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.3264e-06     	|  66             	|  0.00021954     	|  3.0903e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.1377e-06     	|  66             	|  0.00020709     	|  2.915e-05      	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.1553e-06     	|  66             	|  0.00014225     	|  2.0024e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  6.7125e-06     	|  20             	|  0.00013425     	|  1.8897e-05     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  6.0948e-06     	|  22             	|  0.00013409     	|  1.8874e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.1815e-06     	|  20             	|  0.00012363     	|  1.7403e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.8134e-06     	|  66             	|  0.00011968     	|  1.6847e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.3321e-06     	|  66             	|  8.7918e-05     	|  1.2376e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  3.7874e-06     	|  22             	|  8.3322e-05     	|  1.1729e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.6836e-06     	|  22             	|  8.1039e-05     	|  1.1407e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.6082e-06     	|  22             	|  7.9379e-05     	|  1.1174e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.4574e-05     	|  2              	|  6.9148e-05     	|  9.7335e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  6.6599e-05     	|  1              	|  6.6599e-05     	|  9.3747e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.2395e-06     	|  20             	|  6.4789e-05     	|  9.1199e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.6102e-06     	|  22             	|  5.7425e-05     	|  8.0834e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.2916e-06     	|  22             	|  5.0415e-05     	|  7.0966e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.2803e-06     	|  22             	|  5.0167e-05     	|  7.0616e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.8751e-06     	|  20             	|  3.7502e-05     	|  5.2788e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.5954e-06     	|  22             	|  3.5099e-05     	|  4.9406e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.4224e-06     	|  20             	|  2.8448e-05     	|  4.0044e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.041e-05      	|  2              	|  2.082e-05      	|  2.9306e-06     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.8169e-05     	|  1              	|  1.8169e-05     	|  2.5575e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.614e-05      	|  1              	|  1.614e-05      	|  2.2719e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  6.265e-06      	|  2              	|  1.253e-05      	|  1.7638e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.185e-05      	|  1              	|  1.185e-05      	|  1.6681e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  4.7102e-06     	|  2              	|  9.4203e-06     	|  1.326e-06      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  4.6594e-06     	|  2              	|  9.3188e-06     	|  1.3117e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  7.7104e-06     	|  1              	|  7.7104e-06     	|  1.0853e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  3.2452e-06     	|  2              	|  6.4904e-06     	|  9.1361e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  3.1241e-06     	|  2              	|  6.2482e-06     	|  8.7952e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  3.1004e-06     	|  2              	|  6.2007e-06     	|  8.7284e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  6.0499e-06     	|  1              	|  6.0499e-06     	|  8.516e-07      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.8703e-06     	|  2              	|  5.7407e-06     	|  8.0807e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  5.4501e-06     	|  1              	|  5.4501e-06     	|  7.6717e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  5.3998e-06     	|  1              	|  5.3998e-06     	|  7.6009e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  5.1996e-06     	|  1              	|  5.1996e-06     	|  7.3191e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  5.18e-06       	|  1              	|  5.18e-06       	|  7.2915e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  4.9192e-06     	|  1              	|  4.9192e-06     	|  6.9245e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.8e-06        	|  1              	|  4.8e-06        	|  6.7567e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.2645e-06     	|  2              	|  4.529e-06      	|  6.3752e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  4.3698e-06     	|  1              	|  4.3698e-06     	|  6.151e-07      	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  4.2301e-06     	|  1              	|  4.2301e-06     	|  5.9544e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  3.8799e-06     	|  1              	|  3.8799e-06     	|  5.4614e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.8803e-06     	|  2              	|  3.7607e-06     	|  5.2936e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  1.865e-06      	|  2              	|  3.7299e-06     	|  5.2504e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.8147e-06     	|  2              	|  3.6294e-06     	|  5.1088e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  3.2093e-06     	|  1              	|  3.2093e-06     	|  4.5176e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.079e-06      	|  1              	|  3.079e-06      	|  4.334e-07      	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  2.97e-06       	|  1              	|  2.97e-06       	|  4.1806e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.9402e-06     	|  1              	|  2.9402e-06     	|  4.1387e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.9402e-06     	|  1              	|  2.9402e-06     	|  4.1387e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.4151e-06     	|  2              	|  2.8303e-06     	|  3.984e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.8098e-06     	|  1              	|  2.8098e-06     	|  3.9552e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  2.7698e-06     	|  1              	|  2.7698e-06     	|  3.8988e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.6999e-06     	|  1              	|  2.6999e-06     	|  3.8005e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.6394e-06     	|  1              	|  2.6394e-06     	|  3.7153e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.5798e-06     	|  1              	|  2.5798e-06     	|  3.6314e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.5108e-06     	|  1              	|  2.5108e-06     	|  3.5343e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.2799e-06     	|  1              	|  2.2799e-06     	|  3.2092e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  2.2696e-06     	|  1              	|  2.2696e-06     	|  3.1948e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.2193e-06     	|  1              	|  2.2193e-06     	|  3.124e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.1998e-06     	|  1              	|  2.1998e-06     	|  3.0965e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.085e-06      	|  2              	|  2.17e-06       	|  3.0545e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.1597e-06     	|  1              	|  2.1597e-06     	|  3.0401e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.1504e-06     	|  1              	|  2.1504e-06     	|  3.027e-07      	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  2.0703e-06     	|  1              	|  2.0703e-06     	|  2.9143e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  2.0703e-06     	|  1              	|  2.0703e-06     	|  2.9143e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  2.0498e-06     	|  1              	|  2.0498e-06     	|  2.8854e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.7099e-06     	|  1              	|  1.7099e-06     	|  2.4069e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.3998e-06     	|  1              	|  1.3998e-06     	|  1.9704e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.3001e-06     	|  1              	|  1.3001e-06     	|  1.8301e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  1.2992e-06     	|  1              	|  1.2992e-06     	|  1.8288e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.2601e-06     	|  1              	|  1.2601e-06     	|  1.7737e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.1893e-06     	|  1              	|  1.1893e-06     	|  1.6741e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
