FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  880977         	|  1081.1         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  52.973         	|  20             	|  1059.5         	|  98.001         	|
|  run_training_batch                                                                                                                                                                                         	|  0.046869       	|  20000          	|  937.37         	|  86.709         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.046617       	|  20000          	|  932.34         	|  86.243         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.013332       	|  20000          	|  266.63         	|  24.664         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0027662      	|  20000          	|  55.323         	|  5.1175         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  1.3586         	|  32             	|  43.475         	|  4.0216         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0020872      	|  20000          	|  41.744         	|  3.8614         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00047659     	|  20000          	|  9.5317         	|  0.8817         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  7.8173         	|  1              	|  7.8173         	|  0.72311        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.18895        	|  20             	|  3.779          	|  0.34956        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00011795     	|  20032          	|  2.3627         	|  0.21856        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  7.5061e-05     	|  20032          	|  1.5036         	|  0.13909        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.020624       	|  32             	|  0.65996        	|  0.061047       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  1.9186e-05     	|  20000          	|  0.38371        	|  0.035494       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.4164e-05     	|  20000          	|  0.28328        	|  0.026204       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.3462e-05     	|  20000          	|  0.26924        	|  0.024905       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.2323e-06     	|  20000          	|  0.084645       	|  0.0078299      	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.038391       	|  2              	|  0.076783       	|  0.0071026      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.2404e-06     	|  20000          	|  0.064808       	|  0.0059949      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.0997e-06     	|  20000          	|  0.061995       	|  0.0057346      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  2.9259e-06     	|  20000          	|  0.058518       	|  0.005413       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  2.8034e-06     	|  20000          	|  0.056068       	|  0.0051864      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  2.6399e-06     	|  20000          	|  0.052799       	|  0.004884       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.5053e-06     	|  20000          	|  0.050106       	|  0.0046349      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.3915e-06     	|  20000          	|  0.04783        	|  0.0044244      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.3271e-06     	|  20000          	|  0.046543       	|  0.0043053      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.2495e-06     	|  20000          	|  0.04499        	|  0.0041617      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.2138e-06     	|  20000          	|  0.044276       	|  0.0040956      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.1976e-06     	|  20000          	|  0.043953       	|  0.0040657      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.176e-06      	|  20000          	|  0.04352        	|  0.0040257      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.1567e-06     	|  20000          	|  0.043134       	|  0.00399        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.1013e-06     	|  20000          	|  0.042026       	|  0.0038875      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.0525e-06     	|  20000          	|  0.04105        	|  0.0037972      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.0478e-06     	|  20000          	|  0.040956       	|  0.0037885      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.0193e-06     	|  20000          	|  0.040386       	|  0.0037358      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  1.9935e-06     	|  20000          	|  0.039871       	|  0.0036881      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  1.991e-06      	|  20000          	|  0.039821       	|  0.0036835      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  1.957e-06      	|  20000          	|  0.03914        	|  0.0036206      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  1.9117e-06     	|  20000          	|  0.038234       	|  0.0035367      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.3095e-06     	|  20032          	|  0.026231       	|  0.0024264      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.0993e-06     	|  20000          	|  0.021986       	|  0.0020338      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.0285e-06     	|  20032          	|  0.020602       	|  0.0019058      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  9.8273e-07     	|  20000          	|  0.019655       	|  0.0018181      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  9.2519e-07     	|  20000          	|  0.018504       	|  0.0017116      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  8.7202e-07     	|  20000          	|  0.01744        	|  0.0016133      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  8.2156e-07     	|  20000          	|  0.016431       	|  0.0015199      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  8.1984e-07     	|  20000          	|  0.016397       	|  0.0015167      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  8.1129e-07     	|  20000          	|  0.016226       	|  0.0015009      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  8.1076e-07     	|  20000          	|  0.016215       	|  0.0014999      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00048267     	|  32             	|  0.015446       	|  0.0014287      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  7.4245e-07     	|  20000          	|  0.014849       	|  0.0013736      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0055709      	|  2              	|  0.011142       	|  0.0010306      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0094174      	|  1              	|  0.0094174      	|  0.00087113     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00023527     	|  20             	|  0.0047055      	|  0.00043527     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00023414     	|  20             	|  0.0046828      	|  0.00043317     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00018408     	|  20             	|  0.0036815      	|  0.00034055     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  9.7461e-05     	|  32             	|  0.0031188      	|  0.00028849     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00060476     	|  2              	|  0.0012095      	|  0.00011188     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00051912     	|  2              	|  0.0010382      	|  9.604e-05      	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00044455     	|  2              	|  0.00088909     	|  8.2243e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00064607     	|  1              	|  0.00064607     	|  5.9763e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.0002829      	|  2              	|  0.0005658      	|  5.2338e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00032752     	|  1              	|  0.00032752     	|  3.0296e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00026564     	|  1              	|  0.00026564     	|  2.4572e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.0002257      	|  1              	|  0.0002257      	|  2.0878e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  6.9806e-06     	|  32             	|  0.00022338     	|  2.0663e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  4.4905e-06     	|  32             	|  0.00014369     	|  1.3292e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  6.8575e-05     	|  2              	|  0.00013715     	|  1.2687e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.7446e-06     	|  32             	|  0.00011983     	|  1.1084e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  5.5217e-06     	|  20             	|  0.00011043     	|  1.0215e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.3077e-06     	|  32             	|  0.00010585     	|  9.7911e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.2149e-06     	|  32             	|  0.00010288     	|  9.5164e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.477e-06      	|  20             	|  8.9539e-05     	|  8.2826e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.0113e-06     	|  22             	|  6.625e-05      	|  6.1282e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  6.5691e-05     	|  1              	|  6.5691e-05     	|  6.0765e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.9574e-06     	|  32             	|  6.2638e-05     	|  5.7941e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  2.7453e-06     	|  22             	|  6.0397e-05     	|  5.5869e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.8872e-06     	|  32             	|  6.0392e-05     	|  5.5863e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.6854e-06     	|  20             	|  5.3708e-05     	|  4.9681e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  1.9528e-06     	|  22             	|  4.2961e-05     	|  3.974e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  1.9405e-06     	|  22             	|  4.2691e-05     	|  3.949e-06      	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  1.94e-06       	|  22             	|  4.2681e-05     	|  3.948e-06      	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  1.9168e-06     	|  22             	|  4.217e-05      	|  3.9008e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.312e-06      	|  32             	|  4.1983e-05     	|  3.8835e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.3521e-06     	|  22             	|  2.9746e-05     	|  2.7516e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.2936e-06     	|  22             	|  2.8458e-05     	|  2.6325e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.4141e-06     	|  20             	|  2.8282e-05     	|  2.6162e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.3728e-06     	|  20             	|  2.7456e-05     	|  2.5398e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  1.02e-05       	|  2              	|  2.04e-05       	|  1.887e-06      	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.9751e-05     	|  1              	|  1.9751e-05     	|  1.827e-06      	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.248e-05      	|  1              	|  1.248e-05      	|  1.1544e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  5.1605e-06     	|  2              	|  1.0321e-05     	|  9.5471e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  4.8596e-06     	|  2              	|  9.7193e-06     	|  8.9905e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  8.5207e-06     	|  1              	|  8.5207e-06     	|  7.8818e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.4655e-06     	|  2              	|  6.9309e-06     	|  6.4112e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  6.9197e-06     	|  1              	|  6.9197e-06     	|  6.4009e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  6.3805e-06     	|  1              	|  6.3805e-06     	|  5.9021e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  5.9204e-06     	|  1              	|  5.9204e-06     	|  5.4765e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  5.4203e-06     	|  1              	|  5.4203e-06     	|  5.0139e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.2396e-06     	|  1              	|  5.2396e-06     	|  4.8468e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  5.1996e-06     	|  1              	|  5.1996e-06     	|  4.8097e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  4.9099e-06     	|  1              	|  4.9099e-06     	|  4.5418e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.4252e-06     	|  2              	|  4.8503e-06     	|  4.4867e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.4196e-06     	|  2              	|  4.8392e-06     	|  4.4763e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.7507e-06     	|  1              	|  4.7507e-06     	|  4.3945e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.53e-06       	|  1              	|  4.53e-06       	|  4.1903e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.4899e-06     	|  1              	|  4.4899e-06     	|  4.1533e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.2999e-06     	|  1              	|  4.2999e-06     	|  3.9775e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.2804e-06     	|  1              	|  4.2804e-06     	|  3.9594e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.2794e-06     	|  1              	|  4.2794e-06     	|  3.9586e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  3.9889e-06     	|  1              	|  3.9889e-06     	|  3.6898e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  1.9604e-06     	|  2              	|  3.9209e-06     	|  3.6269e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  1.9399e-06     	|  2              	|  3.8799e-06     	|  3.589e-07      	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  3.8799e-06     	|  1              	|  3.8799e-06     	|  3.589e-07      	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  3.7905e-06     	|  1              	|  3.7905e-06     	|  3.5063e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  3.7905e-06     	|  1              	|  3.7905e-06     	|  3.5063e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  3.7206e-06     	|  1              	|  3.7206e-06     	|  3.4417e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  1.8352e-06     	|  2              	|  3.6703e-06     	|  3.3951e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.8352e-06     	|  2              	|  3.6703e-06     	|  3.3951e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  3.4906e-06     	|  1              	|  3.4906e-06     	|  3.2289e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  3.4394e-06     	|  1              	|  3.4394e-06     	|  3.1815e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.4901e-06     	|  2              	|  2.9802e-06     	|  2.7568e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  2.7902e-06     	|  1              	|  2.7902e-06     	|  2.581e-07      	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.6897e-06     	|  1              	|  2.6897e-06     	|  2.488e-07      	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  2.4894e-06     	|  1              	|  2.4894e-06     	|  2.3028e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.4699e-06     	|  1              	|  2.4699e-06     	|  2.2847e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.4308e-06     	|  1              	|  2.4308e-06     	|  2.2485e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.3702e-06     	|  1              	|  2.3702e-06     	|  2.1925e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.1004e-06     	|  2              	|  2.2007e-06     	|  2.0357e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.1998e-06     	|  1              	|  2.1998e-06     	|  2.0348e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.17e-06       	|  1              	|  2.17e-06       	|  2.0073e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.1392e-06     	|  1              	|  2.1392e-06     	|  1.9788e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.1197e-06     	|  1              	|  2.1197e-06     	|  1.9608e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.03e-06       	|  2              	|  2.0601e-06     	|  1.9056e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  1.0151e-06     	|  2              	|  2.0303e-06     	|  1.8781e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.98e-06       	|  1              	|  1.98e-06       	|  1.8315e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.8701e-06     	|  1              	|  1.8701e-06     	|  1.7299e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.6196e-06     	|  1              	|  1.6196e-06     	|  1.4981e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.5106e-06     	|  1              	|  1.5106e-06     	|  1.3973e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.2303e-06     	|  1              	|  1.2303e-06     	|  1.138e-07      	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.0598e-06     	|  1              	|  1.0598e-06     	|  9.8038e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  9.5926e-07     	|  1              	|  9.5926e-07     	|  8.8734e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
