FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881297         	|  3035.8         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  149.69         	|  20             	|  2993.9         	|  98.619         	|
|  run_training_batch                                                                                                                                                                                         	|  0.14207        	|  20000          	|  2841.5         	|  93.598         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.14166        	|  20000          	|  2833.3         	|  93.327         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.040624       	|  20000          	|  812.48         	|  26.763         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0050139      	|  20000          	|  100.28         	|  3.3032         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0039216      	|  20000          	|  78.431         	|  2.5835         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.91656        	|  52             	|  47.661         	|  1.57           	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  29.84          	|  1              	|  29.84          	|  0.98293        	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00050425     	|  20000          	|  10.085         	|  0.3322         	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.23421        	|  20             	|  4.6842         	|  0.1543         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00013649     	|  20052          	|  2.737          	|  0.090155       	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  8.9508e-05     	|  20052          	|  1.7948         	|  0.059121       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.8631e-05     	|  20000          	|  0.57262        	|  0.018862       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.7219e-05     	|  20000          	|  0.34439        	|  0.011344       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.7027e-05     	|  20000          	|  0.34053        	|  0.011217       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.006416       	|  52             	|  0.33363        	|  0.01099        	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.065447       	|  2              	|  0.13089        	|  0.0043117      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.602e-06      	|  20000          	|  0.09204        	|  0.0030318      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.813e-06      	|  20000          	|  0.07626        	|  0.002512       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.2404e-06     	|  20000          	|  0.064808       	|  0.0021348      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.1437e-06     	|  20000          	|  0.062873       	|  0.002071       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  2.9379e-06     	|  20000          	|  0.058758       	|  0.0019355      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  2.9332e-06     	|  20000          	|  0.058664       	|  0.0019324      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.8703e-06     	|  20000          	|  0.057406       	|  0.001891       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.8179e-06     	|  20000          	|  0.056358       	|  0.0018564      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.5021e-06     	|  20000          	|  0.050042       	|  0.0016484      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.4374e-06     	|  20000          	|  0.048749       	|  0.0016058      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.3689e-06     	|  20000          	|  0.047378       	|  0.0015606      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.2876e-06     	|  20000          	|  0.045753       	|  0.0015071      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.2814e-06     	|  20000          	|  0.045628       	|  0.001503       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.2189e-06     	|  20000          	|  0.044377       	|  0.0014618      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.1958e-06     	|  20000          	|  0.043916       	|  0.0014466      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.1592e-06     	|  20000          	|  0.043185       	|  0.0014225      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.1118e-06     	|  20000          	|  0.042235       	|  0.0013912      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.1042e-06     	|  20000          	|  0.042084       	|  0.0013863      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.007e-06      	|  20000          	|  0.040139       	|  0.0013222      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  1.956e-06      	|  20000          	|  0.03912        	|  0.0012886      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  1.934e-06      	|  20000          	|  0.038679       	|  0.0012741      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  1.9214e-06     	|  20000          	|  0.038428       	|  0.0012658      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.488e-06      	|  20052          	|  0.029837       	|  0.00098284     	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.2106e-06     	|  20000          	|  0.024211       	|  0.00079752     	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00046134     	|  52             	|  0.02399        	|  0.00079023     	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.1495e-06     	|  20000          	|  0.02299        	|  0.00075729     	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.1428e-06     	|  20052          	|  0.022916       	|  0.00075484     	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.1213e-06     	|  20000          	|  0.022425       	|  0.00073869     	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.0909e-06     	|  20000          	|  0.021818       	|  0.0007187      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.0862e-06     	|  20000          	|  0.021725       	|  0.00071562     	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  9.6592e-07     	|  20000          	|  0.019318       	|  0.00063635     	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  9.6565e-07     	|  20000          	|  0.019313       	|  0.00063617     	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  9.6032e-07     	|  20000          	|  0.019206       	|  0.00063266     	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  9.186e-07      	|  20000          	|  0.018372       	|  0.00060517     	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.005067       	|  2              	|  0.010134       	|  0.00033381     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0057321      	|  1              	|  0.0057321      	|  0.00018882     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00026572     	|  20             	|  0.0053145      	|  0.00017506     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00023064     	|  20             	|  0.0046128      	|  0.00015195     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  5.4106e-05     	|  52             	|  0.0028135      	|  9.2678e-05     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00010868     	|  20             	|  0.0021735      	|  7.1596e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.0005787      	|  2              	|  0.0011574      	|  3.8125e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.0005692      	|  2              	|  0.0011384      	|  3.7499e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00051618     	|  2              	|  0.0010324      	|  3.4006e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00081495     	|  1              	|  0.00081495     	|  2.6845e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00052775     	|  1              	|  0.00052775     	|  1.7384e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00024464     	|  2              	|  0.00048928     	|  1.6117e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  6.9989e-06     	|  52             	|  0.00036395     	|  1.1988e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00030025     	|  1              	|  0.00030025     	|  9.8902e-06     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00028286     	|  1              	|  0.00028286     	|  9.3173e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  5.0338e-06     	|  52             	|  0.00026176     	|  8.6223e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.1544e-06     	|  52             	|  0.00016403     	|  5.4032e-06     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  2.9914e-06     	|  52             	|  0.00015555     	|  5.1239e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.9011e-06     	|  52             	|  0.00015086     	|  4.9692e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.9723e-06     	|  20             	|  0.00013945     	|  4.5934e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.151e-06      	|  52             	|  0.00011185     	|  3.6844e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  4.8295e-05     	|  2              	|  9.6589e-05     	|  3.1817e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.7679e-06     	|  20             	|  9.5358e-05     	|  3.1411e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.7806e-06     	|  52             	|  9.259e-05      	|  3.0499e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.2917e-06     	|  22             	|  7.2418e-05     	|  2.3854e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.305e-06      	|  52             	|  6.7859e-05     	|  2.2353e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  2.9758e-06     	|  22             	|  6.5468e-05     	|  2.1565e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.7699e-06     	|  20             	|  5.5398e-05     	|  1.8248e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.1577e-06     	|  22             	|  4.747e-05      	|  1.5636e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.1299e-06     	|  22             	|  4.6859e-05     	|  1.5435e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  4.559e-05      	|  1              	|  4.559e-05      	|  1.5017e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.0699e-06     	|  22             	|  4.5538e-05     	|  1.5e-06        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.0617e-06     	|  22             	|  4.5356e-05     	|  1.494e-06      	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.4513e-06     	|  22             	|  3.1929e-05     	|  1.0517e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.5955e-06     	|  20             	|  3.1911e-05     	|  1.0511e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.4105e-06     	|  22             	|  3.1031e-05     	|  1.0222e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.362e-05      	|  2              	|  2.7239e-05     	|  8.9726e-07     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.343e-06      	|  20             	|  2.686e-05      	|  8.8478e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  5.865e-06      	|  2              	|  1.173e-05      	|  3.8639e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.088e-05      	|  1              	|  1.088e-05      	|  3.5838e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  5.3798e-06     	|  2              	|  1.076e-05      	|  3.5442e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  5.35e-06       	|  2              	|  1.07e-05       	|  3.5246e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  4.9602e-06     	|  2              	|  9.9204e-06     	|  3.2678e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  8.4601e-06     	|  1              	|  8.4601e-06     	|  2.7868e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  7.2801e-06     	|  1              	|  7.2801e-06     	|  2.3981e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.4454e-06     	|  2              	|  6.8909e-06     	|  2.2699e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  3.3849e-06     	|  2              	|  6.7698e-06     	|  2.23e-07       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  3.2103e-06     	|  2              	|  6.4205e-06     	|  2.1149e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  5.8804e-06     	|  1              	|  5.8804e-06     	|  1.937e-07      	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  5.6401e-06     	|  1              	|  5.6401e-06     	|  1.8578e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.8098e-06     	|  2              	|  5.6196e-06     	|  1.8511e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.7446e-06     	|  2              	|  5.4892e-06     	|  1.8082e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.5854e-06     	|  2              	|  5.1707e-06     	|  1.7032e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  5.0906e-06     	|  1              	|  5.0906e-06     	|  1.6768e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.9304e-06     	|  1              	|  4.9304e-06     	|  1.6241e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  2.3949e-06     	|  2              	|  4.7898e-06     	|  1.5778e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  4.7004e-06     	|  1              	|  4.7004e-06     	|  1.5483e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.3399e-06     	|  2              	|  4.6799e-06     	|  1.5416e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.57e-06       	|  1              	|  4.57e-06       	|  1.5054e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.2692e-06     	|  1              	|  4.2692e-06     	|  1.4063e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  3.7402e-06     	|  1              	|  3.7402e-06     	|  1.232e-07      	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  3.5707e-06     	|  1              	|  3.5707e-06     	|  1.1762e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.7849e-06     	|  2              	|  3.5698e-06     	|  1.1759e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  3.4999e-06     	|  1              	|  3.4999e-06     	|  1.1529e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  3.4794e-06     	|  1              	|  3.4794e-06     	|  1.1461e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.4105e-06     	|  1              	|  3.4105e-06     	|  1.1234e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.39e-06       	|  1              	|  3.39e-06       	|  1.1167e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  3.35e-06       	|  1              	|  3.35e-06       	|  1.1035e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  3.3099e-06     	|  1              	|  3.3099e-06     	|  1.0903e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  3.2997e-06     	|  1              	|  3.2997e-06     	|  1.0869e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  3.1199e-06     	|  1              	|  3.1199e-06     	|  1.0277e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.9197e-06     	|  1              	|  2.9197e-06     	|  9.6175e-08     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.8703e-06     	|  1              	|  2.8703e-06     	|  9.4549e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.8396e-06     	|  1              	|  2.8396e-06     	|  9.3537e-08     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  2.6794e-06     	|  1              	|  2.6794e-06     	|  8.826e-08      	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.5798e-06     	|  1              	|  2.5798e-06     	|  8.4978e-08     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.3199e-06     	|  1              	|  2.3199e-06     	|  7.6418e-08     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.1597e-06     	|  1              	|  2.1597e-06     	|  7.1142e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.1495e-06     	|  1              	|  2.1495e-06     	|  7.0804e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.1299e-06     	|  1              	|  2.1299e-06     	|  7.016e-08      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.1094e-06     	|  1              	|  2.1094e-06     	|  6.9485e-08     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.0601e-06     	|  1              	|  2.0601e-06     	|  6.7859e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.0396e-06     	|  1              	|  2.0396e-06     	|  6.7184e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9902e-06     	|  1              	|  1.9902e-06     	|  6.5558e-08     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  1.9595e-06     	|  1              	|  1.9595e-06     	|  6.4546e-08     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.8803e-06     	|  1              	|  1.8803e-06     	|  6.1938e-08     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.6894e-06     	|  1              	|  1.6894e-06     	|  5.565e-08      	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.6792e-06     	|  1              	|  1.6792e-06     	|  5.5312e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.5507e-06     	|  1              	|  1.5507e-06     	|  5.1079e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.37e-06       	|  1              	|  1.37e-06       	|  4.5127e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.1194e-06     	|  1              	|  1.1194e-06     	|  3.6875e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
