FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881233         	|  1122.9         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  54.552         	|  20             	|  1091.0         	|  97.165         	|
|  run_training_batch                                                                                                                                                                                         	|  0.043159       	|  20000          	|  863.19         	|  76.874         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.042761       	|  20000          	|  855.22         	|  76.164         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.012684       	|  20000          	|  253.68         	|  22.592         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0085288      	|  20000          	|  170.58         	|  15.191         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0022289      	|  20000          	|  44.578         	|  3.97           	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.44181        	|  48             	|  21.207         	|  1.8886         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  16.285         	|  1              	|  16.285         	|  1.4503         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00068875     	|  20000          	|  13.775         	|  1.2268         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00024094     	|  20048          	|  4.8303         	|  0.43018        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.15722        	|  20             	|  3.1443         	|  0.28003        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00015045     	|  20048          	|  3.0163         	|  0.26862        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.5283e-05     	|  20000          	|  0.70565        	|  0.062844       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.2721e-05     	|  20000          	|  0.45442        	|  0.04047        	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.2282e-05     	|  20000          	|  0.44563        	|  0.039687       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0084996      	|  48             	|  0.40798        	|  0.036334       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.063501       	|  2              	|  0.127          	|  0.01131        	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.3083e-06     	|  20000          	|  0.10617        	|  0.009455       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.9468e-06     	|  20000          	|  0.098936       	|  0.008811       	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  4.8383e-06     	|  20048          	|  0.096998       	|  0.0086385      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  4.8288e-06     	|  20000          	|  0.096576       	|  0.0086009      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  4.4422e-06     	|  20000          	|  0.088845       	|  0.0079123      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.6367e-06     	|  20000          	|  0.072734       	|  0.0064776      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.5251e-06     	|  20000          	|  0.070502       	|  0.0062787      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.5231e-06     	|  20000          	|  0.070462       	|  0.0062752      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  3.4728e-06     	|  20000          	|  0.069456       	|  0.0061856      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.4636e-06     	|  20000          	|  0.069272       	|  0.0061692      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.9928e-06     	|  20000          	|  0.059856       	|  0.0053307      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.9345e-06     	|  20000          	|  0.058691       	|  0.0052269      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.8199e-06     	|  20000          	|  0.056399       	|  0.0050228      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.559e-06      	|  20000          	|  0.051181       	|  0.004558       	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.5219e-06     	|  20000          	|  0.050439       	|  0.004492       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.5157e-06     	|  20000          	|  0.050314       	|  0.0044809      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.4122e-06     	|  20000          	|  0.048244       	|  0.0042965      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.4018e-06     	|  20000          	|  0.048036       	|  0.004278       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.3564e-06     	|  20000          	|  0.047128       	|  0.0041971      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.3211e-06     	|  20000          	|  0.046423       	|  0.0041343      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.3063e-06     	|  20000          	|  0.046126       	|  0.0041078      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.2583e-06     	|  20000          	|  0.045165       	|  0.0040223      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.2432e-06     	|  20000          	|  0.044863       	|  0.0039954      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.7547e-06     	|  20000          	|  0.035093       	|  0.0031253      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.7343e-06     	|  20000          	|  0.034686       	|  0.0030891      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.6761e-06     	|  20000          	|  0.033523       	|  0.0029855      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.6321e-06     	|  20048          	|  0.032719       	|  0.0029139      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.5085e-06     	|  20000          	|  0.03017        	|  0.0026868      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.5013e-06     	|  20000          	|  0.030025       	|  0.002674       	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.4577e-06     	|  20000          	|  0.029153       	|  0.0025963      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.2125e-06     	|  20000          	|  0.02425        	|  0.0021597      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.2054e-06     	|  20000          	|  0.024108       	|  0.002147       	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.1598e-06     	|  20000          	|  0.023195       	|  0.0020657      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.0004824      	|  48             	|  0.023155       	|  0.0020622      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.010417       	|  1              	|  0.010417       	|  0.00092774     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00048962     	|  20             	|  0.0097924      	|  0.00087209     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00039238     	|  20             	|  0.0078477      	|  0.0006989      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0034554      	|  2              	|  0.0069109      	|  0.00061547     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00013119     	|  20             	|  0.0026237      	|  0.00023366     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  3.3681e-05     	|  48             	|  0.0016167      	|  0.00014398     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00055427     	|  2              	|  0.0011085      	|  9.8725e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00039815     	|  2              	|  0.00079629     	|  7.0916e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00036704     	|  2              	|  0.00073407     	|  6.5375e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00069515     	|  1              	|  0.00069515     	|  6.1909e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00031151     	|  2              	|  0.00062301     	|  5.5484e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.000561       	|  1              	|  0.000561       	|  4.9962e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00031202     	|  1              	|  0.00031202     	|  2.7788e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  6.4777e-06     	|  48             	|  0.00031093     	|  2.7691e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.0002832      	|  1              	|  0.0002832      	|  2.5221e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.6798e-06     	|  48             	|  0.00017663     	|  1.573e-05      	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.4021e-06     	|  48             	|  0.0001633      	|  1.4543e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.261e-06      	|  48             	|  0.00015653     	|  1.394e-05      	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  7.5456e-06     	|  20             	|  0.00015091     	|  1.344e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.817e-06      	|  48             	|  0.00013522     	|  1.2042e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  4.5696e-06     	|  22             	|  0.00010053     	|  8.953e-06      	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.9817e-06     	|  20             	|  9.9634e-05     	|  8.8732e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.8852e-06     	|  48             	|  9.049e-05      	|  8.0589e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.9932e-06     	|  22             	|  8.7851e-05     	|  7.8238e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.7426e-06     	|  48             	|  8.3647e-05     	|  7.4494e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.4583e-06     	|  48             	|  6.9997e-05     	|  6.2338e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.4182e-06     	|  20             	|  6.8365e-05     	|  6.0884e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.118e-05      	|  2              	|  6.236e-05      	|  5.5537e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.4987e-06     	|  22             	|  5.497e-05      	|  4.8955e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.4278e-06     	|  22             	|  5.3412e-05     	|  4.7568e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.3691e-06     	|  22             	|  5.2121e-05     	|  4.6417e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.2995e-06     	|  22             	|  5.0589e-05     	|  4.5054e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.6086e-06     	|  22             	|  3.5389e-05     	|  3.1517e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.7559e-06     	|  20             	|  3.5118e-05     	|  3.1276e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.5174e-06     	|  22             	|  3.3383e-05     	|  2.973e-06      	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.6207e-06     	|  20             	|  3.2414e-05     	|  2.8867e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  2.9971e-05     	|  1              	|  2.9971e-05     	|  2.6691e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  5.9451e-06     	|  2              	|  1.189e-05      	|  1.0589e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  5.8403e-06     	|  2              	|  1.1681e-05     	|  1.0403e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  5.0152e-06     	|  2              	|  1.003e-05      	|  8.9328e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  4.7698e-06     	|  2              	|  9.5395e-06     	|  8.4957e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  8.4899e-06     	|  1              	|  8.4899e-06     	|  7.561e-07      	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  7.8697e-06     	|  1              	|  7.8697e-06     	|  7.0086e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  3.8999e-06     	|  2              	|  7.7998e-06     	|  6.9464e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  7.0799e-06     	|  1              	|  7.0799e-06     	|  6.3052e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.4748e-06     	|  2              	|  6.9495e-06     	|  6.1891e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.3551e-06     	|  2              	|  6.7102e-06     	|  5.9759e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  6.6301e-06     	|  1              	|  6.6301e-06     	|  5.9046e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  6.2892e-06     	|  1              	|  6.2892e-06     	|  5.601e-07      	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  5.0394e-06     	|  1              	|  5.0394e-06     	|  4.488e-07      	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  5.0003e-06     	|  1              	|  5.0003e-06     	|  4.4531e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.4801e-06     	|  2              	|  4.9602e-06     	|  4.4175e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  2.4452e-06     	|  2              	|  4.8904e-06     	|  4.3553e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.4345e-06     	|  2              	|  4.869e-06      	|  4.3362e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.3851e-06     	|  2              	|  4.7702e-06     	|  4.2483e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  4.5998e-06     	|  1              	|  4.5998e-06     	|  4.0965e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.5598e-06     	|  1              	|  4.5598e-06     	|  4.0608e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.5504e-06     	|  1              	|  4.5504e-06     	|  4.0525e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.4201e-06     	|  1              	|  4.4201e-06     	|  3.9364e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.0894e-06     	|  2              	|  4.1788e-06     	|  3.7216e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  4.1295e-06     	|  1              	|  4.1295e-06     	|  3.6776e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  1.9702e-06     	|  2              	|  3.9404e-06     	|  3.5093e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  3.8305e-06     	|  1              	|  3.8305e-06     	|  3.4114e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  3.3202e-06     	|  1              	|  3.3202e-06     	|  2.9569e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.7698e-06     	|  1              	|  2.7698e-06     	|  2.4667e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.74e-06       	|  1              	|  2.74e-06       	|  2.4401e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  2.7306e-06     	|  1              	|  2.7306e-06     	|  2.4318e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.6701e-06     	|  1              	|  2.6701e-06     	|  2.3779e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.6682e-06     	|  1              	|  2.6682e-06     	|  2.3763e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  2.6599e-06     	|  1              	|  2.6599e-06     	|  2.3688e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.5602e-06     	|  1              	|  2.5602e-06     	|  2.2801e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  2.5602e-06     	|  1              	|  2.5602e-06     	|  2.2801e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  2.55e-06       	|  1              	|  2.55e-06       	|  2.2709e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.55e-06       	|  1              	|  2.55e-06       	|  2.2709e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.4904e-06     	|  1              	|  2.4904e-06     	|  2.2179e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  2.4103e-06     	|  1              	|  2.4103e-06     	|  2.1465e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.3404e-06     	|  1              	|  2.3404e-06     	|  2.0843e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.3395e-06     	|  1              	|  2.3395e-06     	|  2.0835e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.1607e-06     	|  1              	|  2.1607e-06     	|  1.9242e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.1495e-06     	|  1              	|  2.1495e-06     	|  1.9143e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.0645e-06     	|  2              	|  2.129e-06      	|  1.896e-07      	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.0498e-06     	|  1              	|  2.0498e-06     	|  1.8255e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.0005e-06     	|  1              	|  2.0005e-06     	|  1.7816e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.7397e-06     	|  1              	|  1.7397e-06     	|  1.5493e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.7304e-06     	|  1              	|  1.7304e-06     	|  1.5411e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.6894e-06     	|  1              	|  1.6894e-06     	|  1.5046e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.6587e-06     	|  1              	|  1.6587e-06     	|  1.4772e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.37e-06       	|  1              	|  1.37e-06       	|  1.2201e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.18e-06       	|  1              	|  1.18e-06       	|  1.0509e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.1297e-06     	|  1              	|  1.1297e-06     	|  1.0061e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.0505e-06     	|  1              	|  1.0505e-06     	|  9.3558e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
