FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881073         	|  251.12         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  11.561         	|  20             	|  231.22         	|  92.074         	|
|  run_training_batch                                                                                                                                                                                         	|  0.0088903      	|  20000          	|  177.81         	|  70.804         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.0086411      	|  20000          	|  172.82         	|  68.819         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0041499      	|  20000          	|  82.998         	|  33.05          	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.002245       	|  20000          	|  44.9           	|  17.879         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0010591      	|  20000          	|  21.182         	|  8.4348         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.25378        	|  38             	|  9.6435         	|  3.8401         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00042759     	|  20000          	|  8.5519         	|  3.4054         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  7.3611         	|  1              	|  7.3611         	|  2.9313         	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.18118        	|  20             	|  3.6237         	|  1.443          	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00012061     	|  20038          	|  2.4167         	|  0.96237        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  7.595e-05      	|  20038          	|  1.5219         	|  0.60603        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.024672       	|  38             	|  0.93754        	|  0.37334        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.0313e-05     	|  20000          	|  0.40625        	|  0.16177        	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.373e-05      	|  20000          	|  0.27461        	|  0.10935        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.3638e-05     	|  20000          	|  0.27276        	|  0.10862        	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.080859       	|  2              	|  0.16172        	|  0.064398       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.5006e-06     	|  20000          	|  0.090012       	|  0.035844       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.041825       	|  2              	|  0.08365        	|  0.03331        	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.5908e-06     	|  20000          	|  0.071817       	|  0.028598       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.5111e-06     	|  20000          	|  0.070221       	|  0.027963       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.4675e-06     	|  20000          	|  0.069351       	|  0.027616       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.4344e-06     	|  20000          	|  0.068688       	|  0.027352       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.2091e-06     	|  20000          	|  0.064182       	|  0.025558       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.7277e-06     	|  20000          	|  0.054554       	|  0.021724       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.6943e-06     	|  20000          	|  0.053887       	|  0.021458       	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.6923e-06     	|  20000          	|  0.053846       	|  0.021442       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.6508e-06     	|  20000          	|  0.053016       	|  0.021111       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.5942e-06     	|  20000          	|  0.051883       	|  0.02066        	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.5877e-06     	|  20000          	|  0.051754       	|  0.020609       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.5033e-06     	|  20000          	|  0.050065       	|  0.019936       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.4892e-06     	|  20000          	|  0.049784       	|  0.019824       	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.4698e-06     	|  20000          	|  0.049396       	|  0.01967        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.4443e-06     	|  20000          	|  0.048885       	|  0.019466       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.439e-06      	|  20000          	|  0.04878        	|  0.019425       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.4384e-06     	|  20000          	|  0.048768       	|  0.01942        	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.4261e-06     	|  20000          	|  0.048521       	|  0.019322       	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.3586e-06     	|  20000          	|  0.047173       	|  0.018785       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.24e-06       	|  20000          	|  0.0448         	|  0.01784        	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.0846e-06     	|  20000          	|  0.041692       	|  0.016602       	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.2731e-06     	|  20000          	|  0.025461       	|  0.010139       	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.2419e-06     	|  20038          	|  0.024885       	|  0.0099095      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.2076e-06     	|  20000          	|  0.024152       	|  0.0096175      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.0807e-06     	|  20038          	|  0.021656       	|  0.0086234      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  9.8272e-07     	|  20000          	|  0.019654       	|  0.0078265      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  9.5023e-07     	|  20000          	|  0.019005       	|  0.0075678      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  9.3094e-07     	|  20000          	|  0.018619       	|  0.0074142      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  8.7071e-07     	|  20000          	|  0.017414       	|  0.0069345      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  8.6487e-07     	|  20000          	|  0.017297       	|  0.006888       	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  8.4752e-07     	|  20000          	|  0.01695        	|  0.0067498      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  8.3648e-07     	|  20000          	|  0.01673        	|  0.0066619      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00026979     	|  38             	|  0.010252       	|  0.0040824      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00024977     	|  20             	|  0.0049954      	|  0.0019892      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00024871     	|  20             	|  0.0049741      	|  0.0019807      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0048688      	|  1              	|  0.0048688      	|  0.0019388      	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00019311     	|  20             	|  0.0038622      	|  0.001538       	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  6.3526e-05     	|  38             	|  0.002414       	|  0.00096127     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00051106     	|  2              	|  0.0010221      	|  0.00040702     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00047361     	|  2              	|  0.00094721     	|  0.00037719     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00043987     	|  2              	|  0.00087973     	|  0.00035032     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.0003272      	|  2              	|  0.0006544      	|  0.00026059     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00056122     	|  1              	|  0.00056122     	|  0.00022348     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00043463     	|  1              	|  0.00043463     	|  0.00017307     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00029539     	|  1              	|  0.00029539     	|  0.00011763     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00026502     	|  1              	|  0.00026502     	|  0.00010553     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  4.2026e-06     	|  38             	|  0.0001597      	|  6.3594e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.0309e-06     	|  20             	|  0.00012062     	|  4.8031e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  2.7915e-06     	|  38             	|  0.00010608     	|  4.2241e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  4.5568e-06     	|  22             	|  0.00010025     	|  3.992e-05      	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.7653e-06     	|  20             	|  9.5306e-05     	|  3.7952e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.2516e-06     	|  38             	|  8.5561e-05     	|  3.4071e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.124e-06      	|  38             	|  8.0712e-05     	|  3.214e-05      	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.6499e-06     	|  22             	|  8.0297e-05     	|  3.1975e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.667e-05      	|  2              	|  7.334e-05      	|  2.9205e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.9591e-06     	|  20             	|  5.9182e-05     	|  2.3567e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  1.5476e-06     	|  38             	|  5.8808e-05     	|  2.3418e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.2865e-06     	|  22             	|  5.0304e-05     	|  2.0031e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.2664e-06     	|  22             	|  4.9861e-05     	|  1.9855e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.2557e-06     	|  22             	|  4.9625e-05     	|  1.9761e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.1954e-06     	|  22             	|  4.8298e-05     	|  1.9233e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.1473e-06     	|  38             	|  4.3597e-05     	|  1.7361e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  9.0919e-07     	|  38             	|  3.4549e-05     	|  1.3758e-05     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.6806e-06     	|  20             	|  3.3611e-05     	|  1.3384e-05     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.4364e-06     	|  22             	|  3.1601e-05     	|  1.2584e-05     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.3876e-06     	|  22             	|  3.0528e-05     	|  1.2156e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  7.9903e-07     	|  38             	|  3.0363e-05     	|  1.2091e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  2.863e-05      	|  1              	|  2.863e-05      	|  1.1401e-05     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.2601e-06     	|  20             	|  2.5203e-05     	|  1.0036e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  8.1197e-06     	|  2              	|  1.6239e-05     	|  6.4667e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  6.6501e-06     	|  2              	|  1.33e-05       	|  5.2963e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.242e-05      	|  1              	|  1.242e-05      	|  4.9458e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.141e-05      	|  1              	|  1.141e-05      	|  4.5434e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  5.4752e-06     	|  2              	|  1.095e-05      	|  4.3606e-06     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  1.062e-05      	|  1              	|  1.062e-05      	|  4.2289e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  9.1502e-06     	|  1              	|  9.1502e-06     	|  3.6437e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  8.5393e-06     	|  1              	|  8.5393e-06     	|  3.4004e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  8.4899e-06     	|  1              	|  8.4899e-06     	|  3.3808e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  3.9251e-06     	|  2              	|  7.8501e-06     	|  3.126e-06      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.6401e-06     	|  2              	|  7.2801e-06     	|  2.899e-06      	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  6.5099e-06     	|  1              	|  6.5099e-06     	|  2.5923e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  6.4597e-06     	|  1              	|  6.4597e-06     	|  2.5723e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  5.7193e-06     	|  1              	|  5.7193e-06     	|  2.2775e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.8452e-06     	|  2              	|  5.6904e-06     	|  2.266e-06      	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.6503e-06     	|  1              	|  5.6503e-06     	|  2.25e-06       	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.7702e-06     	|  2              	|  5.5404e-06     	|  2.2063e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  4.9006e-06     	|  1              	|  4.9006e-06     	|  1.9515e-06     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  4.8904e-06     	|  1              	|  4.8904e-06     	|  1.9474e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.76e-06       	|  1              	|  4.76e-06       	|  1.8955e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.7591e-06     	|  1              	|  4.7591e-06     	|  1.8951e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  4.6296e-06     	|  1              	|  4.6296e-06     	|  1.8436e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.5197e-06     	|  1              	|  4.5197e-06     	|  1.7998e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.4405e-06     	|  1              	|  4.4405e-06     	|  1.7683e-06     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.38e-06       	|  1              	|  4.38e-06       	|  1.7442e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.0601e-06     	|  2              	|  4.1202e-06     	|  1.6407e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.0894e-06     	|  1              	|  4.0894e-06     	|  1.6285e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  1.9898e-06     	|  2              	|  3.9795e-06     	|  1.5847e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  1.8748e-06     	|  2              	|  3.7495e-06     	|  1.4931e-06     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  3.7001e-06     	|  1              	|  3.7001e-06     	|  1.4734e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.8352e-06     	|  2              	|  3.6703e-06     	|  1.4616e-06     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  3.3304e-06     	|  1              	|  3.3304e-06     	|  1.3262e-06     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.2308e-06     	|  1              	|  3.2308e-06     	|  1.2865e-06     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  3.2298e-06     	|  1              	|  3.2298e-06     	|  1.2861e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.56e-06       	|  2              	|  3.1199e-06     	|  1.2424e-06     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.0696e-06     	|  1              	|  3.0696e-06     	|  1.2224e-06     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.5101e-06     	|  2              	|  3.0203e-06     	|  1.2027e-06     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  2.9998e-06     	|  1              	|  2.9998e-06     	|  1.1945e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.9905e-06     	|  1              	|  2.9905e-06     	|  1.1908e-06     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.6803e-06     	|  1              	|  2.6803e-06     	|  1.0673e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.2852e-06     	|  2              	|  2.5705e-06     	|  1.0236e-06     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.5304e-06     	|  1              	|  2.5304e-06     	|  1.0076e-06     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  2.4904e-06     	|  1              	|  2.4904e-06     	|  9.9168e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.4503e-06     	|  1              	|  2.4503e-06     	|  9.7574e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.3805e-06     	|  1              	|  2.3805e-06     	|  9.4792e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.3404e-06     	|  1              	|  2.3404e-06     	|  9.3197e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.1802e-06     	|  1              	|  2.1802e-06     	|  8.6819e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.1495e-06     	|  1              	|  2.1495e-06     	|  8.5595e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.1197e-06     	|  1              	|  2.1197e-06     	|  8.4408e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.0899e-06     	|  1              	|  2.0899e-06     	|  8.3221e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  2.0508e-06     	|  1              	|  2.0508e-06     	|  8.1664e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.939e-06      	|  1              	|  1.939e-06      	|  7.7213e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.9306e-06     	|  1              	|  1.9306e-06     	|  7.688e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9297e-06     	|  1              	|  1.9297e-06     	|  7.6842e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881073         	|  516.06         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  24.726         	|  20             	|  494.52         	|  95.826         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.011014       	|  20000          	|  220.29         	|  42.687         	|
|  run_training_batch                                                                                                                                                                                         	|  0.010801       	|  20000          	|  216.03         	|  41.861         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.010449       	|  20000          	|  208.97         	|  40.494         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0055617      	|  20000          	|  111.23         	|  21.555         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0023714      	|  20000          	|  47.428         	|  9.1905         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.6777         	|  38             	|  25.753         	|  4.9903         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  12.165         	|  1              	|  12.165         	|  2.3573         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.0005368      	|  20000          	|  10.736         	|  2.0804         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00023069     	|  20038          	|  4.6226         	|  0.89575        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00014613     	|  20038          	|  2.9282         	|  0.56741        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.13336        	|  20             	|  2.6673         	|  0.51686        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.024681       	|  38             	|  0.93788        	|  0.18174        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.4019e-05     	|  20000          	|  0.68038        	|  0.13184        	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.3134e-05     	|  20000          	|  0.46269        	|  0.089658       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.2218e-05     	|  20000          	|  0.44437        	|  0.086109       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.098603       	|  2              	|  0.19721        	|  0.038214       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  5.7621e-06     	|  20000          	|  0.11524        	|  0.022331       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  5.5261e-06     	|  20000          	|  0.11052        	|  0.021416       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  5.3431e-06     	|  20000          	|  0.10686        	|  0.020707       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.2492e-06     	|  20000          	|  0.10498        	|  0.020343       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  4.2421e-06     	|  20000          	|  0.084841       	|  0.01644        	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  4.1031e-06     	|  20000          	|  0.082062       	|  0.015902       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  4.0247e-06     	|  20000          	|  0.080494       	|  0.015598       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  4.0125e-06     	|  20000          	|  0.08025        	|  0.015551       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.5943e-06     	|  20000          	|  0.071886       	|  0.01393        	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  3.5479e-06     	|  20000          	|  0.070957       	|  0.01375        	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  3.4618e-06     	|  20000          	|  0.069237       	|  0.013416       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.4327e-06     	|  20000          	|  0.068654       	|  0.013304       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  3.3101e-06     	|  20000          	|  0.066201       	|  0.012828       	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  3.2435e-06     	|  20038          	|  0.064994       	|  0.012594       	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  3.0324e-06     	|  20000          	|  0.060649       	|  0.011752       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.946e-06      	|  20000          	|  0.058919       	|  0.011417       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.8894e-06     	|  20000          	|  0.057788       	|  0.011198       	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.8481e-06     	|  20000          	|  0.056962       	|  0.011038       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.8259e-06     	|  20000          	|  0.056519       	|  0.010952       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.7864e-06     	|  20000          	|  0.055729       	|  0.010799       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.7424e-06     	|  20000          	|  0.054847       	|  0.010628       	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.6064e-06     	|  20000          	|  0.052127       	|  0.010101       	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  2.5225e-06     	|  20038          	|  0.050546       	|  0.0097947      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.3871e-06     	|  20000          	|  0.047742       	|  0.0092513      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.8876e-06     	|  20000          	|  0.037751       	|  0.0073154      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.5345e-06     	|  20000          	|  0.03069        	|  0.005947       	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.3425e-06     	|  20000          	|  0.02685        	|  0.005203       	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.3412e-06     	|  20000          	|  0.026824       	|  0.0051979      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.244e-06      	|  20000          	|  0.024881       	|  0.0048213      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.209e-06      	|  20000          	|  0.024181       	|  0.0046857      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.167e-06      	|  20000          	|  0.023341       	|  0.0045229      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.1167e-06     	|  20000          	|  0.022334       	|  0.0043277      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.0513e-06     	|  20000          	|  0.021026       	|  0.0040744      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.019283       	|  1              	|  0.019283       	|  0.0037366      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.0004143      	|  38             	|  0.015744       	|  0.0030507      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00055579     	|  20             	|  0.011116       	|  0.002154       	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.004081       	|  2              	|  0.008162       	|  0.0015816      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00027654     	|  20             	|  0.0055309      	|  0.0010718      	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00019009     	|  20             	|  0.0038018      	|  0.00073669     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.9472e-05     	|  38             	|  0.00188        	|  0.00036429     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00056751     	|  2              	|  0.001135       	|  0.00021994     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00045397     	|  2              	|  0.00090794     	|  0.00017594     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00042234     	|  2              	|  0.00084467     	|  0.00016368     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00040654     	|  2              	|  0.00081308     	|  0.00015756     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  0.00063264     	|  1              	|  0.00063264     	|  0.00012259     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00059187     	|  1              	|  0.00059187     	|  0.00011469     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00038448     	|  1              	|  0.00038448     	|  7.4503e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00026739     	|  1              	|  0.00026739     	|  5.1814e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00024525     	|  1              	|  0.00024525     	|  4.7523e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  5.4331e-06     	|  38             	|  0.00020646     	|  4.0007e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.7521e-06     	|  38             	|  0.00014258     	|  2.7629e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.7038e-06     	|  38             	|  0.00014075     	|  2.7273e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.4298e-06     	|  38             	|  0.00013033     	|  2.5256e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.4075e-06     	|  20             	|  0.00012815     	|  2.4832e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.6929e-06     	|  20             	|  0.00011386     	|  2.2063e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.9236e-06     	|  38             	|  0.0001111      	|  2.1528e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  4.1044e-06     	|  22             	|  9.0296e-05     	|  1.7497e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.6362e-06     	|  22             	|  7.9996e-05     	|  1.5501e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.426e-06      	|  20             	|  6.852e-05      	|  1.3278e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.7166e-06     	|  38             	|  6.5229e-05     	|  1.264e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.2139e-05     	|  2              	|  6.4279e-05     	|  1.2456e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.608e-06      	|  38             	|  6.1102e-05     	|  1.184e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.64e-06       	|  22             	|  5.808e-05      	|  1.1255e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.5373e-06     	|  22             	|  5.5822e-05     	|  1.0817e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.4559e-06     	|  22             	|  5.403e-05      	|  1.047e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.4182e-06     	|  22             	|  5.3201e-05     	|  1.0309e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.2145e-06     	|  38             	|  4.6152e-05     	|  8.9431e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.9774e-06     	|  20             	|  3.9548e-05     	|  7.6634e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.7426e-06     	|  22             	|  3.8338e-05     	|  7.429e-06      	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.6772e-06     	|  22             	|  3.6898e-05     	|  7.15e-06       	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.7925e-06     	|  20             	|  3.5849e-05     	|  6.9468e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  8.5253e-06     	|  2              	|  1.7051e-05     	|  3.304e-06      	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  5.73e-06       	|  2              	|  1.146e-05      	|  2.2207e-06     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  1.021e-05      	|  1              	|  1.021e-05      	|  1.9785e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.0041e-05     	|  1              	|  1.0041e-05     	|  1.9456e-06     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  8.4909e-06     	|  1              	|  8.4909e-06     	|  1.6453e-06     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  4.245e-06      	|  2              	|  8.4899e-06     	|  1.6452e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  4.0242e-06     	|  2              	|  8.0485e-06     	|  1.5596e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  8.0401e-06     	|  1              	|  8.0401e-06     	|  1.558e-06      	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  6.5397e-06     	|  1              	|  6.5397e-06     	|  1.2673e-06     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  6.2902e-06     	|  1              	|  6.2902e-06     	|  1.2189e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.1451e-06     	|  2              	|  6.2902e-06     	|  1.2189e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  6.129e-06      	|  1              	|  6.129e-06      	|  1.1877e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.855e-06      	|  2              	|  5.7099e-06     	|  1.1065e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.725e-06      	|  2              	|  5.4501e-06     	|  1.0561e-06     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  5.3607e-06     	|  1              	|  5.3607e-06     	|  1.0388e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  5.2284e-06     	|  1              	|  5.2284e-06     	|  1.0132e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.5956e-06     	|  2              	|  5.1912e-06     	|  1.0059e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  5.18e-06       	|  1              	|  5.18e-06       	|  1.0038e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  5.1595e-06     	|  1              	|  5.1595e-06     	|  9.998e-07      	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  2.4401e-06     	|  2              	|  4.8801e-06     	|  9.4566e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.7898e-06     	|  1              	|  4.7898e-06     	|  9.2815e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.1751e-06     	|  2              	|  4.3502e-06     	|  8.4297e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.1649e-06     	|  2              	|  4.3297e-06     	|  8.39e-07       	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.0601e-06     	|  2              	|  4.1202e-06     	|  7.9839e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  1.9851e-06     	|  2              	|  3.9702e-06     	|  7.6934e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  3.6694e-06     	|  1              	|  3.6694e-06     	|  7.1105e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  3.2401e-06     	|  1              	|  3.2401e-06     	|  6.2785e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.1805e-06     	|  1              	|  3.1805e-06     	|  6.163e-07      	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  1.5101e-06     	|  2              	|  3.0203e-06     	|  5.8526e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  2.8796e-06     	|  1              	|  2.8796e-06     	|  5.5801e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.7595e-06     	|  1              	|  2.7595e-06     	|  5.3473e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  2.7101e-06     	|  1              	|  2.7101e-06     	|  5.2516e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  2.5695e-06     	|  1              	|  2.5695e-06     	|  4.9791e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.55e-06       	|  1              	|  2.55e-06       	|  4.9412e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.5304e-06     	|  1              	|  2.5304e-06     	|  4.9033e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  2.4801e-06     	|  1              	|  2.4801e-06     	|  4.8059e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.4792e-06     	|  1              	|  2.4792e-06     	|  4.8041e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.4606e-06     	|  1              	|  2.4606e-06     	|  4.768e-07      	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.4391e-06     	|  1              	|  2.4391e-06     	|  4.7265e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  2.4298e-06     	|  1              	|  2.4298e-06     	|  4.7084e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.2799e-06     	|  1              	|  2.2799e-06     	|  4.4179e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.2603e-06     	|  1              	|  2.2603e-06     	|  4.38e-07       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.2305e-06     	|  1              	|  2.2305e-06     	|  4.3222e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.2193e-06     	|  1              	|  2.2193e-06     	|  4.3006e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.1998e-06     	|  1              	|  2.1998e-06     	|  4.2627e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.9902e-06     	|  1              	|  1.9902e-06     	|  3.8566e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.9707e-06     	|  1              	|  1.9707e-06     	|  3.8187e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.9697e-06     	|  1              	|  1.9697e-06     	|  3.8169e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.9586e-06     	|  1              	|  1.9586e-06     	|  3.7953e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9502e-06     	|  1              	|  1.9502e-06     	|  3.779e-07      	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.37e-06       	|  1              	|  1.37e-06       	|  2.6547e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.3001e-06     	|  1              	|  1.3001e-06     	|  2.5193e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.1399e-06     	|  1              	|  1.1399e-06     	|  2.2089e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  9.7044e-07     	|  1              	|  9.7044e-07     	|  1.8805e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
