FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  882001         	|  489.07         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  22.671         	|  20             	|  453.42         	|  92.712         	|
|  run_training_batch                                                                                                                                                                                         	|  0.018195       	|  20000          	|  363.91         	|  74.409         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.017933       	|  20000          	|  358.66         	|  73.336         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0064337      	|  20000          	|  128.67         	|  26.31          	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0024995      	|  20000          	|  49.989         	|  10.221         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.46652        	|  96             	|  44.786         	|  9.1575         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  22.482         	|  1              	|  22.482         	|  4.5969         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.00096199     	|  20000          	|  19.24          	|  3.934          	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.0004614      	|  20000          	|  9.2279         	|  1.8868         	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.17419        	|  20             	|  3.4839         	|  0.71235        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00013308     	|  20096          	|  2.6743         	|  0.54682        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.018442       	|  96             	|  1.7705         	|  0.36201        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  8.2819e-05     	|  20096          	|  1.6643         	|  0.34031        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  1.5639e-05     	|  20000          	|  0.31277        	|  0.063954       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.5233e-05     	|  20000          	|  0.30467        	|  0.062296       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.2165e-05     	|  20000          	|  0.2433         	|  0.049747       	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.080556       	|  2              	|  0.16111        	|  0.032943       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.9873e-06     	|  20000          	|  0.099745       	|  0.020395       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.041622       	|  2              	|  0.083245       	|  0.017021       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.6497e-06     	|  20000          	|  0.072993       	|  0.014925       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.129e-06      	|  20000          	|  0.062579       	|  0.012796       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.0866e-06     	|  20000          	|  0.061732       	|  0.012622       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.0683e-06     	|  20000          	|  0.061366       	|  0.012548       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  2.9287e-06     	|  20000          	|  0.058573       	|  0.011977       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.8103e-06     	|  20000          	|  0.056207       	|  0.011493       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.7672e-06     	|  20000          	|  0.055345       	|  0.011316       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  2.7272e-06     	|  20000          	|  0.054544       	|  0.011153       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  2.6989e-06     	|  20000          	|  0.053979       	|  0.011037       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.6797e-06     	|  20000          	|  0.053593       	|  0.010958       	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.6491e-06     	|  20000          	|  0.052982       	|  0.010833       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.4435e-06     	|  20000          	|  0.048871       	|  0.0099927      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.3442e-06     	|  20000          	|  0.046883       	|  0.0095863      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.2458e-06     	|  20000          	|  0.044915       	|  0.0091839      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.0592e-06     	|  20000          	|  0.041185       	|  0.0084211      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.0502e-06     	|  20000          	|  0.041005       	|  0.0083843      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.0363e-06     	|  20000          	|  0.040725       	|  0.0083272      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.0345e-06     	|  20000          	|  0.040689       	|  0.0083198      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  1.9609e-06     	|  20000          	|  0.039217       	|  0.0080188      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  1.9299e-06     	|  20000          	|  0.038599       	|  0.0078923      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  1.8978e-06     	|  20000          	|  0.037957       	|  0.0077611      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00027978     	|  96             	|  0.026859       	|  0.0054919      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.3005e-06     	|  20096          	|  0.026135       	|  0.0053439      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.1477e-06     	|  20096          	|  0.023065       	|  0.0047161      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.0884e-06     	|  20000          	|  0.021768       	|  0.004451       	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  9.4976e-07     	|  20000          	|  0.018995       	|  0.003884       	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  9.1271e-07     	|  20000          	|  0.018254       	|  0.0037325      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  8.7028e-07     	|  20000          	|  0.017406       	|  0.003559       	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  8.4491e-07     	|  20000          	|  0.016898       	|  0.0034552      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  7.469e-07      	|  20000          	|  0.014938       	|  0.0030544      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  7.1634e-07     	|  20000          	|  0.014327       	|  0.0029294      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  7.06e-07       	|  20000          	|  0.01412        	|  0.0028871      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  6.8191e-07     	|  20000          	|  0.013638       	|  0.0027886      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.01036        	|  1              	|  0.01036        	|  0.0021184      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00023293     	|  20             	|  0.0046587      	|  0.00095257     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00022195     	|  20             	|  0.0044389      	|  0.00090764     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.1819e-05     	|  96             	|  0.0040147      	|  0.00082089     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00012798     	|  20             	|  0.0025596      	|  0.00052337     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00060282     	|  2              	|  0.0012056      	|  0.00024652     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00052665     	|  2              	|  0.0010533      	|  0.00021537     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00047974     	|  2              	|  0.00095948     	|  0.00019619     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00059263     	|  1              	|  0.00059263     	|  0.00012118     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00028917     	|  2              	|  0.00057834     	|  0.00011825     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00041383     	|  1              	|  0.00041383     	|  8.4616e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  4.303e-06      	|  96             	|  0.00041309     	|  8.4464e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.0002797      	|  1              	|  0.0002797      	|  5.719e-05      	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  2.7536e-06     	|  96             	|  0.00026435     	|  5.4052e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00025991     	|  1              	|  0.00025991     	|  5.3144e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.2136e-06     	|  96             	|  0.00021251     	|  4.3452e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.1984e-06     	|  96             	|  0.00021105     	|  4.3153e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  1.6132e-06     	|  96             	|  0.00015487     	|  3.1667e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  5.4595e-06     	|  20             	|  0.00010919     	|  2.2326e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  9.6614e-07     	|  96             	|  9.2749e-05     	|  1.8965e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  9.3204e-07     	|  96             	|  8.9476e-05     	|  1.8295e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.4185e-06     	|  20             	|  8.8369e-05     	|  1.8069e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  8.1373e-07     	|  96             	|  7.8118e-05     	|  1.5973e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.836e-05      	|  2              	|  7.672e-05      	|  1.5687e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.4262e-06     	|  22             	|  7.5377e-05     	|  1.5412e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  2.9331e-06     	|  22             	|  6.4529e-05     	|  1.3194e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.5636e-06     	|  20             	|  5.1272e-05     	|  1.0484e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.1819e-06     	|  22             	|  4.8001e-05     	|  9.8149e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  4.6371e-05     	|  1              	|  4.6371e-05     	|  9.4815e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.0854e-06     	|  22             	|  4.588e-05      	|  9.3811e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.0487e-06     	|  22             	|  4.507e-05      	|  9.2156e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.0396e-06     	|  22             	|  4.4871e-05     	|  9.1749e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.3787e-06     	|  22             	|  3.033e-05      	|  6.2017e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.479e-06      	|  20             	|  2.9581e-05     	|  6.0484e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.2822e-06     	|  22             	|  2.8208e-05     	|  5.7677e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.0515e-05     	|  2              	|  2.103e-05      	|  4.3001e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.0496e-06     	|  20             	|  2.0993e-05     	|  4.2925e-06     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.902e-05      	|  1              	|  1.902e-05      	|  3.8891e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  7.1907e-06     	|  2              	|  1.4381e-05     	|  2.9406e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  7.0599e-06     	|  2              	|  1.412e-05      	|  2.8871e-06     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  5.2652e-06     	|  2              	|  1.053e-05      	|  2.1532e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  1.024e-05      	|  1              	|  1.024e-05      	|  2.0938e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  9.8804e-06     	|  1              	|  9.8804e-06     	|  2.0203e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  9.5097e-06     	|  1              	|  9.5097e-06     	|  1.9445e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  9.1e-06        	|  1              	|  9.1e-06        	|  1.8607e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.499e-06      	|  2              	|  6.998e-06      	|  1.4309e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  3.3551e-06     	|  2              	|  6.7102e-06     	|  1.372e-06      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.3397e-06     	|  2              	|  6.6794e-06     	|  1.3658e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  3.2499e-06     	|  2              	|  6.4997e-06     	|  1.329e-06      	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  6.2399e-06     	|  1              	|  6.2399e-06     	|  1.2759e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  5.7705e-06     	|  1              	|  5.7705e-06     	|  1.1799e-06     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.7295e-06     	|  1              	|  5.7295e-06     	|  1.1715e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.5905e-06     	|  2              	|  5.1809e-06     	|  1.0594e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.57e-06       	|  2              	|  5.14e-06       	|  1.051e-06      	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  5.0599e-06     	|  1              	|  5.0599e-06     	|  1.0346e-06     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  5.0496e-06     	|  1              	|  5.0496e-06     	|  1.0325e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.8401e-06     	|  1              	|  4.8401e-06     	|  9.8966e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  4.7497e-06     	|  1              	|  4.7497e-06     	|  9.7119e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.4806e-06     	|  1              	|  4.4806e-06     	|  9.1615e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.2897e-06     	|  1              	|  4.2897e-06     	|  8.7712e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.2198e-06     	|  1              	|  4.2198e-06     	|  8.6283e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.15e-06       	|  1              	|  4.15e-06       	|  8.4855e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.1295e-06     	|  1              	|  4.1295e-06     	|  8.4436e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  4.1099e-06     	|  1              	|  4.1099e-06     	|  8.4036e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.0447e-06     	|  2              	|  4.0894e-06     	|  8.3617e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.0345e-06     	|  2              	|  4.0689e-06     	|  8.3199e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.9502e-06     	|  2              	|  3.9004e-06     	|  7.9752e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  3.7197e-06     	|  1              	|  3.7197e-06     	|  7.6057e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  3.6499e-06     	|  1              	|  3.6499e-06     	|  7.4629e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  3.4999e-06     	|  1              	|  3.4999e-06     	|  7.1563e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.1106e-06     	|  1              	|  3.1106e-06     	|  6.3603e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  3.0706e-06     	|  1              	|  3.0706e-06     	|  6.2784e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  3.0305e-06     	|  1              	|  3.0305e-06     	|  6.1966e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.6599e-06     	|  1              	|  2.6599e-06     	|  5.4387e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.3253e-06     	|  2              	|  2.6505e-06     	|  5.4196e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.5202e-06     	|  1              	|  2.5202e-06     	|  5.153e-07      	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.4894e-06     	|  1              	|  2.4894e-06     	|  5.0902e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.4494e-06     	|  1              	|  2.4494e-06     	|  5.0083e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.3199e-06     	|  1              	|  2.3199e-06     	|  4.7436e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.2799e-06     	|  1              	|  2.2799e-06     	|  4.6617e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.17e-06       	|  1              	|  2.17e-06       	|  4.437e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.1504e-06     	|  1              	|  2.1504e-06     	|  4.397e-07      	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.1495e-06     	|  1              	|  2.1495e-06     	|  4.3951e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.0601e-06     	|  1              	|  2.0601e-06     	|  4.2123e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.02e-06       	|  1              	|  2.02e-06       	|  4.1304e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.8505e-06     	|  1              	|  1.8505e-06     	|  3.7838e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.7807e-06     	|  1              	|  1.7807e-06     	|  3.641e-07      	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.1194e-06     	|  1              	|  1.1194e-06     	|  2.289e-07      	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  1.0394e-06     	|  1              	|  1.0394e-06     	|  2.1252e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  8.801e-07      	|  1              	|  8.801e-07      	|  1.7996e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  882001         	|  990.27         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  47.353         	|  20             	|  947.07         	|  95.637         	|
|  run_training_batch                                                                                                                                                                                         	|  0.025006       	|  20000          	|  500.12         	|  50.504         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.024592       	|  20000          	|  491.83         	|  49.666         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.012762       	|  20000          	|  255.23         	|  25.774         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.010127       	|  20000          	|  202.55         	|  20.454         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  1.4735         	|  96             	|  141.46         	|  14.285         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0025461      	|  20000          	|  50.923         	|  5.1423         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  33.96          	|  1              	|  33.96          	|  3.4293         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00098255     	|  20000          	|  19.651         	|  1.9844         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00043812     	|  20096          	|  8.8044         	|  0.88909        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00026943     	|  20096          	|  5.4145         	|  0.54677        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.12498        	|  20             	|  2.4996         	|  0.25241        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.017905       	|  96             	|  1.7189         	|  0.17358        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  4.3922e-05     	|  20000          	|  0.87844        	|  0.088707       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.5343e-05     	|  20000          	|  0.50686        	|  0.051184       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.4668e-05     	|  20000          	|  0.49336        	|  0.049821       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.07699        	|  2              	|  0.15398        	|  0.015549       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  7.5545e-06     	|  20000          	|  0.15109        	|  0.015257       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  6.2381e-06     	|  20000          	|  0.12476        	|  0.012599       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.6268e-06     	|  20000          	|  0.11254        	|  0.011364       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  5.2243e-06     	|  20000          	|  0.10449        	|  0.010551       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  4.6016e-06     	|  20000          	|  0.092031       	|  0.0092935      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  4.475e-06      	|  20000          	|  0.0895         	|  0.0090379      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  4.1266e-06     	|  20000          	|  0.082532       	|  0.0083343      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.8453e-06     	|  20000          	|  0.076905       	|  0.0077661      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  3.8452e-06     	|  20000          	|  0.076903       	|  0.0077659      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  3.6586e-06     	|  20096          	|  0.073524       	|  0.0074246      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.5916e-06     	|  20000          	|  0.071831       	|  0.0072537      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  3.3553e-06     	|  20000          	|  0.067107       	|  0.0067766      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  3.3368e-06     	|  20096          	|  0.067056       	|  0.0067714      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  3.3467e-06     	|  20000          	|  0.066935       	|  0.0067592      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  3.3241e-06     	|  20000          	|  0.066482       	|  0.0067135      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  3.2273e-06     	|  20000          	|  0.064546       	|  0.006518       	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  3.0938e-06     	|  20000          	|  0.061876       	|  0.0062484      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  3.0874e-06     	|  20000          	|  0.061747       	|  0.0062354      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  3.0199e-06     	|  20000          	|  0.060397       	|  0.006099       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.9966e-06     	|  20000          	|  0.059932       	|  0.0060521      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.8543e-06     	|  20000          	|  0.057085       	|  0.0057646      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.691e-06      	|  20000          	|  0.053821       	|  0.0054349      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.6483e-06     	|  20000          	|  0.052966       	|  0.0053487      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.3639e-06     	|  20000          	|  0.047278       	|  0.0047742      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  2.2466e-06     	|  20000          	|  0.044932       	|  0.0045374      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00044827     	|  96             	|  0.043034       	|  0.0043457      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  2.1239e-06     	|  20000          	|  0.042477       	|  0.0042894      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.6322e-06     	|  20000          	|  0.032644       	|  0.0032965      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.6211e-06     	|  20000          	|  0.032422       	|  0.0032741      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.5974e-06     	|  20000          	|  0.031948       	|  0.0032262      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.5637e-06     	|  20000          	|  0.031274       	|  0.0031582      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.5529e-06     	|  20000          	|  0.031058       	|  0.0031363      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.4566e-06     	|  20000          	|  0.029132       	|  0.0029418      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.3353e-06     	|  20000          	|  0.026707       	|  0.0026969      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0071804      	|  2              	|  0.014361       	|  0.0014502      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00047943     	|  20             	|  0.0095886      	|  0.00096828     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0085955      	|  1              	|  0.0085955      	|  0.000868       	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00026503     	|  20             	|  0.0053005      	|  0.00053526     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  2.6511e-05     	|  96             	|  0.0025451      	|  0.00025701     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00010988     	|  20             	|  0.0021975      	|  0.00022191     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00096563     	|  2              	|  0.0019313      	|  0.00019502     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00069141     	|  2              	|  0.0013828      	|  0.00013964     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00055756     	|  2              	|  0.0011151      	|  0.00011261     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  0.00081574     	|  1              	|  0.00081574     	|  8.2376e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00074464     	|  1              	|  0.00074464     	|  7.5196e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  7.4141e-06     	|  96             	|  0.00071176     	|  7.1875e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00034451     	|  2              	|  0.00068902     	|  6.9579e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.0005442      	|  1              	|  0.0005442      	|  5.4954e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00041232     	|  1              	|  0.00041232     	|  4.1637e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.74e-06       	|  96             	|  0.00035904     	|  3.6256e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.5437e-06     	|  96             	|  0.00034019     	|  3.4353e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.4637e-06     	|  96             	|  0.00033252     	|  3.3578e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.0003293      	|  1              	|  0.0003293      	|  3.3253e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.0727e-06     	|  96             	|  0.00029498     	|  2.9788e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.97e-06       	|  96             	|  0.00018912     	|  1.9098e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.7278e-06     	|  96             	|  0.00016586     	|  1.6749e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.629e-06      	|  20             	|  0.00013258     	|  1.3388e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.324e-06      	|  96             	|  0.0001271      	|  1.2835e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  6.155e-06      	|  20             	|  0.0001231      	|  1.2431e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  4.6535e-05     	|  2              	|  9.307e-05      	|  9.3984e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  4.2098e-06     	|  22             	|  9.2616e-05     	|  9.3526e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  3.7522e-06     	|  22             	|  8.2549e-05     	|  8.336e-06      	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.6341e-06     	|  22             	|  7.9951e-05     	|  8.0737e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.4184e-06     	|  20             	|  6.8368e-05     	|  6.904e-06      	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.4691e-06     	|  22             	|  5.432e-05      	|  5.4854e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.4346e-06     	|  22             	|  5.3561e-05     	|  5.4087e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.4327e-06     	|  22             	|  5.3519e-05     	|  5.4045e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.7581e-06     	|  22             	|  3.8678e-05     	|  3.9058e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.8314e-06     	|  20             	|  3.6628e-05     	|  3.6988e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.8234e-06     	|  20             	|  3.6469e-05     	|  3.6827e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.5305e-06     	|  22             	|  3.3671e-05     	|  3.4002e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.2335e-05     	|  2              	|  2.467e-05      	|  2.4912e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  1.101e-05      	|  1              	|  1.101e-05      	|  1.1118e-06     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  5.3048e-06     	|  2              	|  1.061e-05      	|  1.0714e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  4.95e-06       	|  2              	|  9.9e-06        	|  9.9972e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  9.2201e-06     	|  1              	|  9.2201e-06     	|  9.3107e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  8.869e-06      	|  1              	|  8.869e-06      	|  8.9561e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  8.6203e-06     	|  1              	|  8.6203e-06     	|  8.705e-07      	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  4.2398e-06     	|  2              	|  8.4797e-06     	|  8.563e-07      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.9949e-06     	|  2              	|  7.9898e-06     	|  8.0683e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.4897e-06     	|  2              	|  6.9793e-06     	|  7.0479e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  6.5602e-06     	|  1              	|  6.5602e-06     	|  6.6247e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  5.5796e-06     	|  1              	|  5.5796e-06     	|  5.6344e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  5.5302e-06     	|  1              	|  5.5302e-06     	|  5.5845e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.7097e-06     	|  2              	|  5.4194e-06     	|  5.4726e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  5.3793e-06     	|  1              	|  5.3793e-06     	|  5.4322e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.6803e-06     	|  2              	|  5.3607e-06     	|  5.4134e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.645e-06      	|  2              	|  5.2899e-06     	|  5.3419e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.9602e-06     	|  1              	|  4.9602e-06     	|  5.0089e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  4.76e-06       	|  1              	|  4.76e-06       	|  4.8067e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.3097e-06     	|  2              	|  4.6194e-06     	|  4.6647e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  4.6091e-06     	|  1              	|  4.6091e-06     	|  4.6544e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  2.2799e-06     	|  2              	|  4.5598e-06     	|  4.6045e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  4.4098e-06     	|  1              	|  4.4098e-06     	|  4.4531e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.1998e-06     	|  2              	|  4.3996e-06     	|  4.4428e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.15e-06       	|  2              	|  4.2999e-06     	|  4.3422e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.2003e-06     	|  1              	|  4.2003e-06     	|  4.2415e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  3.7607e-06     	|  1              	|  3.7607e-06     	|  3.7976e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  3.5986e-06     	|  1              	|  3.5986e-06     	|  3.634e-07      	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  3.58e-06       	|  1              	|  3.58e-06       	|  3.6152e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.5698e-06     	|  1              	|  3.5698e-06     	|  3.6048e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  3.4897e-06     	|  1              	|  3.4897e-06     	|  3.5239e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  3.4003e-06     	|  1              	|  3.4003e-06     	|  3.4337e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  3.3705e-06     	|  1              	|  3.3705e-06     	|  3.4036e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.1395e-06     	|  1              	|  3.1395e-06     	|  3.1703e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  3.1395e-06     	|  1              	|  3.1395e-06     	|  3.1703e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.9504e-06     	|  1              	|  2.9504e-06     	|  2.9794e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  1.4598e-06     	|  2              	|  2.9197e-06     	|  2.9484e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.9001e-06     	|  1              	|  2.9001e-06     	|  2.9286e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.6999e-06     	|  1              	|  2.6999e-06     	|  2.7264e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.6096e-06     	|  1              	|  2.6096e-06     	|  2.6352e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.5807e-06     	|  1              	|  2.5807e-06     	|  2.606e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.5798e-06     	|  1              	|  2.5798e-06     	|  2.6051e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  2.5304e-06     	|  1              	|  2.5304e-06     	|  2.5553e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.4997e-06     	|  1              	|  2.4997e-06     	|  2.5242e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.3898e-06     	|  1              	|  2.3898e-06     	|  2.4132e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.1402e-06     	|  1              	|  2.1402e-06     	|  2.1612e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.1299e-06     	|  1              	|  2.1299e-06     	|  2.1509e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.1299e-06     	|  1              	|  2.1299e-06     	|  2.1509e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.9697e-06     	|  1              	|  1.9697e-06     	|  1.9891e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.8897e-06     	|  1              	|  1.8897e-06     	|  1.9082e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.8599e-06     	|  1              	|  1.8599e-06     	|  1.8781e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.79e-06       	|  1              	|  1.79e-06       	|  1.8076e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.7695e-06     	|  1              	|  1.7695e-06     	|  1.7869e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.4305e-06     	|  1              	|  1.4305e-06     	|  1.4446e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
