FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881617         	|  1910.4         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  94.427         	|  20             	|  1888.5         	|  98.857         	|
|  run_training_batch                                                                                                                                                                                         	|  0.088385       	|  20000          	|  1767.7         	|  92.531         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.088092       	|  20000          	|  1761.8         	|  92.224         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.023421       	|  20000          	|  468.41         	|  24.519         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0044098      	|  20000          	|  88.196         	|  4.6166         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.8728         	|  72             	|  62.841         	|  3.2895         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0018391      	|  20000          	|  36.781         	|  1.9253         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  11.561         	|  1              	|  11.561         	|  0.60518        	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00042111     	|  20000          	|  8.4222         	|  0.44086        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.19378        	|  20             	|  3.8756         	|  0.20287        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.000112       	|  20072          	|  2.248          	|  0.11767        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  7.0564e-05     	|  20072          	|  1.4164         	|  0.07414        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0061125      	|  72             	|  0.4401         	|  0.023037       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.1382e-05     	|  20000          	|  0.42764        	|  0.022385       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.3685e-05     	|  20000          	|  0.27369        	|  0.014327       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.2748e-05     	|  20000          	|  0.25496        	|  0.013346       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.053035       	|  2              	|  0.10607        	|  0.0055522      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.3773e-06     	|  20000          	|  0.087545       	|  0.0045826      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.2462e-06     	|  20000          	|  0.064924       	|  0.0033985      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.1929e-06     	|  20000          	|  0.063858       	|  0.0033427      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  2.9669e-06     	|  20000          	|  0.059338       	|  0.0031061      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  2.8268e-06     	|  20000          	|  0.056536       	|  0.0029594      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  2.7474e-06     	|  20000          	|  0.054948       	|  0.0028763      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.5514e-06     	|  20000          	|  0.051028       	|  0.0026711      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.466e-06      	|  20000          	|  0.049319       	|  0.0025816      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.3502e-06     	|  20000          	|  0.047005       	|  0.0024605      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.3259e-06     	|  20000          	|  0.046518       	|  0.002435       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.3041e-06     	|  20000          	|  0.046081       	|  0.0024122      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.2419e-06     	|  20000          	|  0.044838       	|  0.0023471      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.2092e-06     	|  20000          	|  0.044184       	|  0.0023128      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.1979e-06     	|  20000          	|  0.043959       	|  0.002301       	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.1458e-06     	|  20000          	|  0.042915       	|  0.0022464      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.1111e-06     	|  20000          	|  0.042221       	|  0.0022101      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.1001e-06     	|  20000          	|  0.042001       	|  0.0021986      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.0939e-06     	|  20000          	|  0.041878       	|  0.0021921      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.0915e-06     	|  20000          	|  0.041831       	|  0.0021896      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.0183e-06     	|  20000          	|  0.040366       	|  0.002113       	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.0132e-06     	|  20000          	|  0.040265       	|  0.0021077      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  1.9626e-06     	|  20000          	|  0.039252       	|  0.0020546      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00050001     	|  72             	|  0.036001       	|  0.0018845      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.158e-06      	|  20072          	|  0.023243       	|  0.0012167      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.1112e-06     	|  20072          	|  0.022303       	|  0.0011675      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.0614e-06     	|  20000          	|  0.021228       	|  0.0011112      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  9.5349e-07     	|  20000          	|  0.01907        	|  0.00099822     	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  9.4445e-07     	|  20000          	|  0.018889       	|  0.00098876     	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  8.6329e-07     	|  20000          	|  0.017266       	|  0.00090379     	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  8.5182e-07     	|  20000          	|  0.017036       	|  0.00089178     	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  8.2165e-07     	|  20000          	|  0.016433       	|  0.00086019     	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  7.8575e-07     	|  20000          	|  0.015715       	|  0.00082261     	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  7.7195e-07     	|  20000          	|  0.015439       	|  0.00080816     	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  7.5614e-07     	|  20000          	|  0.015123       	|  0.00079161     	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0059615      	|  2              	|  0.011923       	|  0.00062411     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0057631      	|  1              	|  0.0057631      	|  0.00030167     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00024352     	|  20             	|  0.0048704      	|  0.00025494     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00022426     	|  20             	|  0.0044851      	|  0.00023478     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.0015066      	|  2              	|  0.0030131      	|  0.00015772     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  3.5457e-05     	|  72             	|  0.0025529      	|  0.00013363     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00011297     	|  20             	|  0.0022594      	|  0.00011827     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00063166     	|  2              	|  0.0012633      	|  6.6129e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.0005285      	|  2              	|  0.001057       	|  5.533e-05      	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00071872     	|  1              	|  0.00071872     	|  3.7622e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  7.2345e-06     	|  72             	|  0.00052088     	|  2.7266e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00025788     	|  2              	|  0.00051576     	|  2.6998e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00031365     	|  1              	|  0.00031365     	|  1.6418e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  4.1864e-06     	|  72             	|  0.00030142     	|  1.5778e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00027323     	|  1              	|  0.00027323     	|  1.4302e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.5414e-06     	|  72             	|  0.00025498     	|  1.3347e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00024739     	|  1              	|  0.00024739     	|  1.295e-05      	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.2026e-06     	|  72             	|  0.00023059     	|  1.207e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.0831e-06     	|  72             	|  0.00022198     	|  1.162e-05      	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.4573e-06     	|  72             	|  0.00017692     	|  9.2612e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  2.0713e-06     	|  72             	|  0.00014914     	|  7.8066e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  7.4474e-05     	|  2              	|  0.00014895     	|  7.7968e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.9565e-06     	|  72             	|  0.00014086     	|  7.3736e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  5.7798e-06     	|  20             	|  0.0001156      	|  6.0509e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.6737e-06     	|  20             	|  9.3473e-05     	|  4.8929e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.1982e-06     	|  22             	|  7.036e-05      	|  3.683e-06      	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  2.8009e-06     	|  22             	|  6.1619e-05     	|  3.2255e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  2.735e-06      	|  20             	|  5.47e-05       	|  2.8633e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.5983e-06     	|  20             	|  5.1967e-05     	|  2.7202e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.1818e-06     	|  22             	|  4.7999e-05     	|  2.5125e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.1401e-06     	|  22             	|  4.7081e-05     	|  2.4645e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.1004e-06     	|  22             	|  4.6209e-05     	|  2.4188e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.0675e-06     	|  22             	|  4.5486e-05     	|  2.381e-06      	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.8155e-05     	|  2              	|  3.6309e-05     	|  1.9006e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  2.8321e-05     	|  1              	|  2.8321e-05     	|  1.4825e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.2668e-06     	|  22             	|  2.7869e-05     	|  1.4588e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.3929e-06     	|  20             	|  2.7859e-05     	|  1.4583e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.1799e-06     	|  22             	|  2.5959e-05     	|  1.3588e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  6.3553e-06     	|  2              	|  1.2711e-05     	|  6.6535e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  5.5404e-06     	|  2              	|  1.1081e-05     	|  5.8003e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  4.9849e-06     	|  2              	|  9.9698e-06     	|  5.2187e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  4.53e-06       	|  2              	|  9.0599e-06     	|  4.7425e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  8.76e-06       	|  1              	|  8.76e-06       	|  4.5855e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  7.9302e-06     	|  1              	|  7.9302e-06     	|  4.1511e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  7.9293e-06     	|  1              	|  7.9293e-06     	|  4.1506e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  7.3202e-06     	|  1              	|  7.3202e-06     	|  3.8318e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  3.655e-06      	|  2              	|  7.31e-06       	|  3.8264e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  6.6794e-06     	|  1              	|  6.6794e-06     	|  3.4964e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  6.3302e-06     	|  1              	|  6.3302e-06     	|  3.3136e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  6.1505e-06     	|  1              	|  6.1505e-06     	|  3.2195e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.8745e-06     	|  2              	|  5.7491e-06     	|  3.0094e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.7702e-06     	|  2              	|  5.5404e-06     	|  2.9002e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.6049e-06     	|  2              	|  5.2098e-06     	|  2.7271e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  5.18e-06       	|  1              	|  5.18e-06       	|  2.7115e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  4.7497e-06     	|  1              	|  4.7497e-06     	|  2.4863e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.6305e-06     	|  1              	|  4.6305e-06     	|  2.4239e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.5793e-06     	|  1              	|  4.5793e-06     	|  2.3971e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.4405e-06     	|  1              	|  4.4405e-06     	|  2.3244e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  4.2003e-06     	|  1              	|  4.2003e-06     	|  2.1986e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  4.1183e-06     	|  1              	|  4.1183e-06     	|  2.1557e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.0396e-06     	|  2              	|  4.0792e-06     	|  2.1353e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  2.0298e-06     	|  2              	|  4.0596e-06     	|  2.125e-07      	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.0051e-06     	|  2              	|  4.0103e-06     	|  2.0992e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  3.9507e-06     	|  1              	|  3.9507e-06     	|  2.068e-07      	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  3.7905e-06     	|  1              	|  3.7905e-06     	|  1.9841e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.8049e-06     	|  2              	|  3.6098e-06     	|  1.8896e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  3.5493e-06     	|  1              	|  3.5493e-06     	|  1.8579e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  3.5204e-06     	|  1              	|  3.5204e-06     	|  1.8428e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.39e-06       	|  1              	|  3.39e-06       	|  1.7745e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  3.3407e-06     	|  1              	|  3.3407e-06     	|  1.7487e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  3.0901e-06     	|  1              	|  3.0901e-06     	|  1.6175e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.01e-06       	|  1              	|  3.01e-06       	|  1.5756e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.9504e-06     	|  1              	|  2.9504e-06     	|  1.5444e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.4645e-06     	|  2              	|  2.929e-06      	|  1.5332e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.8992e-06     	|  1              	|  2.8992e-06     	|  1.5176e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.8601e-06     	|  1              	|  2.8601e-06     	|  1.4971e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.7996e-06     	|  1              	|  2.7996e-06     	|  1.4654e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.5397e-06     	|  1              	|  2.5397e-06     	|  1.3294e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.4205e-06     	|  1              	|  2.4205e-06     	|  1.267e-07      	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.2603e-06     	|  1              	|  2.2603e-06     	|  1.1832e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.2408e-06     	|  1              	|  2.2408e-06     	|  1.1729e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.2305e-06     	|  1              	|  2.2305e-06     	|  1.1676e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.1802e-06     	|  1              	|  2.1802e-06     	|  1.1413e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.0098e-06     	|  1              	|  2.0098e-06     	|  1.052e-07      	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.9995e-06     	|  1              	|  1.9995e-06     	|  1.0467e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.9604e-06     	|  1              	|  1.9604e-06     	|  1.0262e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.6904e-06     	|  1              	|  1.6904e-06     	|  8.8482e-08     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.6196e-06     	|  1              	|  1.6196e-06     	|  8.4777e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.3197e-06     	|  1              	|  1.3197e-06     	|  6.908e-08      	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.2498e-06     	|  1              	|  1.2498e-06     	|  6.5423e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.0198e-06     	|  1              	|  1.0198e-06     	|  5.3382e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
