FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881521         	|  268.95         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  11.387         	|  20             	|  227.73         	|  84.674         	|
|  run_training_batch                                                                                                                                                                                         	|  0.0085645      	|  20000          	|  171.29         	|  63.688         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.0083324      	|  20000          	|  166.65         	|  61.962         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0039531      	|  20000          	|  79.063         	|  29.397         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0021374      	|  20000          	|  42.748         	|  15.894         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  29.079         	|  1              	|  29.079         	|  10.812         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.00084669     	|  20000          	|  16.934         	|  6.2962         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.25519        	|  66             	|  16.842         	|  6.2622         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00042062     	|  20000          	|  8.4124         	|  3.1278         	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.13965        	|  20             	|  2.7931         	|  1.0385         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00011786     	|  20066          	|  2.365          	|  0.87934        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  7.2945e-05     	|  20066          	|  1.4637         	|  0.54422        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.019548       	|  66             	|  1.2901         	|  0.47969        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  1.8254e-05     	|  20000          	|  0.36509        	|  0.13574        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.3836e-05     	|  20000          	|  0.27672        	|  0.10289        	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.2749e-05     	|  20000          	|  0.25499        	|  0.094807       	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.081078       	|  2              	|  0.16216        	|  0.060291       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.5996e-06     	|  20000          	|  0.091993       	|  0.034204       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.6839e-06     	|  20000          	|  0.073678       	|  0.027394       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.4992e-06     	|  20000          	|  0.069984       	|  0.026021       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.4669e-06     	|  20000          	|  0.069338       	|  0.025781       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.3566e-06     	|  20000          	|  0.067132       	|  0.02496        	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.032961       	|  2              	|  0.065922       	|  0.024511       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.2377e-06     	|  20000          	|  0.064753       	|  0.024076       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.6577e-06     	|  20000          	|  0.053154       	|  0.019763       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.6065e-06     	|  20000          	|  0.05213        	|  0.019382       	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.4906e-06     	|  20000          	|  0.049811       	|  0.01852        	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.4625e-06     	|  20000          	|  0.049251       	|  0.018312       	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.4321e-06     	|  20000          	|  0.048643       	|  0.018086       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.3809e-06     	|  20000          	|  0.047618       	|  0.017705       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.3746e-06     	|  20000          	|  0.047493       	|  0.017658       	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.3735e-06     	|  20000          	|  0.047469       	|  0.01765        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.3719e-06     	|  20000          	|  0.047439       	|  0.017638       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.3609e-06     	|  20000          	|  0.047218       	|  0.017556       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.3316e-06     	|  20000          	|  0.046632       	|  0.017338       	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.3138e-06     	|  20000          	|  0.046277       	|  0.017206       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.3099e-06     	|  20000          	|  0.046199       	|  0.017177       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.2837e-06     	|  20000          	|  0.045675       	|  0.016982       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.1729e-06     	|  20000          	|  0.043458       	|  0.016158       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.0894e-06     	|  20000          	|  0.041788       	|  0.015537       	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.1473e-06     	|  20066          	|  0.023022       	|  0.00856        	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.1193e-06     	|  20000          	|  0.022385       	|  0.0083231      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.0705e-06     	|  20000          	|  0.02141        	|  0.0079605      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.0416e-06     	|  20066          	|  0.0209         	|  0.0077709      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00028679     	|  66             	|  0.018928       	|  0.0070378      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  9.4248e-07     	|  20000          	|  0.01885        	|  0.0070085      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  8.9595e-07     	|  20000          	|  0.017919       	|  0.0066625      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  8.5925e-07     	|  20000          	|  0.017185       	|  0.0063896      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  8.2768e-07     	|  20000          	|  0.016554       	|  0.0061548      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  8.2731e-07     	|  20000          	|  0.016546       	|  0.0061521      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  8.1763e-07     	|  20000          	|  0.016353       	|  0.0060801      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  7.7545e-07     	|  20000          	|  0.015509       	|  0.0057664      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0078646      	|  1              	|  0.0078646      	|  0.0029242      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00024087     	|  20             	|  0.0048174      	|  0.0017912      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00021166     	|  20             	|  0.0042331      	|  0.0015739      	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.8894e-05     	|  66             	|  0.003227       	|  0.0011998      	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00011202     	|  20             	|  0.0022404      	|  0.00083302     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.0004821      	|  2              	|  0.00096419     	|  0.0003585      	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00041209     	|  2              	|  0.00082418     	|  0.00030644     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00034623     	|  2              	|  0.00069245     	|  0.00025746     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00030571     	|  2              	|  0.00061142     	|  0.00022733     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00057776     	|  1              	|  0.00057776     	|  0.00021482     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00050616     	|  1              	|  0.00050616     	|  0.0001882      	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00036236     	|  1              	|  0.00036236     	|  0.00013473     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  4.3698e-06     	|  66             	|  0.00028841     	|  0.00010723     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00026744     	|  1              	|  0.00026744     	|  9.9437e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  2.8784e-06     	|  66             	|  0.00018998     	|  7.0636e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.2438e-06     	|  66             	|  0.00014809     	|  5.5061e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.1485e-06     	|  66             	|  0.0001418      	|  5.2723e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  5.6341e-06     	|  20             	|  0.00011268     	|  4.1896e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  1.5989e-06     	|  66             	|  0.00010553     	|  3.9237e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.6819e-06     	|  20             	|  9.3639e-05     	|  3.4816e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.729e-06      	|  22             	|  8.2038e-05     	|  3.0503e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.1328e-06     	|  22             	|  6.8923e-05     	|  2.5626e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  9.7802e-07     	|  66             	|  6.4549e-05     	|  2.4e-05        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.075e-05      	|  2              	|  6.15e-05       	|  2.2866e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  9.2212e-07     	|  66             	|  6.086e-05      	|  2.2629e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  8.5129e-07     	|  66             	|  5.6185e-05     	|  2.089e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.594e-06      	|  20             	|  5.188e-05      	|  1.929e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  5.0819e-05     	|  1              	|  5.0819e-05     	|  1.8895e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.1836e-06     	|  22             	|  4.8039e-05     	|  1.7862e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.1105e-06     	|  22             	|  4.643e-05      	|  1.7263e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.1026e-06     	|  22             	|  4.6258e-05     	|  1.7199e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.1022e-06     	|  22             	|  4.6249e-05     	|  1.7196e-05     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.4599e-06     	|  22             	|  3.2118e-05     	|  1.1942e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.5435e-05     	|  2              	|  3.087e-05      	|  1.1478e-05     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.5321e-06     	|  20             	|  3.0641e-05     	|  1.1393e-05     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.3782e-06     	|  22             	|  3.032e-05      	|  1.1273e-05     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.112e-06      	|  20             	|  2.2241e-05     	|  8.2694e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  8.1751e-06     	|  2              	|  1.635e-05      	|  6.0792e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  5.6149e-06     	|  2              	|  1.123e-05      	|  4.1754e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.0979e-05     	|  1              	|  1.0979e-05     	|  4.0823e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.049e-05      	|  1              	|  1.049e-05      	|  3.9005e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  1.0259e-05     	|  1              	|  1.0259e-05     	|  3.8146e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  9.1288e-06     	|  1              	|  9.1288e-06     	|  3.3942e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.9698e-06     	|  2              	|  7.9395e-06     	|  2.952e-06      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.6252e-06     	|  2              	|  7.2503e-06     	|  2.6958e-06     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  6.9588e-06     	|  1              	|  6.9588e-06     	|  2.5874e-06     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  6.8005e-06     	|  1              	|  6.8005e-06     	|  2.5285e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  3.3146e-06     	|  2              	|  6.6292e-06     	|  2.4648e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  3.2093e-06     	|  2              	|  6.4187e-06     	|  2.3865e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  3.0552e-06     	|  2              	|  6.1104e-06     	|  2.2719e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  3.0403e-06     	|  2              	|  6.0806e-06     	|  2.2608e-06     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.8906e-06     	|  1              	|  5.8906e-06     	|  2.1902e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  5.4901e-06     	|  1              	|  5.4901e-06     	|  2.0413e-06     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  5.2797e-06     	|  1              	|  5.2797e-06     	|  1.963e-06      	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.9993e-06     	|  1              	|  4.9993e-06     	|  1.8588e-06     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.8792e-06     	|  1              	|  4.8792e-06     	|  1.8141e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  4.7898e-06     	|  1              	|  4.7898e-06     	|  1.7809e-06     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  4.7302e-06     	|  1              	|  4.7302e-06     	|  1.7587e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.61e-06       	|  1              	|  4.61e-06       	|  1.7141e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  4.5896e-06     	|  1              	|  4.5896e-06     	|  1.7065e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.5393e-06     	|  1              	|  4.5393e-06     	|  1.6878e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.5197e-06     	|  1              	|  4.5197e-06     	|  1.6805e-06     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.4107e-06     	|  1              	|  4.4107e-06     	|  1.64e-06       	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.1351e-06     	|  2              	|  4.2701e-06     	|  1.5877e-06     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  4.1397e-06     	|  1              	|  4.1397e-06     	|  1.5392e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.0494e-06     	|  2              	|  4.0988e-06     	|  1.524e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  3.8901e-06     	|  1              	|  3.8901e-06     	|  1.4464e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  1.9353e-06     	|  2              	|  3.8706e-06     	|  1.4391e-06     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  3.8398e-06     	|  1              	|  3.8398e-06     	|  1.4277e-06     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  3.77e-06       	|  1              	|  3.77e-06       	|  1.4017e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  3.7197e-06     	|  1              	|  3.7197e-06     	|  1.383e-06      	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.7448e-06     	|  2              	|  3.4897e-06     	|  1.2975e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.6652e-06     	|  2              	|  3.3304e-06     	|  1.2383e-06     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  3.1702e-06     	|  1              	|  3.1702e-06     	|  1.1787e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.8796e-06     	|  1              	|  2.8796e-06     	|  1.0707e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.8098e-06     	|  1              	|  2.8098e-06     	|  1.0447e-06     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.7502e-06     	|  1              	|  2.7502e-06     	|  1.0226e-06     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.74e-06       	|  1              	|  2.74e-06       	|  1.0187e-06     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  2.4997e-06     	|  1              	|  2.4997e-06     	|  9.2941e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.4205e-06     	|  1              	|  2.4205e-06     	|  8.9997e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.2696e-06     	|  1              	|  2.2696e-06     	|  8.4388e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.21e-06       	|  1              	|  2.21e-06       	|  8.2172e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.1597e-06     	|  1              	|  2.1597e-06     	|  8.0302e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.1197e-06     	|  1              	|  2.1197e-06     	|  7.8813e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.0191e-06     	|  1              	|  2.0191e-06     	|  7.5073e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.9493e-06     	|  1              	|  1.9493e-06     	|  7.2476e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.8608e-06     	|  1              	|  1.8608e-06     	|  6.9186e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.4808e-06     	|  1              	|  1.4808e-06     	|  5.5058e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.3197e-06     	|  1              	|  1.3197e-06     	|  4.9067e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  1.2005e-06     	|  1              	|  1.2005e-06     	|  4.4635e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881521         	|  580.76         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  26.744         	|  20             	|  534.89         	|  92.101         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.011652       	|  20000          	|  233.04         	|  40.126         	|
|  run_training_batch                                                                                                                                                                                         	|  0.011187       	|  20000          	|  223.75         	|  38.526         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.010816       	|  20000          	|  216.33         	|  37.249         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0057674      	|  20000          	|  115.35         	|  19.862         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0025267      	|  20000          	|  50.535         	|  8.7015         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.6671         	|  66             	|  44.028         	|  7.5812         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  36.277         	|  1              	|  36.277         	|  6.2465         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00055882     	|  20000          	|  11.176         	|  1.9244         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00023922     	|  20066          	|  4.8002         	|  0.82654        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00015327     	|  20066          	|  3.0755         	|  0.52956        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.14768        	|  20             	|  2.9537         	|  0.50859        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.018281       	|  66             	|  1.2066         	|  0.20776        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.4159e-05     	|  20000          	|  0.68317        	|  0.11763        	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.5096e-05     	|  20000          	|  0.50193        	|  0.086426       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.1508e-05     	|  20000          	|  0.43016        	|  0.074068       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.064547       	|  2              	|  0.12909        	|  0.022228       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  6.177e-06      	|  20000          	|  0.12354        	|  0.021272       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  6.0277e-06     	|  20000          	|  0.12055        	|  0.020758       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  6.0158e-06     	|  20000          	|  0.12032        	|  0.020717       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  4.637e-06      	|  20000          	|  0.092739       	|  0.015969       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  4.4088e-06     	|  20000          	|  0.088177       	|  0.015183       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  4.2243e-06     	|  20000          	|  0.084487       	|  0.014548       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.8641e-06     	|  20000          	|  0.077282       	|  0.013307       	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  3.8324e-06     	|  20066          	|  0.076901       	|  0.013242       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  3.8249e-06     	|  20000          	|  0.076498       	|  0.013172       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  3.808e-06      	|  20000          	|  0.076159       	|  0.013114       	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  3.7166e-06     	|  20000          	|  0.074331       	|  0.012799       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.6486e-06     	|  20000          	|  0.072971       	|  0.012565       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  3.5996e-06     	|  20000          	|  0.071992       	|  0.012396       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  3.4967e-06     	|  20000          	|  0.069935       	|  0.012042       	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  3.1847e-06     	|  20000          	|  0.063693       	|  0.010967       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  3.1443e-06     	|  20000          	|  0.062887       	|  0.010828       	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  3.1173e-06     	|  20000          	|  0.062346       	|  0.010735       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.9342e-06     	|  20000          	|  0.058684       	|  0.010105       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.785e-06      	|  20000          	|  0.0557         	|  0.0095909      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.6998e-06     	|  20000          	|  0.053996       	|  0.0092975      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.6901e-06     	|  20000          	|  0.053802       	|  0.0092641      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.5407e-06     	|  20000          	|  0.050813       	|  0.0087494      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.5357e-06     	|  20000          	|  0.050715       	|  0.0087325      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.8454e-06     	|  20000          	|  0.036908       	|  0.0063551      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.6341e-06     	|  20000          	|  0.032682       	|  0.0056274      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.6167e-06     	|  20000          	|  0.032334       	|  0.0055675      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.51e-06       	|  20000          	|  0.030201       	|  0.0052003      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00043724     	|  66             	|  0.028858       	|  0.0049689      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.4115e-06     	|  20000          	|  0.02823        	|  0.0048608      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.4103e-06     	|  20000          	|  0.028205       	|  0.0048566      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.3474e-06     	|  20066          	|  0.027036       	|  0.0046553      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.2629e-06     	|  20000          	|  0.025259       	|  0.0043492      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.104e-06      	|  20000          	|  0.022079       	|  0.0038018      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.0877e-06     	|  20000          	|  0.021753       	|  0.0037456      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.017054       	|  1              	|  0.017054       	|  0.0029365      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00058387     	|  20             	|  0.011677       	|  0.0020107      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0037418      	|  2              	|  0.0074837      	|  0.0012886      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00027363     	|  20             	|  0.0054726      	|  0.00094232     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  3.6374e-05     	|  66             	|  0.0024007      	|  0.00041337     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00010771     	|  20             	|  0.0021542      	|  0.00037092     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00050708     	|  2              	|  0.0010142      	|  0.00017462     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00045123     	|  2              	|  0.00090245     	|  0.00015539     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00043467     	|  2              	|  0.00086934     	|  0.00014969     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  0.00072558     	|  1              	|  0.00072558     	|  0.00012494     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00069771     	|  1              	|  0.00069771     	|  0.00012014     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00031558     	|  2              	|  0.00063117     	|  0.00010868     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  5.507e-06      	|  66             	|  0.00036346     	|  6.2584e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00033271     	|  1              	|  0.00033271     	|  5.7288e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00031761     	|  1              	|  0.00031761     	|  5.4688e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.0002514      	|  1              	|  0.0002514      	|  4.3288e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.6919e-06     	|  66             	|  0.00024367     	|  4.1957e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.1149e-06     	|  66             	|  0.00020559     	|  3.5399e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.9919e-06     	|  66             	|  0.00019746     	|  3.4001e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.8886e-06     	|  66             	|  0.00019065     	|  3.2828e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.9498e-06     	|  66             	|  0.00012869     	|  2.2158e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  5.9294e-06     	|  20             	|  0.00011859     	|  2.0419e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.6584e-06     	|  20             	|  0.00011317     	|  1.9486e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.6158e-06     	|  66             	|  0.00010664     	|  1.8363e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.3061e-06     	|  66             	|  8.6199e-05     	|  1.4843e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.8955e-06     	|  22             	|  8.57e-05       	|  1.4757e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.4204e-06     	|  22             	|  7.5248e-05     	|  1.2957e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.4899e-05     	|  2              	|  6.9799e-05     	|  1.2019e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.2986e-06     	|  20             	|  6.5971e-05     	|  1.1359e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.4935e-06     	|  22             	|  5.4858e-05     	|  9.4458e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.4073e-06     	|  22             	|  5.2961e-05     	|  9.1192e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.3964e-06     	|  22             	|  5.2721e-05     	|  9.078e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.3904e-06     	|  22             	|  5.2588e-05     	|  9.055e-06      	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  2.3541e-06     	|  22             	|  5.1791e-05     	|  8.9178e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.7321e-06     	|  20             	|  3.4641e-05     	|  5.9649e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.6668e-06     	|  20             	|  3.3336e-05     	|  5.74e-06       	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.3858e-06     	|  22             	|  3.0487e-05     	|  5.2495e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  8.0452e-06     	|  2              	|  1.609e-05      	|  2.7706e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.38e-05       	|  1              	|  1.38e-05       	|  2.3763e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.011e-05      	|  1              	|  1.011e-05      	|  1.7409e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  4.9593e-06     	|  2              	|  9.9186e-06     	|  1.7079e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  4.2696e-06     	|  2              	|  8.5393e-06     	|  1.4704e-06     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  3.5502e-06     	|  2              	|  7.1004e-06     	|  1.2226e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.4547e-06     	|  2              	|  6.9095e-06     	|  1.1897e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  6.2995e-06     	|  1              	|  6.2995e-06     	|  1.0847e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  6.1803e-06     	|  1              	|  6.1803e-06     	|  1.0642e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  6.1085e-06     	|  1              	|  6.1085e-06     	|  1.0518e-06     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  5.6699e-06     	|  1              	|  5.6699e-06     	|  9.7629e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.6654e-06     	|  2              	|  5.3309e-06     	|  9.1792e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.6096e-06     	|  2              	|  5.2191e-06     	|  8.9867e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.6049e-06     	|  2              	|  5.2098e-06     	|  8.9707e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.5895e-06     	|  2              	|  5.1791e-06     	|  8.9178e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.5346e-06     	|  2              	|  5.0692e-06     	|  8.7285e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  4.8401e-06     	|  1              	|  4.8401e-06     	|  8.334e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.6799e-06     	|  1              	|  4.6799e-06     	|  8.0582e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.3306e-06     	|  2              	|  4.6613e-06     	|  8.0262e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.6203e-06     	|  1              	|  4.6203e-06     	|  7.9556e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  4.5905e-06     	|  1              	|  4.5905e-06     	|  7.9043e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.2105e-06     	|  1              	|  4.2105e-06     	|  7.25e-07       	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  3.8706e-06     	|  1              	|  3.8706e-06     	|  6.6647e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.6201e-06     	|  1              	|  3.6201e-06     	|  6.2333e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  3.4403e-06     	|  1              	|  3.4403e-06     	|  5.9238e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  3.3705e-06     	|  1              	|  3.3705e-06     	|  5.8035e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  3.2298e-06     	|  1              	|  3.2298e-06     	|  5.5614e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.6144e-06     	|  2              	|  3.2289e-06     	|  5.5598e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  1.5497e-06     	|  2              	|  3.0994e-06     	|  5.3369e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.0296e-06     	|  1              	|  3.0296e-06     	|  5.2166e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.97e-06       	|  1              	|  2.97e-06       	|  5.114e-07      	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.8601e-06     	|  1              	|  2.8601e-06     	|  4.9247e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.8098e-06     	|  1              	|  2.8098e-06     	|  4.8381e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.7996e-06     	|  1              	|  2.7996e-06     	|  4.8205e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.6794e-06     	|  1              	|  2.6794e-06     	|  4.6136e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  2.6394e-06     	|  1              	|  2.6394e-06     	|  4.5447e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.6301e-06     	|  1              	|  2.6301e-06     	|  4.5286e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.6096e-06     	|  1              	|  2.6096e-06     	|  4.4934e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  1.2903e-06     	|  2              	|  2.5807e-06     	|  4.4436e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.5798e-06     	|  1              	|  2.5798e-06     	|  4.442e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.5602e-06     	|  1              	|  2.5602e-06     	|  4.4084e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.5397e-06     	|  1              	|  2.5397e-06     	|  4.3731e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  2.4997e-06     	|  1              	|  2.4997e-06     	|  4.3041e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  2.4801e-06     	|  1              	|  2.4801e-06     	|  4.2705e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.3199e-06     	|  1              	|  2.3199e-06     	|  3.9946e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.2398e-06     	|  1              	|  2.2398e-06     	|  3.8567e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.75e-06       	|  1              	|  1.75e-06       	|  3.0132e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.7202e-06     	|  1              	|  1.7202e-06     	|  2.9619e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  1.6904e-06     	|  1              	|  1.6904e-06     	|  2.9106e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.5004e-06     	|  1              	|  1.5004e-06     	|  2.5834e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.3998e-06     	|  1              	|  1.3998e-06     	|  2.4103e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.3905e-06     	|  1              	|  1.3905e-06     	|  2.3942e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.3802e-06     	|  1              	|  1.3802e-06     	|  2.3766e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.1707e-06     	|  1              	|  1.1707e-06     	|  2.0158e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
