Dataset: MNIST
Device: cuda
Batch Size: 32
Optimizer Parameters: lr=1e-05, betas=(0.9, 0.98), eps=1e-06, weight_decay=0.01
Classes: ['0 - zero', '1 - one', '2 - two', '3 - three', '4 - four', '5 - five', '6 - six', '7 - seven', '8 - eight', '9 - nine']
Epsilon:  0.5
Delta:  8.333333333333334e-06
Clip Param C:  0.1
DP-SGD with sampling rate = 0.0533% and noise_multiplier = 1.446459566579306 iterated over 93750 steps satisfies differential privacy with eps = 0.5 and delta = 8.333333333333334e-06.
Noise Scale:  1.446459566579306
**********
Num Epochs: 50
tensor(2.0586, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 0^th epoch *****
**** on training set *****
Accuracy Rate: 0.7722499966621399
*************************
**** on testing set *****
Accuracy Rate: 0.7701677083969116
*************************
tensor(1.9551, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.6406, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5996, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5156, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5117, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5195, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.7539, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4834, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4922, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3838, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3926, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5371, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4023, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3418, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5176, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5508, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3945, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4912, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5098, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 10^th epoch *****
**** on training set *****
Accuracy Rate: 0.9228500127792358
*************************
**** on testing set *****
Accuracy Rate: 0.9233226776123047
*************************
tensor(1.5371, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5781, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4785, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2754, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4336, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4062, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3867, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3906, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3730, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5244, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3428, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4941, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3828, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3008, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4912, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5527, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2891, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3623, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4512, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 20^th epoch *****
**** on training set *****
Accuracy Rate: 0.9395666718482971
*************************
**** on testing set *****
Accuracy Rate: 0.9357028603553772
*************************
tensor(1.5547, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5625, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4297, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4678, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5439, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3984, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4141, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4746, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5098, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3301, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4072, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3750, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4541, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2598, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3242, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4375, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4648, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2852, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4395, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 30^th epoch *****
**** on training set *****
Accuracy Rate: 0.9214833378791809
*************************
**** on testing set *****
Accuracy Rate: 0.9188298583030701
*************************
tensor(1.3428, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.6152, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4824, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3877, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4609, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.6973, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3145, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5244, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3662, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2754, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5449, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4531, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5547, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2744, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3027, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3027, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4238, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5215, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 40^th epoch *****
**** on training set *****
Accuracy Rate: 0.9394167065620422
*************************
**** on testing set *****
Accuracy Rate: 0.9340055584907532
*************************
tensor(1.3916, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2988, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3330, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5596, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4697, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.6133, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3613, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4229, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3506, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4268, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4326, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3008, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.6328, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3203, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3125, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4863, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3750, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
Training Time:  11530.615464687347
Accuracy Rate: 0.9388977289199829
------------------------------------
