Dataset: MNIST
Device: cuda
Batch Size: 32
Optimizer Parameters: lr=1e-05, betas=(0.9, 0.98), eps=1e-06, weight_decay=0.01
Classes: ['0 - zero', '1 - one', '2 - two', '3 - three', '4 - four', '5 - five', '6 - six', '7 - seven', '8 - eight', '9 - nine']
Epsilon:  3
Delta:  8.333333333333334e-06
Clip Param C:  0.1
DP-SGD with sampling rate = 0.0533% and noise_multiplier = 0.6522512742032672 iterated over 93750 steps satisfies differential privacy with eps = 3 and delta = 8.333333333333334e-06.
Noise Scale:  0.6522512742032672
**********
Num Epochs: 50
tensor(1.8613, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 0^th epoch *****
**** on training set *****
Accuracy Rate: 0.8569833636283875
*************************
**** on testing set *****
Accuracy Rate: 0.8533346652984619
*************************
tensor(1.5371, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4551, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4297, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4805, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4316, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4561, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5156, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3359, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.6045, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4941, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3730, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2676, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4746, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3760, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4570, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5137, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4336, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3184, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4404, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 10^th epoch *****
**** on training set *****
Accuracy Rate: 0.9611999988555908
*************************
**** on testing set *****
Accuracy Rate: 0.9620606899261475
*************************
tensor(1.4902, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4404, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3418, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3262, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4092, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4297, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3486, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4102, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4941, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3770, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4375, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3984, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4619, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4102, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2441, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5566, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4570, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3926, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2520, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 20^th epoch *****
**** on training set *****
Accuracy Rate: 0.9788500070571899
*************************
**** on testing set *****
Accuracy Rate: 0.9756389856338501
*************************
tensor(1.3477, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4375, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4883, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3418, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3193, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3633, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4590, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4365, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2383, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2852, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2891, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2656, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2783, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4131, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3164, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5508, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4824, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4434, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3086, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 30^th epoch *****
**** on training set *****
Accuracy Rate: 0.9779000282287598
*************************
**** on testing set *****
Accuracy Rate: 0.9757388234138489
*************************
tensor(1.3965, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3281, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4482, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2910, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2861, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4248, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3760, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2969, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3428, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3125, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4141, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3945, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4727, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3525, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3184, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3066, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4023, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.5820, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
****the 40^th epoch *****
**** on training set *****
Accuracy Rate: 0.9738667011260986
*************************
**** on testing set *****
Accuracy Rate: 0.9739416837692261
*************************
tensor(1.5000, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2891, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2695, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3584, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2695, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2949, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3555, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3057, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3281, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3066, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4473, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3799, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3145, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.2666, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3027, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.4551, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
tensor(1.3652, device='cuda:0', dtype=torch.float16, grad_fn=<DivBackward0>)
Training Time:  11304.790875196457
Accuracy Rate: 0.9781349897384644
------------------------------------
