
=== Start adding workers ===
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=0,shuffle=True)'}
=> Add worker SGDMWorker(index=0, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=1,shuffle=True)'}
=> Add worker SGDMWorker(index=1, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=2,shuffle=True)'}
=> Add worker SGDMWorker(index=2, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=3,shuffle=True)'}
=> Add worker SGDMWorker(index=3, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=4,shuffle=True)'}
=> Add worker SGDMWorker(index=4, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=5,shuffle=True)'}
=> Add worker SGDMWorker(index=5, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=6,shuffle=True)'}
=> Add worker SGDMWorker(index=6, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=7,shuffle=True)'}
=> Add worker SGDMWorker(index=7, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=8,shuffle=True)'}
=> Add worker SGDMWorker(index=8, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=9,shuffle=True)'}
=> Add worker SGDMWorker(index=9, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=10,shuffle=True)'}
=> Add worker SGDMWorker(index=10, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=11,shuffle=True)'}
=> Add worker SGDMWorker(index=11, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=12,shuffle=True)'}
=> Add worker SGDMWorker(index=12, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=13,shuffle=True)'}
=> Add worker SGDMWorker(index=13, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=14,shuffle=True)'}
=> Add worker SGDMWorker(index=14, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=15,shuffle=True)'}
=> Add worker SGDMWorker(index=15, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=16,shuffle=True)'}
=> Add worker SGDMWorker(index=16, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=17,shuffle=True)'}
=> Add worker SGDMWorker(index=17, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=18,shuffle=True)'}
=> Add worker SGDMWorker(index=18, momentum=0.9)
{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': True, 'download': True, 'batch_size': 64, 'shuffle': None, 'sampler': 'DistributedSampler(num_replicas=20,rank=19,shuffle=True)'}
=> Add worker SGDMWorker(index=19, momentum=0.9)

=== Start adding graph ===
<codes.graph_utils.Dumbbell object at 0x7f2ac5632730>

{'Type': 'Setup', 'Dataset': 'cifar10', 'data_dir': '/home/he/dero/datasets/', 'train': False, 'download': True, 'batch_size': 128, 'shuffle': False, 'sampler': None}
Train epoch 1
[E 1B0  |   1280/50000 (  3%) ] Loss: 2.3045 top1= 10.3125

=== Peeking data label distribution E1B0 ===
Worker 0 has targets: tensor([2, 4, 4, 3, 2], device='cuda:0')
Worker 1 has targets: tensor([1, 2, 0, 2, 3], device='cuda:0')
Worker 2 has targets: tensor([4, 4, 2, 4, 3], device='cuda:0')
Worker 3 has targets: tensor([0, 1, 3, 0, 2], device='cuda:0')
Worker 4 has targets: tensor([2, 4, 2, 1, 3], device='cuda:0')
Worker 5 has targets: tensor([2, 1, 4, 1, 4], device='cuda:0')
Worker 6 has targets: tensor([3, 4, 2, 1, 2], device='cuda:0')
Worker 7 has targets: tensor([0, 0, 4, 2, 2], device='cuda:0')
Worker 8 has targets: tensor([0, 4, 3, 0, 2], device='cuda:0')
Worker 9 has targets: tensor([3, 4, 3, 3, 1], device='cuda:0')
Worker 10 has targets: tensor([6, 7, 6, 9, 6], device='cuda:0')
Worker 11 has targets: tensor([9, 8, 6, 8, 8], device='cuda:0')
Worker 12 has targets: tensor([5, 8, 7, 6, 5], device='cuda:0')
Worker 13 has targets: tensor([7, 5, 9, 8, 9], device='cuda:0')
Worker 14 has targets: tensor([5, 9, 7, 7, 5], device='cuda:0')
Worker 15 has targets: tensor([8, 8, 5, 7, 9], device='cuda:0')
Worker 16 has targets: tensor([7, 7, 9, 6, 8], device='cuda:0')
Worker 17 has targets: tensor([5, 6, 9, 9, 5], device='cuda:0')
Worker 18 has targets: tensor([5, 9, 9, 9, 9], device='cuda:0')
Worker 19 has targets: tensor([7, 5, 8, 8, 9], device='cuda:0')


[E 1B10 |  14080/50000 ( 28%) ] Loss: 2.0836 top1= 18.6719
[E 1B20 |  26880/50000 ( 54%) ] Loss: 1.8314 top1= 18.9062
[E 1B30 |  39680/50000 ( 79%) ] Loss: 1.7107 top1= 20.3906

=> Averaged model (Global Average Validation Accuracy) | Eval Loss=2.3050 top1=  9.9860


=> Averaged model (Clique1 Average Validation Accuracy) | Eval Loss=3.1818 top1=  9.8958


=> Averaged model (Clique2 Average Validation Accuracy) | Eval Loss=3.2795 top1= 12.3998

Train epoch 2
[E 2B0  |   1280/50000 (  3%) ] Loss: 1.6351 top1= 21.6406
[E 2B10 |  14080/50000 ( 28%) ] Loss: 1.6519 top1= 22.6562
[E 2B20 |  26880/50000 ( 54%) ] Loss: 1.6000 top1= 26.0938
