D-(DP)2SGD: Decentralized Parallel SGD with Differential Privacy in Dynamic Networks

Yuan Yuan, Zongrui Zou, Dong Li, Li Yan, Dongxiao Yu

Abstract: Decentralized machine learning has been playing an essential role in improving training efficiency. It has been applied in many real-world scenarios, such as edge computing and IoT. However, in fact, networks are dynamic, and there is a risk of information leaking during the communication process. To address this problem, we propose a decentralized parallel stochastic gradient descent algorithm (D-(DP)²SGD) with differential privacy in dynamic networks. With rigorous analysis, we show that D-(DP)²SGD converges with a rate of <span class="inline_break"><svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" width="25.481pt" style="vertical-align:-2.2681pt" id="M1" height="14.1172pt" version="1.1" viewBox="-0.0498162 -11.8491 25.481 14.1172"><g transform="matrix(.013,0,0,-0.013,0,0)"><path id="g113-80" d="M699 369C699 549 575 666 407 666C186 666 23 488 23 278C23 101 145 -16 312 -16C535 -16 699 153 699 369ZM600 373C600 210 500 19 321 19C186 19 120 129 120 272C120 450 232 631 399 631C541 631 600 522 600 373Z"/></g><g transform="matrix(.013,0,0,-0.013,9.387,0)"><path id="g113-41" d="M300 -147C201 -63 143 98 143 270S200 602 300 686L282 710C136 610 70 450 70 271V270C70 89 136 -72 282 -170L300 -147Z"/></g><g transform="matrix(.013,0,0,-0.013,13.885,0)"><path id="g113-50" d="M384 0V27C293 34 287 42 287 114V635C232 613 172 594 109 583V559L157 557C201 555 205 550 205 499V114C205 42 199 34 109 27V0H384Z"/></g><g transform="matrix(.013,0,0,-0.013,20.125,0)"><path id="g113-48" d="M368 703H309L44 -163H104L368 703Z"/></g></svg><span class="isep"/><svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" width="31.076pt" style="vertical-align:-2.2681pt" height="14.1172pt" version="1.1" viewBox="25.436183800000002 -11.8491 31.076 14.1172"><g transform="matrix(.013,0,0,-0.013,25.486,-1.608)"><path id="g119-37" d="M782 731H740L476 -79H474L280 352L133 274L151 239L242 286L473 -219L782 731Z"/></g><rect height="0.65243" width="16.4023" x="35.4556" y="-11.1469"/><g transform="matrix(.013,0,0,-0.013,35.456,0)"><path id="g113-76" d="M743 650H503L496 622L527 618C563 613 564 603 532 573C449 495 371 431 323 392C301 374 272 355 246 346L280 522C297 609 300 614 379 622L385 650H135L129 622C209 614 215 609 198 522L124 133C106 39 99 35 23 28L17 0H271L277 28C193 35 192 39 208 133L239 316C264 328 280 325 303 288C368 183 435 90 502 0H652L659 28C602 34 584 43 543 94C495 154 403 283 347 369L574 554C634 603 659 612 735 624L743 650Z"/></g><g transform="matrix(.013,0,0,-0.013,45.273,0)"><path id="g113-111" d="M495 86L479 114C446 82 419 66 409 66C401 66 401 72 406 97C420 166 436 231 453 297C489 435 454 448 428 448C406 448 384 439 354 422C305 394 222 327 161 247H159L183 345C200 415 194 448 173 448C143 448 82 410 23 351L38 325C64 349 95 371 105 371C111 371 116 365 109 336L25 -4L31 -12C50 -4 77 3 107 9C119 69 132 122 145 168C197 254 321 381 370 381C387 381 393 374 378 305L329 95C309 17 320 -12 345 -12C372 -12 430 19 495 86Z"/></g><g transform="matrix(.013,0,0,-0.013,51.858,0)"><path id="g113-42" d="M275 270C275 450 212 609 64 710L45 686C145 604 203 442 203 270S147 -63 45 -147L64 -170C213 -68 275 89 275 270Z"/></g></svg></span> while satisfying <span class="nowrap"><svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" style="vertical-align:-0.2063904pt" id="M2" height="6.1673pt" version="1.1" viewBox="-0.0498162 -5.96091 5.44961 6.1673" width="5.44961pt"><g transform="matrix(.013,0,0,-0.013,0,0)"><path id="g113-247" d="M387 375C387 402 357 448 257 448C172 448 82 404 82 326C82 289 108 255 156 241V239C85 223 23 181 23 116C23 39 89 -12 182 -12C265 -12 336 31 378 91L361 114C320 73 269 47 216 47C157 47 115 82 115 137C115 191 160 219 218 219C243 219 262 218 272 217L304 259L302 266C295 265 281 264 255 264C195 264 163 294 163 335C163 377 200 416 249 416C293 416 321 389 329 342C331 332 335 329 341 329C355 329 387 352 387 375Z"/></g></svg>-</span>DP, which achieves almost the same convergence rate as previous works without privacy concern. To the best of our knowledge, our algorithm is the first known decentralized parallel SGD algorithm that can implement in dynamic networks and take privacy-preserving into consideration.