lipdp.layers module
AddBias
¶
Bases: tf.keras.layers.Layer
Adds a bias to the input.
Remark: the euclidean norm of the bias must be bounded in advance. Note that this is the euclidean norm of the whole bias vector, not the norm of each element of the bias vector.
Warning: beware zero gradients outside the ball of norm norm_max. In the future, we might choose a smoother projection on the ball to ensure that the gradient remains non zero outside the ball.
Source code in lipdp/layers.py
535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 |
|
DPLayer
¶
Wrapper for created differentially private layers, instanciates abstract methods use for computing the bounds of the gradient relatively to the parameters and to the input.
Source code in lipdp/layers.py
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
|
backpropagate_inputs(input_bound, gradient_bound)
abstractmethod
¶
Applies the dilatation of the cotangeant vector norm (upstream gradient) by the Jacobian, i.e. multiply by the Lipschitz constant of the output wrt input.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
input_bound |
Maximum norm of input. |
required | |
gradient_bound |
Maximum norm of gradients (co-tangent vector) |
required |
Returns:
Type | Description |
---|---|
Maximum norm of tangent vector. |
Source code in lipdp/layers.py
53 54 55 56 57 58 59 60 61 62 63 64 65 |
|
backpropagate_params(input_bound, gradient_bound)
abstractmethod
¶
Corresponds to the Lipschitz constant of the output wrt the parameters, i.e. the norm of the Jacobian of the output wrt the parameters times the norm of the cotangeant vector.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
input_bound |
Maximum norm of input. |
required | |
gradient_bound |
Maximum norm of gradients (co-tangent vector) |
required |
Returns:
Type | Description |
---|---|
Maximum norm of tangent vector. |
Source code in lipdp/layers.py
40 41 42 43 44 45 46 47 48 49 50 51 |
|
propagate_inputs(input_bound)
abstractmethod
¶
Maximum norm of output of element.
Remark: when the layer is linear, this coincides with its Lipschitz constant * input_bound.
Source code in lipdp/layers.py
67 68 69 70 71 72 73 |
|
DP_AddBias
¶
Adds a bias to the input.
The bias is projected on the ball of norm norm_max
during training.
The projection on the ball is a 1-Lipschitz function, since the ball
is convex.
Source code in lipdp/layers.py
565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 |
|
DP_BoundedInput
¶
Bases: tf.keras.layers.Layer
, DPLayer
Input layer that clips the input to a given norm.
Remark: every pipeline should start with this layer.
Attributes:
Name | Type | Description |
---|---|---|
upper_bound |
Maximum norm of the input. |
|
enforce_clipping |
If True (default), the input is clipped to the given norm. |
Source code in lipdp/layers.py
182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 |
|
DP_ClipGradient
¶
Bases: tf.keras.layers.Layer
, DPLayer
Clips the gradient during the backward pass.
Behave like identity function during the forward pass. The clipping is done automatically during the backward pass.
Attributes:
Name | Type | Description |
---|---|---|
clip_value |
float
|
The maximum norm of the gradient. |
Source code in lipdp/layers.py
499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 |
|
DP_MaxPool2D
¶
Bases: tf.keras.layers.MaxPool2D
, DPLayer
Max pooling layer that preserves the gradient norm.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
layer_cls |
Class of the layer to wrap. |
required |
Returns:
Type | Description |
---|---|
A differentially private layer that doesn't have parameters. |
Source code in lipdp/layers.py
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 |
|
DP_ScaledL2NormPooling2D
¶
Bases: lip.layers.ScaledL2NormPooling2D
, DPLayer
Max pooling layer that preserves the gradient norm.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
layer_cls |
Class of the layer to wrap. |
required |
Returns:
Type | Description |
---|---|
A differentially private layer that doesn't have parameters. |
Source code in lipdp/layers.py
153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
|
DP_WrappedResidual
¶
Bases: tf.keras.layers.Layer
, DPLayer
Source code in lipdp/layers.py
647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 |
|
nm_coef
property
¶
Returns the norm multiplier coefficient of the layer.
Remark: this is a property to mimic the behavior of an attribute.
DP_GNP_Factory(layer_cls)
¶
Factory for creating differentially private gradient norm preserving layers that don't have parameters.
Remark: the layer is assumed to be GNP. This means that the gradient norm is preserved by the layer (i.e its Jacobian norm is 1). Pllease ensure that the layer is GNP before using this factory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
layer_cls |
Class of the layer to wrap. |
required |
Returns:
Type | Description |
---|---|
A differentially private layer that doesn't have parameters. |
Source code in lipdp/layers.py
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
|
clip_gradient(x, clip_value)
¶
Clips the gradient during the backward pass.
Behave like identity function during the forward pass.
Source code in lipdp/layers.py
483 484 485 486 487 488 489 490 491 492 493 494 495 496 |
|
make_residuals(merge_policy, wrapped_layers)
¶
Returns a list of layers that implement a residual block.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
merge_policy |
either "add" or "1-lip-add". |
required | |
wrapped_layers |
a list of layers that will be wrapped in residual blocks. |
required |
Returns:
Type | Description |
---|---|
A list of layers that implement a residual block. |
Source code in lipdp/layers.py
692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 |
|