Skip to content

lipdp.sensitivity module

get_max_epochs(epsilon_max, model, epochs_max=1024)

Return the maximum number of epochs to reach a given epsilon_max value.

The computation of (epsilon, delta) is slow since it involves solving a minimization problem (mandatory to go from RDP accountant to DP results). Hence each call takes typically around 1s. This function is used to avoid calling get_eps_delta too many times be leveraging the fact that epsilon is a non-decreasing function of the number of epochs: we unlocks the dichotomy search.

Hence the number of calls is typically log2(epochs_max) + 1. The maximum of epochs is set to 1024 by default to avoid too long computation times, even in high privacy regimes.

Parameters:

Name Type Description Default
epsilon_max

The maximum value of epsilon we want to reach.

required
model

The model used to compute the values of epsilon.

required
epochs_max

The maximum number of epochs to reach epsilon_max. Defaults to 1024. If None, the dichotomy search is used to find the upper bound.

1024

Returns:

Type Description

The maximum number of epochs to reach epsilon_max.

Source code in lipdp/sensitivity.py
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
def get_max_epochs(epsilon_max, model, epochs_max=1024):
    """Return the maximum number of epochs to reach a given epsilon_max value.

    The computation of (epsilon, delta) is slow since it involves solving a minimization problem
    (mandatory to go from RDP accountant to DP results). Hence each call takes typically around 1s.
    This function is used to avoid calling get_eps_delta too many times be leveraging the fact that
    epsilon is a non-decreasing function of the number of epochs: we unlocks the dichotomy search.

    Hence the number of calls is typically log2(epochs_max) + 1.
    The maximum of epochs is set to 1024 by default to avoid too long computation times, even in high
    privacy regimes.

    Args:
        epsilon_max: The maximum value of epsilon we want to reach.
        model: The model used to compute the values of epsilon.
        epochs_max: The maximum number of epochs to reach epsilon_max. Defaults to 1024.
                    If None, the dichotomy search is used to find the upper bound.

    Returns:
        The maximum number of epochs to reach epsilon_max."""
    steps_per_epoch = model.dataset_metadata.nb_steps_per_epochs

    def fun(epoch):
        if epoch == 0:
            epsilon = 0
        else:
            epoch = round(epoch)
            niter = (epoch + 1) * steps_per_epoch
            epsilon, _ = get_eps_delta(model, niter)
        return epsilon

    # dichotomy search on the number of epochs.
    if epochs_max is None:
        epochs_max = 512
        epsilon = 0
        while epsilon < epsilon_max:
            epochs_max *= 2
            epsilon = fun(epochs_max)
            print(f"epochs_max = {epochs_max} at epsilon = {epsilon}")

    epochs_min = 0

    while epochs_max - epochs_min > 1:
        epoch = (epochs_max + epochs_min) / 2
        epsilon = fun(epoch)
        if epsilon < epsilon_max:
            epochs_min = epoch
        else:
            epochs_max = epoch
        print(
            f"epoch bounds = {epochs_min, epochs_max} and epsilon = {epsilon} at epoch {epoch}"
        )

    return int(round(epoch))

gradient_norm_check(K_list, model, examples)

Verifies that the values of per-sample gradients on a layer never exceede a theoretical value determined by our theoretical work.

Args

Klist: The list of theoretical upper bounds we have identified for each layer and want to put to the test. model: The model containing the layers we are interested in. Layers must only have one trainable variable. Model must have a given input_shape or has to be built. examples: Relevant examples. Inputting the whole training set might prove very costly to check element wise Jacobians.

Returns

Boolean value. True corresponds to upper bound has been validated.

Source code in lipdp/sensitivity.py
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
def gradient_norm_check(K_list, model, examples):
    """
    Verifies that the values of per-sample gradients on a layer never exceede a theoretical value
    determined by our theoretical work.
    Args :
        Klist: The list of theoretical upper bounds we have identified for each layer and want to
        put to the test.
        model: The model containing the layers we are interested in. Layers must only have one trainable variable.
        Model must have a given input_shape or has to be built.
        examples: Relevant examples. Inputting the whole training set might prove very costly to check element wise Jacobians.
    Returns :
        Boolean value. True corresponds to upper bound has been validated.
    """
    image_axes = tuple(range(1, examples.ndim))
    example_norms = tf.math.reduce_euclidean_norm(examples, axis=image_axes)
    X_max = tf.reduce_max(example_norms).numpy()
    upper_bounds = np.array(K_list) * X_max
    assert len(model.layers) == len(upper_bounds)
    for layer, bound in zip(model.layers, upper_bounds):
        assert check_layer_gradient_norm(bound, layer, examples)