Skip to content

# lipdp.sensitivity module

## `get_max_epochs(epsilon_max, model, epochs_max=1024)`

Return the maximum number of epochs to reach a given epsilon_max value.

The computation of (epsilon, delta) is slow since it involves solving a minimization problem (mandatory to go from RDP accountant to DP results). Hence each call takes typically around 1s. This function is used to avoid calling get_eps_delta too many times be leveraging the fact that epsilon is a non-decreasing function of the number of epochs: we unlocks the dichotomy search.

Hence the number of calls is typically log2(epochs_max) + 1. The maximum of epochs is set to 1024 by default to avoid too long computation times, even in high privacy regimes.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`epsilon_max` |  | The maximum value of epsilon we want to reach. | required
`model` |  | The model used to compute the values of epsilon. | required
`epochs_max` |  | The maximum number of epochs to reach epsilon_max. Defaults to 1024. If None, the dichotomy search is used to find the upper bound. | `1024`

Returns:

Type | Description
--- | ---
| The maximum number of epochs to reach epsilon_max.

Source code in `lipdp/sensitivity.py`

    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    def get_max_epochs(epsilon_max, model, epochs_max=1024):
        """Return the maximum number of epochs to reach a given epsilon_max value.

        The computation of (epsilon, delta) is slow since it involves solving a minimization problem
        (mandatory to go from RDP accountant to DP results). Hence each call takes typically around 1s.
        This function is used to avoid calling get_eps_delta too many times be leveraging the fact that
        epsilon is a non-decreasing function of the number of epochs: we unlocks the dichotomy search.

        Hence the number of calls is typically log2(epochs_max) + 1.
        The maximum of epochs is set to 1024 by default to avoid too long computation times, even in high
        privacy regimes.

        Args:
            epsilon_max: The maximum value of epsilon we want to reach.
            model: The model used to compute the values of epsilon.
            epochs_max: The maximum number of epochs to reach epsilon_max. Defaults to 1024.
                        If None, the dichotomy search is used to find the upper bound.

        Returns:
            The maximum number of epochs to reach epsilon_max."""
        steps_per_epoch = model.dataset_metadata.nb_steps_per_epochs

        def fun(epoch):
            if epoch == 0:
                epsilon = 0
            else:
                epoch = round(epoch)
                niter = (epoch + 1) * steps_per_epoch
                epsilon, _ = get_eps_delta(model, niter)
            return epsilon

        # dichotomy search on the number of epochs.
        if epochs_max is None:
            epochs_max = 512
            epsilon = 0
            while epsilon < epsilon_max:
                epochs_max *= 2
                epsilon = fun(epochs_max)
                print(f"epochs_max = {epochs_max} at epsilon = {epsilon}")

        epochs_min = 0

        while epochs_max - epochs_min > 1:
            epoch = (epochs_max + epochs_min) / 2
            epsilon = fun(epoch)
            if epsilon < epsilon_max:
                epochs_min = epoch
            else:
                epochs_max = epoch
            print(
                f"epoch bounds = {epochs_min, epochs_max} and epsilon = {epsilon} at epoch {epoch}"
            )

        return int(round(epoch))

## `gradient_norm_check(K_list, model, examples)`

Verifies that the values of per-sample gradients on a layer never exceede a theoretical value determined by our theoretical work.

Args

Klist: The list of theoretical upper bounds we have identified for each layer and want to put to the test. model: The model containing the layers we are interested in. Layers must only have one trainable variable. Model must have a given input_shape or has to be built. examples: Relevant examples. Inputting the whole training set might prove very costly to check element wise Jacobians.

Returns

Boolean value. True corresponds to upper bound has been validated.

Source code in `lipdp/sensitivity.py`

     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    def gradient_norm_check(K_list, model, examples):
        """
        Verifies that the values of per-sample gradients on a layer never exceede a theoretical value
        determined by our theoretical work.
        Args :
            Klist: The list of theoretical upper bounds we have identified for each layer and want to
            put to the test.
            model: The model containing the layers we are interested in. Layers must only have one trainable variable.
            Model must have a given input_shape or has to be built.
            examples: Relevant examples. Inputting the whole training set might prove very costly to check element wise Jacobians.
        Returns :
            Boolean value. True corresponds to upper bound has been validated.
        """
        image_axes = tuple(range(1, examples.ndim))
        example_norms = tf.math.reduce_euclidean_norm(examples, axis=image_axes)
        X_max = tf.reduce_max(example_norms).numpy()
        upper_bounds = np.array(K_list) * X_max
        assert len(model.layers) == len(upper_bounds)
        for layer, bound in zip(model.layers, upper_bounds):
            assert check_layer_gradient_norm(bound, layer, examples)