List of Questions to Explore

[brandon-edwards]

Regarding the exploration of the MLPrivacy Meter during federated training using OpenFederatedLearning. Here are some questions raised during our video chat which can be very helpful for our integration and further development of our privacy meter.

About privacy meter
In the transfer learning setting, the base model is trained on dataset A. Now, we fine-tune the base model on dataset B and release the fine-tuned model B. Can we use the privacy meter to measure information leakage of model B on dataset A?
When using the shadow models to conduct membership inference attacks, is it necessary to train the shadow models for the same learning task as the target model? For example, can we use shadow models trained for classification tasks to conduct membership inference attacks on the target model built for face recognition?
Similar to the previous question, what if the adversary doesn’t know the loss used for training the target model? Can we still attack the target model with similar accuracy?
About privacy meter in federated learning:
What is the interaction required if we want to implement active attacks in FL to measure the privacy risk?
Who will use the privacy meter to measure the privacy risk (server or parties)? If the server measures privacy risk, what is the information server needs to broadcast to local parties in the end?
The interesting application is to enable parties to assert the privacy risk locally. There are two possibilities: (1) measuring the information leakage through the local update (gradients sent to the server) (2) measuring the information leakage through the aggregate update (aggregate gradient every party obtains).
Besides, by using the results of the privacy meter, what can we do? One potential application is to use the results to determine how much noise the parties need to add to ensure differential privacy.

[alexey-gruzdev]

Here is few clarifications from my side:

In the transfer learning setting, the base model is trained on dataset A. Now, we fine-tune the base model on dataset B and release the fine-tuned model B. Can we use the privacy meter to measure information leakage of model B on dataset A?

Maybe I will add a simple addition (I hope it would be simpler:) ). We are fine-tuning the model pre-trained, for example, on ImageNet.
Could we use privacy meter to understand if the fine-tuned model was initialized with ImageNet weights?

When using the shadow models to conduct membership inference attacks, is it necessary to train the shadow models for the same learning task as the target model? For example, can we use shadow models trained for classification tasks to conduct membership inference attacks on the target model built for face recognition?
Similar to the previous question, what if the adversary doesn’t know the loss used for training the target model? Can we still attack the target model with similar accuracy?

It would be great help to have a full list of information adversary should know to conduct inference attacks (model architecture, loss function, other hyperparameter used in training, etc.).

The interesting application is to enable parties to assert the privacy risk locally. There are two possibilities: (1) measuring the information leakage through the local update (gradients sent to the server) (2) measuring the information leakage through the aggregate update (aggregate gradient every party obtains).

Currently, in OpenFL we don't use gradients exchange. Instead, we are exchanging checkpoints, so the same questions are valid with one addition that we mean by local update checkpoint not gradients.

[brandon-edwards]

Thanks @alexey-gruzdev, I agree with the use of 'update' instead of 'gradient'. The update is the result of multiple changes to the model, each change performed by the optimizer as a function of an aggregate of gradients over a batch.
Also, to highlight the times we do use of gradients for measuring leakage, once the updated model is constructed, we use the gradients of certain layers with respect to individual samples (in addition to other features) as features to the attack model.

[changhongyan123]

Hi, I would like to add two new research questions here based on our meeting on Jul 7.

1. Can we reduce the computational time of training an attack model in each round of FL? The current plan is to train an attack model in each round of the FL because the parties may want to use PM results to decide if they wish to share more information. The issue is that the training attack model might become the bottleneck of the FL. The question is how to reduce the computation time.
2. When computing cumulative privacy risk, can we use the membership information from the previous round instead of the local model weights/global model weights from the previous round? The problem is that when computing cumulative privacy risk, we can concatenate the model weights from each round and call PM to compute the risk. However, it requires saving the model weights in each round. The question is how to reduce the memory requirement.