babylonpolice.com

With the wide-spread application of machine learning models, it has become
critical to study the potential data leakage of models trained on sensitive
data. Recently, various membership inference (MI) attacks are proposed to
determine if a sample was part of the training set or not. The question is
whether these attacks can be reliably used in practice. We show that MI models
frequently misclassify neighboring nonmember samples of a member sample as
members. In other words, they have a high false positive rate on the
subpopulations of the exact member samples that they can identify. We then
showcase a practical application of MI attacks where this issue has a
real-world repercussion. Here, MI attacks are used by an external auditor
(investigator) to show to a judge/jury that an auditee unlawfully used
sensitive data. Due to the high false positive rate of MI attacks on member's
subpopulations, auditee challenges the credibility of the auditor by revealing
the performance of the MI attacks on these subpopulations. We argue that
current membership inference attacks can identify memorized subpopulations, but
they cannot reliably identify which exact sample in the subpopulation was used
during the training.