A Study of Privacy Preservation in Machine Learning Systems

Abstract

As machine learning systems become more popular, and the quantity and scope of the data used to train them increases, questions regarding individual privacy naturally follow. This thesis is presented as an investigation into two methods of privacy preservation in machine learning: federated learning (FL) and computational proof based methods of model provenance. Our study of FL involves developing attacks that break its promise of private machine learning, using as a case study one of the largest real world deployments of federated learning, Google's Gboard. We develop methods that can recover private user data and highlight the ineffectiveness of current defense strategies. We ultimately conclude that, using Gboard as a case study, that the privacy benefit of FL requires re-evaluation. By considering the wider system implementation of which FL is a part of, we trace out aspects that can be exploited by parties whose honest participation is required for FL to be truly private. We then devote further research into other methods of privacy preservation, focusing heavily on model provenance. Here we address recent literature that has challenged the integrity of proof of learning (PoL), a computational proof-based method of certifying a model's training trajectory and data. Data forging attacks are recent developments against computational proofs of model provenance, and we scrutinise their performance and find that they are detectable in practice under informed verifiers. In addition, we provide theoretical evidence to the computational infeasibility of developing data forging attacks that are not detectable, and ultimately affirm the use of computational proofs for model provenance.