![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEiLQvL5pFQZ3D2NH8DXg_h7vPTF7IGiscC9zSISa7WCuX2GRvsrySJRcVsT1WWxhHMF72zmbOvWc_0wElk3bXpmVSyz4k0IVhONETBaLzTOI-C5Wrf-Q2Qk0TkPcHUy6w2-9mJChJLnVIy2RdKEgQE9eOg3bzbOhbCYgXTfWlufh7q4uInlW6f3KqCHvQ\/s1100\/DPTransfer.png\"){kind=spacer}
<\/p>\n
\nLarge deep learning models are becoming the workhorse of a variety of critical machine learning (ML) tasks. However, it has been shown that without any protection it is plausible for bad actors to attack a variety of models, across modalities, to reveal information from individual training examples. As such, it\u2019s essential to protect against this sort of information leakage.\n<\/p>\n
<\/p>\n \nDifferential privacy<\/a> (DP) provides formal protection against an attacker who aims to extract information about the training data. The most popular method for DP training in deep learning is differentially private stochastic gradient descent<\/a> (DP-SGD). The core recipe implements a common theme in DP: \u201cfuzzing\u201d an algorithm\u2019s outputs with noise to obscure the contributions of any individual input.\n<\/p>\n \nIn practice, DP training<\/a> can be very expensive or even ineffective for very large models. Not only does the computational cost typically increase when requiring privacy guarantees, but the noise also increases proportionally. Given these challenges, there has recently been much interest in developing methods that enable efficient<\/a><\/em> DP training. The goal is to develop simple and practical methods for producing high-quality large-scale private models.\n<\/p>\n \nThe ImageNet classification benchmark<\/a> is an effective test bed for this goal because 1) it is a challenging task even in the non-private setting, that requires sufficiently large models to successfully classify large numbers of varied images and 2) it is a public, open-source dataset, which other researchers can access and use for collaboration. With this approach, researchers may simulate a practical situation where a large model is required to train on private data with DP guarantees.\n<\/p>\n \nTo that end, today we discuss improvements we\u2019ve made in training high-utility, large-scale private models. First, in \u201cLarge-Scale Transfer Learning for Differentially Private Image Classification<\/a>\u201d, we share strong results on the challenging task of image classification on the ImageNet-1k dataset with DP constraints. We show that with a combination of large-scale transfer learning<\/a> and carefully chosen hyperparameters<\/a> it is indeed possible to significantly reduce the gap between private and non-private performance even on challenging tasks and high-dimensional models. Then in \u201cDifferentially Private Image Classification from Features<\/a>\u201d, we further show that privately fine-tuning just the last layer of pre-trained model with more advanced optimization algorithms improves the performance even further, leading to new state-of-the-art DP results across a variety of popular image classification benchmarks, including ImageNet-1k. To encourage further development in this direction and enable other researchers to verify our findings, we are also releasing the associated source code<\/a>.\n<\/p>\n \nThe main idea behind transfer learning is to reuse the knowledge gained from solving one problem and then apply it to a related problem. This is especially useful when there is limited or low-quality data available for the target problem as it allows us to leverage the knowledge gained from a larger and more diverse public dataset.\n<\/p>\n \nIn the context of DP, transfer learning has emerged as a promising technique to improve the accuracy of private models<\/a>, by leveraging knowledge learned from pre-training tasks. For example, if a model has already been trained on a large public dataset for a similar privacy-sensitive task, it can be fine-tuned on a smaller and more specific dataset for the target DP task. More specifically, one first pre-trains a model on a large dataset with no privacy concerns, and then privately fine-tunes the model on the sensitive dataset. In our work, we improve the effectiveness of DP transfer learning and illustrate it by simulating private training on publicly available datasets, namely ImageNet-1k, CIFAR-100, and CIFAR-10<\/a>.\n<\/p>\n \nTo start exploring how transfer learning can be effective for differentially private image classification tasks, we carefully examined hyperparameters affecting DP performance. Surprisingly, we found that with carefully chosen hyperparameters (e.g., initializing the last layer to zero and choosing large batch sizes), privately fine-tuning just the last layer of a pre-trained model yields significant improvements over the baseline. Training just the last layer also significantly improves the cost-utility ratio of training a high-quality image classification model with DP.\n<\/p>\n \nAs shown below, we compare the performance on ImageNet of the best hyperparameter recommendations both with and without privacy and across a variety of model and pre-training dataset sizes. We find that scaling the model and using a larger pre-training dataset decreases the gap in accuracy coming from the addition of the privacy guarantee. Typically, privacy guarantees of a system are characterized by a positive parameter \u03b5, with smaller \u03b5 corresponding to better privacy. In the following figure, we use the privacy guarantee of \u03b5 = 10.\n<\/p>\nTransfer learning and<\/em> differential privacy<\/h2>\n
Better pre-training improves DP performance<\/h2>\n
![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEgeXZ6zumu9happIaEL1sUtVwo4w7fVejo2S4uodu5uQWN0M4fVbzNWGNd_FEMu9ceUnQlE8mAbJ4CdSsALfqULkONss2w4YyVh09gBHpJ-zAQhghJ2l6Rd_BSgDrflT7PYLFTNZy-UEzJpWsfm5H39PqeP8K9vR8X74uVve_tsQP4BRbYR1_bS2B8-Ng\/w640-h464\/image1.png\")
<\/a><\/td>\n<\/tr>\n![](\"https:\/\/blogger.googleusercontent.com\/img\/b\/R29vZ2xl\/AVvXsEj4Gp5Z0q5vYQkXhOoFLaZUt0ah5hQzlDhZD2nzN35pqFDArejojDHQTiKN8v4j9JMc8wSdyrZwUXlWCXX1ha4VuRLokuoLJPRGy570hqZ1YdUlx_PQE0OXo-uhv63XQneAcejswYE4F_Idfz3-kA1rU5Q9dGIy-4tGPKo1EEUm2hFaEtMCFMSSieN2YQ\/s16000\/image2.png\"\/)
<\/a><\/td>\n<\/tr>\n