Improving Diversity in Black-box Few-shot Knowledge Distillation Tri-Nhan Vo1 , Dang Nguyen1 , Kien Do1 , and Sunil Gupta1
arXiv:2604.25795v1 [cs.CV] 28 Apr 2026
1 Applied Artificial Intelligence Institute (A2I2), Deakin University, Australia 1 {s223032975,
d.nguyen, k.do, sunil.gupta}@deakin.edu.au
Accepted at ECML-PKDD 2024
Abstract Knowledge distillation (KD) is a well-known technique to effectively compress a large network (teacher) to a smaller network (student) with little sacrifice in performance. However, most KD methods require a large training set and internal access to the teacher, which are rarely available due to various restrictions. These challenges have originated a more practical setting known as black-box few-shot KD, where the student is trained with few images and a black-box teacher. Recent approaches typically generate additional synthetic images but lack an active strategy to promote their diversity, a crucial factor for student learning. To address these problems, we propose a novel training scheme for generative adversarial networks, where we adaptively select high-confidence images under the teacher’s supervision and introduce them to the adversarial learning on-the-fly. Our approach helps expand and improve the diversity of the distillation set, significantly boosting student accuracy. Through extensive experiments, we achieve state-of-the-art results among other few-shot KD methods on seven image datasets. The code is available at https://github.com/votrinhan88/divbfkd.
1
Introduction
Over the recent years, deep learning models have made impressive progress—from classifying millions of images [1] to intelligently conversing with humans on various modalities and tasks. However, their ever-increasing sizes are a constant barrier to real-world deployment. This upscaling trend is the primary concern of model compression techniques, which aim to deliver powerful models at a deployable size (e.g., for mobile phones or embedded systems). While the removal of redundancy, quantization, or parameter sharing in the original model can help fulfill this goal, knowledge distillation (KD) [2] stands out for its efficiency and versatility. The main idea of KD is to transfer the knowledge from a complex pre-trained network (teacher ) to a simpler network (student). The pioneering solution was to train the student to simultaneously give correct predictions and mimic the teacher’s outputs [2]. With additional guidance, the student can approximate the teacher’s performance despite its smaller size. 1
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
Figure 1: Common KD methods (a) assume no constraints whereas black-box few-shot KD, and (b) only has access to a few images and the teacher’s predictive probabilities. This work has brought forth a new research area that most current KD methods are built upon [3, 4, 5]. The fruitful performance of KD typically relies on the availability of an extensive training set to train the student. Traditional KD methods often assume that the student is distilled with the same data used for training the teacher [2, 6, 7, 8]. However, in real-world scenarios, the distillation often happens at an external party side, where one can only afford a limitedsize training set. For example, to distill Google’s FaceNet (trained using approximately 200 million non-public face images [9]) into an in-house model without access to a large-scale dataset like Google’s, an external party may only be able to gather a few thousand images or even less. This scenario poses the few-shot setting and makes it more challenging than standard KD. Another assumption made by traditional KD methods is the requirement of a white-box teacher i.e., we have complete access to its parameters, activations, or gradients, etc. [10, 11, 12, 13]. However, the teacher model is often black-box, i.e., only its final predictions are accessible. Understandably, companies often avoid full disclosure of the model parameters as it is their competitive edge and intellectual property. For instance, even with a paid subscription to ChatGPT API [14], only text outputs and their prediction confidence can be retrieved. The realistic constraints in the black-box few-shot KD setting (Fig. 1) render most KD methods inapplicable and demand novel approaches. Recently, some methods have addressed the few-shot KD problem, but most of them still require a white-box teacher [12, 15]. To the best of our knowledge, only two methods apply a black-box teacher to few-shot KD, namely BBKD [16] and FS-BBT [17]. Both these methods leverage MixUp [18] to generate synthetic images. However, as MixUp linearly interpolates between two original images, the synthetic images are either very similar to the original images or do not have realistic semantics, adding little value to the student’s training. To overcome this problem, FS-BBT employs a conditional variational autoencoder (CVAE) [19] to generate out-of-distribution synthetic images. However, while the pixel-based loss in CVAE is beneficial for reconstructing images, it is not intended for promoting image diversity and have side-effect blurring artifacts [20]. We address the problem of black-box few-shot KD i.e., distilling the student with only a few images and a black-box teacher. To boost student performance, we expand the distillation set with diverse synthetic images. To achieve diversity in the synthetic images, we propose a novel training scheme for Wasserstein Generative Adversarial Network (WGAN) [21]: 2
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
• Firstly, we specifically define high-confidence images, which are synthetic images that the teacher can predict confidently w.r.t. a threshold. To avoid biased and abnormal images, we leverage the teacher to compute adaptive thresholds in a class-specific and dataset-agnostic manner. • Then, during the WGAN training loop, we select high-confidence images from the latest iteration on-the-fly and introduce them in the role of additional real images. Our WGAN learned with high-confidence images can generate diverse synthetic images close to the teacher’s training images. • Finally, we augment our few-shot images with synthetic images to improve the training of the student, leading to a significantly better performance. We summarize our contributions as follows: 1. We propose Diverse Black-box Few-Shot Knowledge Distillation (DivBFKD), a novel solution for black-box few-shot KD, which directly addresses the diversity aspect of synthetic images to improve distillation. 2. We propose a novel training scheme for WGAN, introducing high-confidence images efficiently and on-the-fly to the adversarial learning, which improves diversity in image generation. 3. We propose adaptive thresholds, a teacher-based criteria to avoid selecting biased and abnormal high-confidence images. The criteria can be adaptively applied to arbitrary classes and datasets. 4. We conduct comprehensive evaluation on multiple image classification tasks and architectures, achieving SOTA results among few-shot KD methods. We also provide thorough analysis of our method for further insights.
2
Related Works
2.1
Knowledge Distillation
Transferring knowledge from a large model (teacher) to a smaller model (student) was first introduced in [22], then formally popularized in [2]. Its key idea is to exploit the knowledge encoded in the teacher’s ‘softened’ probabilistic outputs, which are continuous and often more informative than the ‘hard’ class labels. The additional supervision helps the student capture underlying inter-class relationships and improve its performance. Existing KD methods often involve logits-based, feature-based, and relation-based knowledge [23]. Logits-based methods use the teacher’s logits (i.e., pre-softmax activations) as the guidance for the student [24, 2, 5], optionally with adaptive temperature [25] or an ensemble of teachers [26]. Feature-based methods extract knowledge from the teacher’s internal attributes e.g., parameters and inter-layer connections [27], intermediate features [10], attention maps [28], and feature-space probability distribution [29]. Finally, relation-based methods distill the knowledge in the relationships between different layers or data samples [30, 11, 31]. 3
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
Although useful, these methods would struggle when only a few images are available in the distillation set and we can only access the black-box teacher’s final predictions.
2.2
Knowledge distillation with limited data.
The fundamental challenge of distillation on limited data is that the few available images cannot fully represent the actual images manifold. Consequently, the student fails to generalize to an unseen set. Thus, few-shot KD methods leverage data augmentation or generative models to augment the distillation set [12, 15, 16, 17]. Their approaches also vary depending on the accessibility of the teacher. FSKD [12] uses trainable synthetic images that are part of the student’s parameters. These images require adversarial signals from a white-box teacher for optimization. WaGe [15] explores the data space in the approximate neighborhood of the few-shot data samples. This requires the logits of a white-box teacher for a Wassersteinbased loss. Most recently, BBKD [16] and FS-BBT [17] only require a black-box teacher for the distillation. Both use MixUp to construct synthetic images for a larger distillation set for the student. While BBKD embraces active learning to choose synthetic images for distillation, FS-BBT trains a CVAE as an additional source of synthetic images. FS-BBT is the current SOTA few-shot KD method.
2.3
Data-free knowledge distillation.
As an even more extreme setting than few-shot, data-free KD methods assume no training data to train the student. To compensate, they often extract the knowledge from a whitebox teacher e.g., its activation statistics [32], gradients [33, 13], and batch-norm layers [34]. So far, only two methods use black-box teachers, namely ZSDB3KD [35] and IDEAL [36]. Although the data-free KD setting has several applications e.g., model inversion attack [37], it is not really practical in the context of model compression. When we distill a large model, we often have data to validate the large and distilled models. Thus, few images can be also used for the distillation phase. As will be shown in §4.6., even with a small portion of data, our method significantly outperforms data-free KD methods.
2.4
Generative Adversarial Networks in Knowledge Distillation
Using Generative Adversarial Networks (GAN) to generate synthetic images is a popular approach in data-free KD methods [13, 38, 39, 36]. However, our WGAN is trained with a novel scheme with three-fold differences: 1. Generator: Existing methods typically rely on the teacher’s signals to tame the generator with extra losses e.g., one-hot, feature, and diversity losses. However, these losses are strictly inapplicable in our setting due to the absence of gradients from a black-box teacher. 2. Discriminator: Existing methods cannot use a discriminator due to the lack of real images. In contrast, since our method allows to use any available real images, it uses the discriminator to distinguish real/fake images. Moreover, our method also trains the discriminator with high-confidence images to boost the diversity in image generation. 4
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
3. Teacher: The teacher is trained for predicting image labels rather than distinguishing real/fake images. However, existing methods usually force the teacher to a discriminator’s role, exposing it to knowledge gap. In contrast, we use teacher in a way where its intended strength is guaranteed to support our WGAN training.
3
Framework
3.1
Problem statement
Given a black-box pre-trained teacher network T and a small set of labeled images D = {xi , yi }N i=1 , our goal is to train a student network S on D such that S can approximate the performance of T . A direct solution is to use standard KD [2], which trains S to match both the ground-truth labels yi and the teacher’s predictions T (xi ) via minimizing: h i LKD = E(xi ,yi )∼D (1 − λ)LCE (S(xi ), yi ) + λLKL (S(xi ), T (xi )) (1) where LCE and LKL are the cross-entropy and the Kullback-Leibler divergence losses, S(xi ) and T (xi ) are the student’s and teacher’s predictions (i.e., class probabilities), and λ is a coefficient to balance the losses. Note that the temperature factor in the original paper is not used as it requires access to the teacher’s logits, which violates our black-box teacher assumption. Moreover, with a small distillation set D, common KD approaches lead to a sub-optimal model as they require lots of training images [2, 6, 7, 8]. Proposed Method. To perform black-box few-shot KD, we propose our novel method DivBFKD. Our method has two phases: Generation and Distillation. In the Generation phase, we train a WGAN to produce synthetic images. In the Distillation phase, we train the student with the combined real and synthetic images.
3.2
Generation phase.
We train a WGAN model [40] with several key changes to generate diverse synthetic images. Like standard WGAN, our WGAN consists of a generator G and a discriminator D. Iteratively, G produces a synthetic image x̃ = G(z) from a random latent vector z ∼ N (0, I). Meanwhile, D predicts a scalar score indicating the likelihood of an input image being real. Jointly trained with opposite objectives, G tries to fool D by producing realistic images while D tries to best distinguish between real images x and synthetic images x̃. These objectives can be formulated into the generator loss LG and the discriminator loss LD : LG = −Ez∼N (0,I) [D(G(z))],
(2)
LD = Ez∼N (0,I) [D(G(z))] − Ex∼D [D(x)].
(3)
However, if we only use the few-shot set D to train the WGAN, G will generate synthetic images similar to the images in D [41, 42], which are not diverse enough to facilitate effective distillation. To improve diversity of images generated by G, we propose the use of synthetic 5
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
Confidence score
images that the teacher T can predict confidently. We refer to these images as high-confidence images. High-confidence images. Given a synthetically generated (fake) image x̃, we define its confidence-score cx̃ = max k ∈ {0, . . . , K − 1}T (x̃), by taking the maximum over T ’s predictive probability vector. For example, given three classes {0, 1, 2} and class probabilities T (x̃) = [0.1, 0.7, 0.2], then cx̃ = 0.7. We define x̃ a high-confidence image if cx̃ is above a confidence-threshold τ ∈ [0, 1]. Intuitively, when cx̃ is high, x̃ is likely close to the teacher’s training images as it can be confidently classified. Adaptive thresholds. The critical factor in determining a high-confidence image is the confidence-threshold τ . We observe that the teacher has a class-specific bias, i.e., it predicts different classes with different confidence levels. Take the dataset FMNIST [43] for example, the averaged confidence-score is > 0.99 for classes ‘trouser’ and ‘bag’, but is only 0.75 for class ‘shirt’ (Fig. 2). In this case, fixing a single value τ = 0.95 may lead to high-confidence images being too lenient for ‘trouser’ and ‘bags’, but too strict for the class ‘shirt’. Thus, a single τ for all classes will make high-confidence images imbalanced. Further, considering any single class, the confidence-scores have a long-tailed distribution, where they concentrate near 1 and decay with lower values. Even though τ can be simply set to the averaged confidence-score, it would provide little flexibility and interpretability and also make high-confidence images biased towards outliers.
1.00 0.75 0.50 0.25 at q = 0.1 0.00 p o ouser llover dress coat andal shirt eaker bag boot t / t r i s le tr pu sn t-sh ank
Class
Figure 2: Distributions of confidence-scores across classes on FMNIST. They concentrate around 1 and have a long-tail. Red bars show adaptive confidence-thresholds τ . To address these problems, we use adaptive thresholds {τ k }K−1 k=0 that is class-specific based on the q-quantile of the confidence-scores of real images. Specifically, for each of the K classes: (4) τ k = quantileq ({cxi | (xi , yi ) ∈ D and yi = k}N i=1 ), q ∈ [0, 1]. We choose q = 0.1, i.e., high-confidence images must have teacher’s confidence higher than at least 10% of those for the real ones, as perceived by the teacher T . This way, τ adaptively adjusts to any dataset and any class-wise bias from the teacher. Fig. 2 illustrates τ as red bars using q = 0.1 on FMNIST. Our WGAN training scheme. We aim to harness the teacher’s guidance to promote diversity of synthetic images. Fundamentally, synthetic images would be more diverse if the WGAN can learn on images that resemble the teacher’s unknown training images D∗ , whose diversity is superior to few-shot images D. While there is no direct access to D∗ , high-confidence images can serve as a satisfactory alternative in terms of both quantity (new 6
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
Figure 3: Illustration of our method DivBFKD. (a) Generation: We train our WGAN with the losses LG and Lnew D . When optimizing G, we construct the high-confidence set ỹ Hn = {x̃ = G(z) | cx̃ ≥ τ } from synthetic images x̃, with their confidence-score cx̃ , pseudolabel ỹ, and adaptive threshold τ ỹ . When optimizing D, we sample x from the combined real and high-confidence images set D ∪ Hn−1 . (b) Distillation: We augment few images in D with synthetic images in DG to train the student S on D ∪ DG . high-confidence images are generated at every WGAN training step) and quality (they likely reside close to D∗ , as we previously discussed). In other words, high-confidence images implicitly allow our WGAN to learn from D∗ . By this motivation, we introduce high-confidence images in the role of real images to the adversarial learning (Fig. 3(a)), exposing our WGAN to a wider variety of training images. At each n-th training step: • The generator G is trained with Eq 2. For a fake image x̃, we compute its corresponding confidence-score cx̃ and pseudo-label ỹ = arg maxk∈{0,...,K−1} T (x̃). Using the adaptive thresholds {τ k }K−1 k=0 computed with Eq 4, we add qualified x̃ to the high-confidence set Hn = {x̃ | cx̃ ≥ τ ỹ }. Then, Hn is stored to train the discriminator D in the next step. • Training the discriminator D requires both real and fake images. However, our ‘real’ images include real images that actually come from D and high-confidence images that come from the high-confidence set Hn−1 . Note that Hn−1 is constructed in the previous step, and H0 = ∅. With these changes, we train our discriminator with the new loss as: Lnew D = Ez∼N (0,I) [D(G(z))] − Ex∼(D∪Hn−1 ) [D(x)].
(5)
Viewing as an adversarial game, when D learns to appreciate high-confidence synthetic images as realistic, it also encourages G to create more synthetic images like those. This 7
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
interaction forms a positive feedback loop that helps G to be more diverse. Furthermore, we emphasize that forming Hn is a cheap operator and requires small memory. These modifications not only effectively improve the diversity of synthetic images, but also incur only trivial computation overhead. Summary of our novelty in WGAN. We introduce a new training scheme for WGAN using high-confidence images based on the teacher-supervised adaptive thresholds. Since we aim to generate images close to the teacher’s training images, our synthetic images are more diverse and better cover unseen images, significantly improving the student’s generalization, as shown in §4. Algorithm 1: Proposed method DivBFKD Input : teacher T ; labeled set D = {xi , yi }N i=1 ; quantile q; synthetic budget M ; batch size B Output: student network S Initialize generator G, discriminator D, student S, and high-confidence set H0 ← ∅ ▷ Stage 1: Generation k K 2 Infer T on D to compute adaptive thresholds {τ }k=1 with q Eq. 4 3 while G not converged do 4 Sample latent vectors {zi }B i=1 ∼ N (0, I) 5 Generate synthetic images {x̃i }B i=1 with G B 6 Sample {xi }i=1 ∼ (D ∪ Hn ) as real images 7 Update G on {x̃i }B Eq. 2 i=1 via LG B B new Eq. 5 8 Update D on {xi }i=1 and {x̃i }i=1 via LD 9 Compute pseudo-labels {ỹi } and confidence-scores {cx̃i } for {x̃i } B 10 Construct Hn = x̃i | cx̃i ≥ τ ỹi i=1 1
▷ Stage 2: Distillation Construct DKD with N real and M synthetic images 12 while S not converged do 13 Update S on DKD via LKD 11
14
Eq. 6 Eq. 7
return S
3.3
Distillation phase
After training WGAN in the Generation phase, we use it to generate synthetic images. We employ our generator G to build a set of synthetic images DG , and their pseudo-labels (i.e., class probabilities) are obtained via the teacher T . Following BBKD [16] and FS-BBT [17], we construct a distillation set DKD comprising of N real images from D and M synthetic images from DG : M DKD = {xi ∈ D}N (6) i=1 ∪ {G(zj )|zj ∼ N (0, I)}j=1 ,
8
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
where xi ∈ D and x̃j ∈ DG are real and synthetic images. Like other few-shot KD methods [16, 17], we train the student S with DKD using a cross-entropy loss: LKD = Exi ∼DKD LCE (S(xi ), T (xi )).
(7)
Class balancing. Inspired by [13, 36], we generate a balanced set of synthetic images. Recalling that M is the budget of synthetic images for KD and K is the number of classes, each class would have M/K synthetic images. We ensure class balance through rejection sampling until we achieve the sufficient amount. We summarize our method in Algorithm 1.
4
Experiments
We conduct a wide range of experiments to show our improvements over current SOTA fewshot KD baselines. We also include cross-architecture KD, visualization, ablation studies, and comparison with data-free KD methods. We provide more details in the Supplementary.
4.1
Architectures and datasets
We evaluate our method across seven benchmark image datasets: MNIST [44], FMNIST [43], SVHN [45], CIFAR10, CIFAR100 [46], Tiny-ImageNet [47], and Imagenette [48]. These datasets cover different difficulty levels, including: (1) Simple datasets with MNIST, FMNIST, SVHN, and CIFAR10 (10 classes) and (2) Complex datasets with CIFAR100 (100 classes), Tiny-ImageNet (200 classes), and Imagenette (high-resolution at 224×224). We consider several popular network architectures for distillation, namely LeNet5 [44], AlexNet [49], and ResNet [50], and VGG [51]. These architectures and datasets are widely used to evaluate few-shot KD methods [12, 16, 17].
4.2
Baselines
We compare our method DivBFKD with the following baselines: • Student-Full : The student is trained on the whole teacher’s training set. • Student-Alone: The student is trained on the few-shot set D. • Standard-KD [2]: The student is trained with the standard KD loss (Eq. 1) on D. We select temperature λ = 0.9, a standard value used in KD methods [2, 8, 52, 53]. • Few-shot KD methods: We compare with two white-box methods, FSKD [12] and WaGe [15], and two black-box methods, BBKD [16] and FS-BBT [17]. For fair comparisons, we use the same teacher-student network architecture and the same numbers of real and synthetic images N and M as in other few-shot KD methods [12, 16, 17]. Their accuracy numbers are obtained from [17]1 . For our method, we set the quantile q = 0.1 1
This is possible because we use benchmark datasets with fixed train and test splits.
9
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
to select high-confidence images across all experiments. We repeat each experiment three times with random seeds and report the averaged accuracy with standard deviation on the hold-out test set.
4.3
Distillation performance
4.3.1
Simple datasets Table 1: Classification accuracy (%) on Simple datasets. N and M show the budget of real and synthetic images. Dataset (N, M )
MNIST (2 K, 24 K)
FMNIST SVHN CIFAR10 (2 K, 48 K) (2 K, 40 K) (2 K, 40 K)
Teacher Student-Full
99.28 98.91±0.08
90.90 88.68±0.53
96.16 95.61±0.07
90.05 87.58±0.36
Smaller Architecture Student-Alone Standard-KD(B) FSKD(W) WaGe(W) FS-BBT(B) DivBFKD(B)
95.14±0.19 95.36±0.11 80.43† − 98.42† 98.58±0.06
79.24±1.05 80.76±0.37 68.64† 85.18† 84.73† 86.50±0.18
85.68±0.46 89.11±0.22 − − − 94.47±0.12
59.02±0.54 59.77±0.96 40.58† 73.08† 74.10† 76.97±0.56
Same Architecture BBKD(B) FS-BBT(B) DivBFKD(B)
98.74† 98.91† 98.97±0.04
80.90† 86.53† 87.65±0.37
− − 94.61±0.08
74.60† 76.17† 78.07±0.26
[(W)] white-box; [(B)] black-box; [−] not reported; [†] reported from respective paper; [bold] best results.
Following [17], we conduct distillation in two settings: 1. Smaller architecture: The student has a smaller architecture than the teacher: LeNet5 to LeNet5-Half on MNIST, FMNIST; and AlexNet to AlexNet-Half on SVHN, CIFAR10.2 2. Same architecture: The student has an identical architecture to the teacher. We report the results in Table 1 and make the following observations: • Our method DivBFKD improves by around 3-6% over Standard-KD on MNIST, FMNIST, and SVHN and by significantly 17% on CIFAR10. 2
The -Half version has half the channels in each layer.
10
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
• Compared to other few-shot KD methods, our DivBFKD also excels in both smaller and same architecture settings. It improves upon second-best FS-BBT by around 2% on FMNIST and CIFAR10. • The performance gap between our DivBFKD and its upper bound Student-Full is very close on MNIST (0.33%), FMNIST (2.18%), and SVHN (1.14%). We emphasize that Student-Full uses the teacher’s training set (≥ 60 K real images), much larger than the 2 K real images we started with. 4.3.2
Complex datasets
Similar to [16, 17, 15], we distill from ResNet32 to ResNet20 on CIFAR100 and TinyImageNet (Table 2). With limited data, Student-Alone loses roughly half of its accuracy compared to Student-Full. As a naive solution, Standard-KD improves over Student-Alone 10% on CIFAR100 and 14% on Tiny-ImageNet. Ultimately, our method DivBFKD is the best method, exceeding the second-best FS-BBT by 3% on CIFAR100 and 1% on TinyImageNet. We further evaluate our method on high-resolution images with Imagenette. On 2 K real images, Student-Alone achieves 76.80% while Standard-KD marginally improves to 76.94%. DivBFKD further improves upon this result by 10%, approximating Student-Full. Table 2: Classification accuracy (%) on Complex datasets. N and M show the budget of real and synthetic images. Dataset (N, M )
CIFAR100 (5 K, 40 K)
Tiny-ImageNet (10 K, 50 K)
Imagenette (2 K, 40 K)
Teacher Student-Full
71.41 68.28±0.20
56.27 51.51±0.31
90.47 89.89±0.41
Student-Alone Standard-KD(B) WaGe(W) BBKD(B) FS-BBT(B) DivBFKD(B)
35.71±0.39 45.76±0.48 20.32† 53.41† 56.28† 59.91±0.31
26.59±0.25 40.42±0.19 − 40.01† 43.29† 44.25±0.38
76.80±0.78 76.94±0.14 − − − 86.48±0.72
[(W)] white-box; [(B)] black-box; [−] not reported; [†] reported from respective paper; [bold] best results.
4.3.3
Cross-architecture distillation
In previous subsections, we distill from the teacher to a student networks of similar architectures to have a consistent setting with other few-shot KD methods (e.g., LeNet5 to LeNet5-Half, AlexNet to AlexNet-Half, and ResNet32 to ResNet20). This section further explores the cases where the teacher and student architectures belong to different families: 11
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
ResNet [50], VGG [51], and AlexNet [49]. We adopt the same setting from the standard experiment on CIFAR10 from §4.3.2. Table 3 shows the performance of our method DivBFKD with three teachers ResNet32, VGG16, and AlexNet combined with three students ResNet20, VGG11, and AlexNet-Half on CIFAR10. Table 3: Student accuracy (%) of cross-architecture distillation on CIFAR10. Architecture Teacher Student
Teacher
ResNet32 VGG16 AlexNet
ResNet20 ResNet20 ResNet20
93.34 91.30 90.05
ResNet32 VGG16 AlexNet
VGG11 VGG11 VGG11
93.34 91.30 90.05
ResNet32 AlexNet-Half VGG16 AlexNet-Half AlexNet AlexNet-Half
93.34 91.30 90.05
S-Full 92.34±0.06
89.12±0.17
87.58±0.36
Accuracy S-Alone Standard-KD
DivBFKD
64.52±1.60
65.86±1.90 64.76±0.89 65.29±0.89
84.05±0.10 80.69±0.48 79.57±0.16
53.34±0.57
53.78±0.78 53.80±1.21 53.51±0.14
69.42±0.83 75.41±0.06 74.60±0.29
59.02±0.54
60.36±0.82 60.47±0.10 59.77±0.96
74.70±0.31 76.16±0.57 76.97±0.56
Same teacher-student pairs share the same Student-Full and Student-Alone accuracy.
We have the following observations: • Among three architecture families, ResNet is the best one with a 93.34% Teacher, 92.34% Student-Full, and 64.52% Student-Alone. • Standard-KD is helpful and it is always better than Student-Alone in all teacherstudent combinations. • VGG is better than AlexNet when the teacher and the student are trained on the whole training set. However, VGG student is worse than AlexNet student when it is trained on the few-shot dataset. Namely, Student-Alone of VGG11 has 53.34% accuracy while Student-Alone of AlexNet-Half has 59.02% accuracy. • Our method DivBFKD remarkably improves accuracy over Standard-KD by 14–22% on all teacher-student combinations. • Distillation is often most effective when the teacher’s and student’s architecture belong to the same family. Overall, the robust performance across multiple data scales and architectures suggests that our method successfully tackles the black-box few-shot KD setting.
12
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
4.4
Diversity of our synthetic images
4.4.1
Comparison with the teacher’s training images.
As we mentioned earlier, our goal is to generate synthetic images close to the teacher’s training images. To prove this argument, we embed and visualize the teacher’s training images (they are unknown and not used in the KD process) and our synthetic images of CIFAR10 in Fig. 4. It shows that our synthetic images belong to the distribution of the teacher’s training images. With the teacher’s supervision, our synthetic images show their strong resemblance to the teacher’s training images. For quantitative results, we compute the Coverage metric [54] to measure the fraction of teacher’s training samples whose neighborhoods contain at least one synthetic sample. Our synthetic images achieve 0.42, compared to 0.18 of the few-shot images and 0.29 of the synthetic images generated from the standard WGAN. This shows that our synthetic images resemble many variability of the teacher’s training images, thus improving image diversity.
Figure 4: Embeddings of teacher’s training images, few-shot images, and our synthetic images on CIFAR10.
4.4.2
Quality and quantitative measurements
We randomly visualize 24 real images and 24 synthetic images generated by our WGAN in Fig. 5. Primarily, our goal is not to generate visually aesthetic images but to generate diverse images that the teacher confidently predicts their labels. As a side note, image quality is also not a strict requirement in KD methods relied on synthetic images [13, 16, 35]. However, our synthetic images in Fig. 5(b) are able to show convincing properties of their real counterparts. Inception score (IS) and Fréchet inception distance (FID). To quantify the diversity and quality of synthetic images, we compute the IS [55] (higher is better) and FID [56] 13
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
Real images images Real
Teacher-confidence images images Teacher-confidence
automobile automobile 0.9992 0.9992
deer deer 0.9994 0.9994
dog dog 0.9993 0.9993
airplane airplane 0.9998 0.9998
cat cat 0.9986 0.9986
truck truck 0.9988 0.9988
deer deer 0.9981 0.9981
automobile automobile 0.9864 0.9864
airplane airplane 0.9994 0.9994
horse horse 0.9992 0.9992
bird bird 0.9988 0.9988
frog frog 0.9983 0.9983
dog dog 0.9993 0.9993
ship ship 0.9992 0.9992
cat cat 0.9975 0.9975
frog frog 0.9983 0.9983
bird bird 0.9992 0.9992
airplane airplane 0.9997 0.9997
horse horse 0.9991 0.9991
ship ship 0.9990 0.9990
frog frog 0.9988 0.9988
dog dog 0.9990 0.9990
truck truck 0.9986 0.9986
deer deer 0.9995 0.9995
airplane airplane 0.9998 0.9998
horse horse 0.9993 0.9993
dog dog 0.9931 0.9931
cat cat 0.9976 0.9976
ship ship 0.9988 0.9988
deer deer 0.9983 0.9983
airplane airplane 0.9989 0.9989
bird bird 0.9991 0.9991
ship ship 0.9985 0.9985
frog frog 0.9990 0.9990
cat cat 0.9987 0.9987
dog dog 0.9974 0.9974
horse horse 0.9984 0.9984
truck truck 0.9988 0.9988
ship ship 0.9993 0.9993
bird bird 0.9990 0.9990
automobile automobile 0.9989 0.9989
frog frog 0.9992 0.9992
cat cat 0.9987 0.9987
truck truck 0.9990 0.9990
automobile automobile 0.9990 0.9990
airplane airplane 0.9988 0.9988
bird bird 0.9985 0.9985
ship ship 0.9980 0.9980
(a) Real images
(b) Synthetic images
Figure 5: Visualization of real and synthetic images along with their predictive labels and confidence-scores provided by the teacher on CIFAR10. (lower is better) of our synthetic images in Table 4. We compare with the standard WGAN and FS-BBT (the second-best baseline), where our method achieves the best scores. These results confirm that our method generate synthetic images with high diversity and quality, and explain their effectiveness for the distillation step. Table 4: IS and FID of synthetic images on CIFAR10. Method
IS (↑)
FID (↓)
CVAE (from [17]) Standard WGAN WGAN with AT (Ours)
3.51 3.99 4.42
19.91 14.79 13.96
[AT] adaptive thresholds.
4.5
Ablation studies
4.5.1
Impacts of different components
As described in §3, the core components of our method include WGAN and adaptive thresholds in the generation phase, and class balancing in the distillation phase. Adding both latter components to WGAN completes our method. Table 5 breaks down the contribution of each component, where both adaptive thresholds (AT) and class balancing (CB) improve our accuracy when applied either independently or jointly.
14
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
Table 5: Student accuracy (%) with incremental components of DivBFKD. Components
CIFAR10
Tiny-ImageNet
WGAN [21] WGAN with AT WGAN with CB DivBFKD
75.59±0.69 76.35±0.21 76.88±0.45 76.97±0.56
42.70±0.25 43.25±0.71 43.75±0.40 44.25±0.38
[AT] adaptive thresholds; [CB] class balancing.
4.5.2
Impacts of the numbers of real and synthetic images
100
80
80
75
Accuracy (%)
Accuracy (%)
We investigate our performance with varying amounts of data on CIFAR10. Fig. 6(a) shows that both Standard-KD and our DivBFKD improve student accuracy with more real images, as expected. At N = 5 K, our DivBFKD achieves 79.93%, while Standard-KD only achieves 72.66%. Conversely, at N = 250, Standard-KD drops abruptly to 33.33%, while our method still has an acceptable 66.76%. This trend proves that our method can still distill robustly in extremely limited data cases. Besides, we also expect that our performance should improve with more synthetic images. This agrees with Fig. 6(b) where our DivBFKD clearly outperforms Standard-KD and improves with larger M .
60 40
DivBFKD Standard-KD Mean ± 1 STD
20 0
250
1000
2000
3000
No. real images N (a)
4000
DivBFKD Standard-KD Mean ± 1 STD
70 65 60
5000
0
250 500 1K
2K
5K 10K 20K 40K 80K
No. synthetic images M (b)
Figure 6: Accuracy versus the budget of real (N ) and synthetic images (M ) on CIFAR10.
4.5.3
Impact of the quantile
In our method, the quantile q is principally used to compute adaptive thresholds to determine high-confidence images. Contrarily, while too small q might introduce counter-productive noise, too large q does not allow enough high-confidence images to improve diversity. We achieve a relatively stable accuracy of 76.22–76.97% with q ∈ [0.03, 0.10] (Fig. 7) and recommend this range for a good trade-off between image diversity and training stability.
4.6
Comparison with data-free KD methods
We also compare our method with several popular data-free (zero-shot) KD methods, in15
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
Accuracy (%)
77.0 76.5 76.0 75.5 75.0
0.02
0.03
0.05
0.07
Quantile q
0.1
0.15
Figure 7: Accuracy versus the quantile q on CIFAR10. cluding four white-box methods: Meta-KD [32], ZSKD [33], DAFL [13], and DFKD [57], and two black-box methods ZSDB3KD [35] and IDEAL [36]. From Table 6, our method DivBFKD can successfully take advantage of a small amount of data (2 K images) and perform much better than data-free methods on FMNIST and CIFAR10, while it is comparable on MNIST. Table 6: Classification accuracy (%) comparison with zero-shot KD methods. Dataset Meta-KD(W) ZSKD(W) DAFL(W) DFKD(W) ZSDB3KD(B) IDEAL(B) DivBFKD(B)
MNIST 92.47 98.77 98.20 99.08 96.54 96.32 98.97
FMNIST − 79.62 − − 72.31 83.92 87.65
CIFAR10 − 69.56 66.38 73.91 59.46 65.61 78.07
[(W)] white-box; [(B)] black-box; [−] not reported; [bold] best results.
5
Conclusion
While a large training set and internal access to the teacher are the prerequisites to most knowledge distillation (KD) approaches, they are rarely available due to various restrictions. To address these practical challenges, we propose DivBFKD, a novel method to bridge the gap for this black-box few-shot KD problem. We investigate the diversity of training data, a crucial but inadequately addressed factor for KD. Specifically, we propose the use of high-confidence images—ones that the teacher can classify confidently. We also define 16
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
adaptive thresholds, the teacher-supervised criteria to negate teacher’s class-wise bias during the selection of these images. Under the teacher’s supervision, we introduce high-confidence images to the WGAN training loop on-the-fly to boost the diversity of image generation process with negligible cost. Through comprehensive evaluation, we have shown that our DivBFKD could leverage diversity to achieve state-of-the-art KD performance among other few-shot KD methods on multiple image datasets and architectures at different scales. As a future work, it would be interesting to study the robustness aspects of the distilled model relative to the original model to have the assurance [58] for successful deployment in the real world.
References [1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. [2] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurIPS Workshop, 2015. [3] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. NeurIPS, 2017. [4] Zhong Meng, Jinyu Li, Yong Zhao, and Yifan Gong. Conditional teacher-student learning. In ICASSP, 2019. [5] Dang Nguyen, Sunil Gupta, Trong Nguyen, Santu Rana, Phuoc Nguyen, Truyen Tran, Ky Le, Shannon Ryan, and Svetha Venkatesh. Knowledge distillation with distribution mismatch. In ECML-PKDD. Springer, 2021. [6] Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. NeurIPS, 2018. [7] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019. [8] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020. [9] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015. [10] Romero et al. FitNets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. [11] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
17
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
[12] Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, and Naonori Ueda. Few-shot learning of neural networks from scratch by pseudo example optimization. In BMVC, 2018. [13] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In ICCV, 2019. [14] OpenAI. Introducing ChatGPT, 2022. [15] Shumin Kong, Tianyu Guo, Shan You, and Chang Xu. Learning student networks with few data. In AAAI, 2020. [16] Dongdong Wang, Yandong Li, Liqiang Wang, and Boqing Gong. Neural networks are more productive teachers than human raters: Active mixup for data-efficient knowledge distillation from a blackbox model. In CVPR, 2020. [17] Dang Nguyen, Sunil Gupta, Kien Do, and Svetha Venkatesh. Black-box few-shot knowledge distillation. In ECCV. Springer, 2022. [18] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018. [19] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. NeurIPS, 2015. [20] Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models. PAMI, 2021. [21] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML. Pmlr, 2017. [22] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, 2006. [23] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. IJCV, 2021. [24] Lei J Ba and Rich Caruana. Do deep nets really need to be deep? NeurIPS, 2014. [25] Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In AAAI, 2023. [26] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In CVPR, 2020. [27] Junjie Liu, Dongchao Wen, Hongxing Gao, Wei Tao, Tse-Wei Chen, Kinya Osa, and Masami Kato. Knowledge representing: Efficient, sparse representation of prior knowledge for knowledge distillation. In CVPR Workshop, 2019. 18
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
[28] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928, 2016. [29] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018. [30] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019. [31] Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Learning student networks via feature embedding. IEEE Transactions on Neural Networks and Learning Systems, 2020. [32] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535, 2017. [33] Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In ICML, 2019. [34] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In CVPR, 2020. [35] Zi Wang. Zero-shot knowledge distillation from a decision-based black-box model. In ICML. PMLR, 2021. [36] Jie Zhang, Chen Chen, and Lingjuan Lyu. Ideal: Query-efficient data-free learning from black-box models. In ICLR, 2023. [37] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In CVPR, 2020. [38] Sravanti Addepalli, Gaurav Kumar Nayak, Anirban Chakraborty, and Venkatesh Babu Radhakrishnan. Degan: Data-enriching gan for retrieving representative samples from a trained classifier. In AAAI, 2020. [39] Kien Do, Thai Hung Le, Dung Nguyen, Dang Nguyen, Haripriya Harikumar, Truyen Tran, Santu Rana, and Svetha Venkatesh. Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distillation. NeurIPS, 2022. [40] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. NeurIPS, 2017. [41] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NeurIPS, 2020. 19
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
[42] Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. NeurIPS, 2021. [43] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017. [44] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 2002. [45] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop, 2011. [46] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, MIT, NYU, 2009. CIFAR10 and CIFAR100 were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. [47] Yann Le, Xuan Yang, et al. Tiny imagenet visual recognition challenge. CS 231N, 2015. [48] Jeremy Howard. Imagenette: A subset of 10 easily classified classes from imagenet. https://github.com/fastai/imagenette, 2020. [49] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 2012. [50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. [51] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556, 2014. [52] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR, 2020. [53] Haoyu Ma and Tianlong Chen. Undistillable: Making a nasty teacher that cannot teach students. In ICLR, 2021. [54] Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela Van Der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In ICML. PMLR, 2022. [55] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurIPS, 2016. [56] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017. [57] Zi Wang. Data-free knowledge distillation with soft targeted transfer set synthesis. In AAAI, 2021. 20
T-N. Vo et al.
Improving Diversity in Black-box Few-shot KD
[58] Shivapratap Gopakumar, Sunil Gupta, Santu Rana, Vu Nguyen, and Svetha Venkatesh. Algorithmic assurance: An active approach to algorithmic testing using bayesian optimisation. NeurIPS, 2018. [59] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
21
T-N. Vo et al.
Supplementary Material: DivBFKD
[Supplementary Material]
Improving Diversity in Black-box Few-shot Knowledge Distillation In this supplementary, we provide the implementation details on our experiments, including description on the datasets (full and few-shot) as well as the architecture and training of our teacher, student, and WGAN networks.
A
Datasets
Table 7: Dataset details and their augmentation transforms. N (×103 ) denotes the number of images in thousands and K denotes the number of classes. Dimension (images and transforms) are in pixels (px). Dataset
N (×103 )
K
Dimension
Transforms
MNIST FMNIST
60 60
10 10
1 × 28 × 28 1 × 28 × 28
SVHN
73
10
3 × 32 × 32
CIFAR10
50
10
3 × 32 × 32
CIFAR100
50
100
3 × 32 × 32
Tiny-ImageNet
100
200
3 × 64 × 64
Imagenette
9.5
10
3 × 408 × 472(∗)
Resize to 32×32 Resize to 32×32 h Zero-padding (4 px/side) i Random crop to 32 × 32 " Zero-padding (4 px/side) # Random horizontal flip Random crop to 32 × 32 " Zero-padding (4 px/side) # Random horizontal flip Random crop to 32 × 32 " Zero-padding (8 px/side) # Random horizontal flip Random crop to 64 × 64 # " Resize to 256 × 256 Random horizontal flip Random crop to 224 × 224
[∗] Imagenette dimensions are reported as mean values
We use a total of seven image classification datasets for our experiments: MNIST [44], FMNIST [43], SVHN [45], CIFAR10, CIFAR100 [46], Tiny-ImageNet [47], and Imagenette [48]. Only MNIST and FMNIST have black-and-white images (1 color channel) while the other datasets have colored images (3 color channels). Depending on their relative difficulty, we only use basic image transformations to augment our data, including resizing, zero-padding, random horizontally flipping, random cropping. We present the datasets’ details and corre22
T-N. Vo et al.
Supplementary Material: DivBFKD
sponding transforms in Table 7. For Imagenette, we report the average value; their image dimension ranges from 27×80 to 4368×2912.
B
Construction of the few-shot dataset
Following [16, 17], to construct the few-shot dataset D, we used N = 2000 images for MNIST, FMNIST, SVHN, CIFAR10, and Imagenette, N = 5000 images for CIFAR100, and N = 10000 images for Tiny-ImageNet. They were randomly drawn from the original training set such that each class had N/K images (K is the number of classes).
C
Training setting of the Teacher
We trained the teacher network with a stochastic gradient descent (SGD) optimizer using 0.9 momentum, 5 × 10−4 weight decay, and 128 batch size. We fixed the learning rate to 10−2 on MNIST and FMNIST. On other datasets, we scheduled the learning rate to 10−1 at the beginning and decreased it by ten times at 50% and 75% progress. We trained for 100 epochs on MNIST and FMNIST, and 200 epochs on other datasets.
D
Our WGAN architecture
We employed a deep convolutional GAN as in [59], where the generator and discriminator had symmetrical architectures: • The generator accepts latent vectors of dimension 100 for MNIST, 200 for FMNIST, and 256 for other datasets. The latent inputs are then projected to 256 base feature maps of dimension 8×8 (except for Imagenette, 7×7), which then undergo batch normalization. These feature maps are passed through upscaling convolutional blocks, each consisting of a nearest neighbor (2×) upsample layer, a convolutional layer with half the channels, a batch normalization, and a leaky ReLU activation. Finally, once the feature maps have identical dimension to that of the training images, they are compressed to either grayscale or RGB with a final convolutional layer and a sigmoid activation to map the output pixel values to range [0, 1]. • The discriminator feeds forward input images through convolutional blocks, each consisting of a (0.5×) downscaling convolutional layer, a batch normalization (except for the first block), and a leaky ReLU activation. While the final block has 256 base feature maps of dimension 8×8 (except for Imagenette, 7×7), each previously adjacent block has half the channels. These base feature maps are eventually passed through a linear layer to yield a scalar score.
E
Training setting of our WGAN
We followed the implementation in [40] to train our WGAN [21]. As recommended, we enforced the Lipschitz constraint on the discriminator with gradient penalty, whose coefficient 23
T-N. Vo et al.
Supplementary Material: DivBFKD
with the Wasserstein loss was set to 10:1. The number of discriminator updates per generator updates was set to 5. However, we alternatively used the RMSProp optimizer with learning rate 5 × 10−5 and batch size 250 for both models. We used the number of epochs of 500 for MNIST, 1000 for FMNIST and SVHN, 2000 for CIFAR10 and Imagenette, and 5000 for CIFAR100 and Tiny-ImageNet.
F
Training setting of our Student
For a fair comparison, we used the same number of synthetic images for distillation as in other few-shot KD methods [12, 16, 17], which is 24 K for MNIST, 48 K for FMNIST, 40 K for SVHN, CIFAR10, CIFAR100, and Imagenette, and 50 K for Tiny-ImageNet. We trained the Student with an SGD optimizer using 0.9 momentum and 5×10−4 weight decay. We fixed the learning rate to 10−2 on MNIST and FMNIST. On other datasets, we scheduled the learning rate to 10−1 at the beginning and decrease it by ten times at 50% and 75% distillation progress. We used a batch size of 250 and the number of epochs of 100 for MNIST, 200 for FMNIST and SVHN, 400 for CIFAR10 and Imagenette, and 1000 for CIFAR100 and Tiny-ImageNet.
24