CNN based Metrics for Performance Evaluation of Generative Adversarial Networks

Abstract

In this work, we propose two Convolutional Neural Network (CNN) based metrics, Classification Score (CS) and Distribution Score (DS), for performance evaluation of Generative Adversarial Networks (GANs). Though GAN-generated images can be evaluated through manual assessment of visual fidelity, it is prolonged, subjective, challenging, tiresome, and can be misleading. Existing quantitative methods are biased towards memory GAN and fail to detect over-fitting. CS and DS allow us to experimentally prove that training of GANs is actually guided by the data set, that it improves with every epoch and gets closer to following the distribution of the data set. Both methods are based on GAN-generated image classification by CNN. CS is the root mean square (RMS) value of three different classification techniques, Direct Classification (DC), Indirect Classification (IC), and Blind Classification (BC). It exhibits the degree to which GAN can learn the features and generate fake images similar to real data sets. DS shows the contrast between the mean distribution of GAN-generated data and the real data. It indicates the extent to which GANs can create synthetic images with similar distribution to real data sets. We evaluated CS and DS metrics for different variants of GANs and compared their performances with existing metrics. Results show that CS and DS can evaluate the different variants of GANs quantitatively and qualitatively while detecting over-fitting and mode collapse.