Generative artificial intelligence (GenAI) has been advancing with many notable achievements like ChatGPT and Bard. The deep generative model (DGM) is a branch of GenAI, which is preeminent in generating raster data such as image and sound due to the strong role of deep neural networks (DNNs) in inference and recognition. The built-in inference mechanism of DNN, which simulates and aims at synaptic plasticity of the human neuron network, fosters the generation ability of DGM, which produces surprising results with the support of statistical flexibility. Two popular approaches in DGM are the variational autoencoder (VAE) and generative adversarial network (GAN). Both VAE and GAN have their own strong points although they share and imply the underlying theory of statistics as well as significant complex via hidden layers of DNN when DNN becomes effective encoding/decoding functions without concrete specifications. This research unifies VAE and GAN into a consistent and consolidated model called the adversarial variational autoencoder (AVA) in which the VAE and GAN complement each other; for instance, the VAE is a good data generator by encoding data via the excellent ideology of Kullback–Leibler divergence and the GAN is a significantly important method to assess the reliability of data as to whether it is real or fake. In other words, the AVA aims to improve the accuracy of generative models; besides, the AVA extends the function of simple generative models. In methodology, this research focuses on the combination of applied mathematical concepts and skillful techniques of computer programming in order to implement and solve complicated problems as simply as possible.
Keywords
deep generative model (DGM)
variational autoencoder (VAE)
generative adversarial network (GAN)
Author information
Introduction
The variational autoencoder (VAE) and the generative adversarial network (GAN) are two popular approaches for developing a deep generative model (DGM) [1] with the support of a deep neural network (DNN). The high capacity of DNN contributes significantly to the success of GAN and VAE. Some works have combined the VAE and the GAN. Larsen et al. [2] proposed a traditional combination of VAE and GAN by considering a decoder of VAE as a generator of GAN [2, p. 1558]. They constructed the target optimization function as the sum of the likelihood function of VAE and the target function of GAN [2, p. 1560]. This research is similar to theirs [2, p. 1561] except that the construction optimization function is slightly different. The construction optimization function in this research does not include the target function of GAN according to the traditional approach of GAN. However, uncorrelated variables are removed after gradients are determined. Moreover, because the encoded data z is basically randomized, this work does not construct a new random z′ to included in the target function of GAN. This study also mentions skillful techniques of derivatives in a backpropagation algorithm.
Mescheder et al. [3] transformed the gain function of VAE including Kullback–Leibler divergence into the gain function of GAN via a so-called real-valued discrimination network [3, p. 2394] related to the Nash equilibrium equation and the sigmoid function. Then they trained the transformed VAE by the stochastic gradient descent (SGD) method. They estimated three parameters [3, p. 2395] as in this research, but their method focused on mathematical transformation while this work focuses on skillful techniques in implementation. In other words, Mescheder et al. [3] tried to fuse VAE into GAN whereas this work combines them in a mutual and balancing manner; but both studies try to unify VAE and GAN. Rosca et al. [4, p. 4] used a density ratio trick to convert the Kullback–Leibler divergence of VAE into the mathematical form log(x∕(1 − x)), which is similar to the GAN target function log(x) + log(1 − x). Actually, they carried out a fusion of VAE and GAN like Mescheder et al. did. The essence of their methods is based on the convergence of the Nash equilibrium equation.
Ahmad et al. [5] combined VAE and GAN separately as did previous experimental research. First, they trained VAE and swapped the encoder–decoder network to a decoder–encoder network so that the output of VAE is transformed into some useful information, which in turn becomes the input of GAN instead of random information that is the usual input [5, p. 6]. Miolane et al. [6] combined VAE and GAN by summing the target functions of VAE and GAN weighted with regular hyperparameters [6, p. 974]. Later, they first trained VAE and then sent the output of VAE to the input of GAN [6, p. 975]. Ding et al. [7] proposed an interesting research that applies VAE and GAN to credit card fraud detection. The main point of their research is that because small fraud data is not enough to train well-supervised learning models like classification and discriminant analysis in a better way, VAE is applied to generate pseudo training data so that GAN will be trained well based on such sufficiently large training data in order to obtain a better discrimination function for detecting credit card fraud. In their VAEGAN model [7, p. 83,682], online credit data (original data) is fed into the VAE encoder to train the VAE decoder as a generator. Then the generator is used to generate fake data so that such fake data and the real data are integrated into sufficiently large data, which is used to train a GAN discriminator. As a result, such a trained discriminator is applied to detect credit card fraud.
In general, both VAE and GAN have their own strong points. For instance, they not only take advantage of solid statistical theory as well as DNN but they also suffer from drawbacks. For example, VAE does not have a mechanism to distinguish fake data from real data and GAN does not handle explicitly probabilistic distribution of encoded data. It is better to utilize their strong points and alleviate their weak points. Therefore, this research focuses on incorporating GAN into VAE by skillful techniques related to both SGD and software engineering architecture, which are neither based on purely mathematical fusion nor on experimental tasks. In practice, many complex mathematical problems can be solved effectively by some skillful techniques of computer programming. Moreover, the proposed model called adversarial variational autoencoder (AVA) aims to extend functions of VAE and GAN as a general architecture for the generative model. For instance, AVA will provide an encoding function that GAN does not possess and a discrimination function that VAE needs to distinguish fake data from real data. The combination of VAE and GAN into AVA is strengthened by a regular and balance mechanism, which obviously is natural and like the fusion mechanism. In some cases, it is better than the fusion mechanism because both built-in VAE and GAN inside AVA can retain their own strong features. Therefore, the experiment in this work is not very significant regarding large data when only AVA, VAE, and GAN are compared within a small dataset, which aims to prove the proposed method mentioned in the next section.
Methodology
This research proposes a method as well as a generative model that incorporates GAN into VAE for extending and improving the DGM because GAN does not deal with the coding of original data and VAE lacks mechanisms to assess the quality of generated data. Note that data coding is necessary for some essential applications such as image compression and recognition whereas auditing quality can improve the accuracy of generated data. As convention, let vector variables x = (x1, x2, … , xm)T and z = (z1, z2, … , zn)T be the original data and encoded data whose dimensions are m and n (m > n), respectively. A generative model is represented by a function f(x|𝛩) = z, f(x|𝛩) ≈ z, or f(x|𝛩) → z, where f(x|𝛩) is implemented by a DNN whose weights are 𝛩, which converts the original data x to the encoded data z and is called an encoder in VAE. A decoder in VAE that converts expectedly the encoded data z back to the original data x is represented by a function g(z|𝛷) = x′, where g(z|𝛷) is also implemented by a DNN whose weights are 𝛷 with the expectation that the decoded data x′ is approximated to the original data x as x′ ≈ x. The essence of VAE developed by Kingma and Welling [8] is to minimize the following loss function for estimating the encoded parameter 𝛩 and the decoded parameter 𝛷:
such that
Note that ∥x − x′∥ is the Euclidean distance between x and x′ whereas KL(𝜇(x), 𝛴(x)|N (0, I)), is the Kullback–Leibler divergence between the Gaussian distribution of x whose mean vector and covariance matrix are 𝜇(x) and 𝛴(x), respectively. The standard Gaussian distribution N (0, I) has the mean vector and covariance matrix 0 and identity matrix I, respectively.
The GAN developed by Goodfellow et al. [9] does not act on the encoder f(x|𝛩) = z but it focuses on optimizing the decoder g(z|𝛷) = x′ by introducing a so-called discriminator, which is a discrimination function d(x|𝛹):
x → [0, 1] from the considered data x or x′ to range [0,1] in which d(x|𝛹) can distinguish fake data from real data. In other words, the larger the result the discriminator d(x′|𝛹) derives, the more real the generated data x′. Obviously, d(x|𝛹) is implemented by a DNN whose weights are 𝛹 noting that this DNN has only one output neuron denoted by d0. The essence of GAN is to optimize mutually the following target function for estimating the decoder parameter 𝛷 and the discriminator parameter 𝛹 [9, p. 3]:
such that 𝛷 and 𝛹 are optimized mutually as follows:
The proposed generative model in this research is called adversarial variational autoencoder because it combines VAE and GAN by the fusing mechanism in which the loss function and the balance function are optimized parallelly. The AVA loss function implies loss information in encoder f(x|𝛩), decoder g(z|𝛷), and discriminator d(x|𝛹) as follows:
The balance function of AVA is to supervise the decoding mechanism, which is the GAN target function as follows:
The key point of AVA is that the discriminator function occurs in both the loss function and the balance function via the expression log(1 − d(g(z|𝛷)|𝛹)), which means that the capacity of how to distinguish fake data from real data by the discriminator function affects the decoder DNN. As a result, the three parameters 𝛩, 𝛷, and 𝛹 are optimized mutually according to both the loss function and the balance function as follows:
Because the encoder parameter 𝛩 is independent of both the decoder parameter 𝛷 and the discriminator parameter 𝛹, its estimate is specified as follows:
Because the decoder parameter 𝛷 is independent of the encoder parameter 𝛩, its estimate is specified as follows:
Note that the Euclidean distance ∥x − x′∥ is only dependent on 𝛷. Because the discriminator tries to increase the credible degree of real data and decrease the credible degree of fake data, its parameter 𝛹 has the following estimate:
By applying the SGD algorithm to the backpropagation algorithm, these estimates are determined based on gradients of the loss function and the balance function as follows:
where 𝛾 (0 < 𝛾 ≤ 1) is the learning rate. Let af(. ), ag(. ), and ad(. ) be activation functions of encoder DNN, decoder DNN, and discriminator DNN, respectively, and so, let , , and be derivatives of these activation functions, respectively. The encoder gradient regarding 𝛩 is ([8, p. 5], [10, p. 9], [11, p. 43])
The decoder gradient regarding 𝛷 is
where
The discriminator gradient regarding 𝛹 is
As a result, the SGD algorithm incorporated into the backpropagation algorithm for solving AVA is totally determined as follows:
where notation [i] denotes the ith element in the vector. Note the derivatives , , and because they are helpful techniques to consolidate AVA. The reason for two different occurrences of derivatives (d(x′|𝛹∗)) and (x′) in the decoder gradient regarding 𝛷 is nontrivial because the unique output neuron of the discriminator DNN is considered the effect of the output layer of all output neurons in the decoder DNN.
Figure 1.
Causality–effect relationship between decoder DNN and discriminator DNN.
When weights are assumed to be 1, the error of the causal decoder neuron is the error of the discriminator neuron multiplied by the derivative at the decoder neuron. Moreover, the error of the discriminator neuron, in turn, is the product of its minus bias − d′(. ) and its derivative , where d′(. ) is the derivative of the discriminator, shown in Figure 1.
It is necessary to describe AVA architecture because skillful techniques cannot be applied to AVA without clear and solid architecture. The key point to incorporate GAN into VAE is that the error of generated data is included in both the decoder and the discriminator besides the decoded data x′, which is the output of the decoder DNN and which becomes the input of the discriminator DNN:
The AVA architecture follows an important aspect of VAE where the encoder f(x|𝛩) does not produce directly decoded data z as f(x|𝛩) = z. It actually produces the mean vector 𝜇(x) and the covariance matrix 𝛴(x) belonging to x instead. In this research, 𝜇(x) and 𝛴(x) are flattened into an array of neurons’ output layer of the encoder f(x|𝛩):
The actual decoded data z is calculated randomly from 𝜇(x) and 𝛴(x) along with a random vector r:
where r follows the standard Gaussian distribution with mean vector 0 and identity covariance matrix I, and each element of (𝛴(x))1∕2 is the square root of the corresponding element of 𝛴(x). This is an excellent finding in the traditional literature that made the calculation of Kullback–Leibler divergence much easier without loss of information.
The balance function bAVA(𝛷, 𝛹) aims to balance the decoding task and the discrimination task without partiality, but it can lean forward the decoding task for improving the accuracy of the decoder by including the error of the original data x and the decoded data x′ into the balance function as follows:
As a result, the estimate of the discriminator parameter 𝛹 is
where d0 = d(x′|𝛹) as usual. In a reverse causality–effect relationship, the unique output neuron of discriminator DNN is the cause of all output neurons of decoder DNN as shown in Figure 3.
Figure 3.
Reverse causality–effect relationship between discriminator DNN and decoder DNN.
Suppose the bias of each decoder output neuron is bias[i] and the error of the discriminator output neuron, error[i], is the sum of weighted biases, which is in turn multiplied with the derivative at the discriminator output neuron noting that every weighted bias is also multiplied with the derivative at every decoder output neuron. Suppose all weights are 1; we have
Because the balance function bAVA(𝛷, 𝛹) aims to improve the decoder g(z|𝛷), it is possible to improve the encoder f(x|𝛩) by a similar technique noting that the output of the encoder is the mean vector 𝜇(x) and the covariance matrix 𝛴(x). This research proposes another balance function BAVA(𝛩, 𝛬) to assess the reliability of the mean vector 𝜇(x) because 𝜇(x) is very important to randomize z and 𝜇(x) is linear. Let D(𝜇(x)|𝛬) be the discrimination function for encoder DNN from 𝜇(x) to range [0,1] in which D(𝜇(x)|𝛬) can distinguish fake mean 𝜇(x′) from real mean 𝜇(x). Obviously, D(𝜇(x)|𝛬) is implemented by a so-called encoding discriminator DNN whose weights are 𝛬 noting that this DNN has only one output neuron denoted by D0. The balance function BAVA(𝛩, 𝛬) is specified as follows:
Note that
The AVA loss function is modified with regard to the balance function BAVA(𝛩, 𝛬) as follows:
By following a similar way of applying the SGD algorithm, it is easy to estimate the encoding discriminator parameter 𝛬 as follows:
where aD(. ) and are activation functions of the discriminator D(𝜇(x)|𝛬) and its derivative, respectively.
The encoder parameter 𝛩 consists of two separate parts 𝛩𝜇 and 𝛩𝛴 because the output of encoder f(x|𝛩) consists of mean vector 𝜇(x) and covariance matrix 𝛴(x):
where
When the balance function BAVA(𝛩, 𝛬) is included in the AVA loss function, the part 𝛩𝜇 is recalculated whereas the part 𝛩𝛴 is kept intact as follows:
Figure 4 shows the AVA architecture with the support of the assessing encoder.
Figure 4.
AVA architecture with support of assessing encoder.
Similarly, the balance function BAVA(𝛷, 𝛬) can lean forward the encoding task for improving the accuracy of encoder f(x|𝛩) by considering the error of original mean 𝜇(x) and decoded data mean 𝜇(x′) as follows:
Without repeating explanations, the estimate of discriminator parameter 𝛬 is modified as follows:
where D0 = D(x′|𝛬) as usual. These variants of AVA are summarized, and their tests are described in the next section. Moreover, the ideology of fusing VAE and GAN like AVA does is not new when reviewing the research by Larsen et al. [2] in which their unification mechanism is like AVA. The contribution of this research is to propose a solid architecture of a generative model based on two powerful models VAE and GAN, which aims at flexibility with plentiful functions including encoder, decoder, and leaning mechanism that allows developers to customize AVA according to their individual purposes. The generative AI application supporting AVA is available at https://github.com/ngphloc/ai/tree/main/3_implementation, which requires Java 15.
Experimental results and discussion
In this experiment, AVA is tested with VAE and GAN; but there are five versions of AVA such as AVA1, AVA2, AVA3, AVA4, and AVA5. Recall that AVA1 is the normal version of AVA whose parameters are listed as follows:
AVA2 leans forward, improving the accuracy of decoder DNN by modifying discriminator parameter 𝛹 as follows:
AVA3 supports the balance function BAVA(𝛩, 𝛬) for assessing the reliability of encoder f(x|𝛩). Its parameters are listed as follows:
AVA4 is a variant of AVA3 along with the leaning forward mechanism, improving the accuracy of encoder f(x|𝛩) like AVA2. Its parameters are listed as follows:
The last version AVA5 supports all functions such as decoder supervising, leaning decoder, encoder supervising, and leaning encoder:
The experiment is performed on a laptop with CPU AMD64 4 sub-processor core, 4 GB RAM, Windows 10, and Java 15. The given dataset is a set of thirty-six 100 × 64 images available at https://github.com/ngphloc/ai/tree/main/3_implementation/datasets/orbit/base-100x64. The 36 images are animated images that imitate the movements of a dragon and a tiger in a bamboo jungle. Each image depicts the position of a dragon or a tiger; note that the background, which is the bamboo jungle, is not changed. For example, the following two images (Figure 5) depict two positions of a dragon and a tiger among 36 positions. For each tested image, DGMs are not retrained for fair testing because there is no splitting of the training set and the testing set.
Figure 5.
Images for DGM training and testing.
It is necessary to define how efficient DGMs such as VAE, GAN, and AVA are. Let imageGen be the best image generated by a DGM, which is compared with the ith image denoted by images[i] in the dataset. Then let dij be the pixel distance between imageGen and the ith image at the jth pixel as follows:
Obviously, image[i][j] (imageGen[j]) is the jth pixel of the ith image (the generated image). The notation ∥⋅∥ denotes the norm of a pixel. For example, the norm of the RGB pixel is , where r, g, and b are red, green, and blue colors of such pixels. Suppose all pixel values are normalized in the interval [0,1]. The quantity dij implies the difference between two images, and so it expresses the similarity quality of the generated image, which is as small as possible. The inverse 1 − dij expresses the diversity quality of the generated image, which is as large as possible. Therefore, the best image should balance the quantities dij and 1 − dij so that the product dij(1 − dij) becomes as larger as possible:
Because the product dij(1 − dij) is a second-order function, its maximizer exists, and so the generated image whose product dij(1 − dij) is larger is the better image when its balance is more stable. As a result, let the balance metric (BM) be the metric to assess the quality of the generated image (the best image) with regard to the ith image, which is formulated as follows:
where ni is the number of pixels of the ith image. The larger the BMi, the better the generated image and the better the balance of similarity and diversity. The overall BM of a DGM is the average BM[i] over N = 36 test images as follows:
where
Recall that the larger the BM, the better the DGM. However, regarding the similarity quality, the DGM will be better when its BM is smaller because a small BM implies good similarity in this test; note that such a small BM implies small distance or small diversity. Therefore, the DGM whose BM is largest or smallest is preeminent. The DGM whose BM is the largest is the best in balance of similarity and diversity. The DGM whose BM is the smallest is the best in similarity. Both the maximum and minimum of BM, which indicate both balance quality and similarity quality, respectively, are considered in this test but the balance quality with a large BM is more important.
The four AVA variants (AVAs) as well as VAE and GAN are evaluated by BM with 19 learning rates (𝛾 = 1, 0.9, … , 0.1, 0.09, … , 0.01) because the SGD algorithm is affected by the learning rate and the accuracy of AVA varies slightly within a learning rate because of randomizing encoded data z in the VAE algorithm. Table 1 shows the BM values of AVAs, VAE, and GAN with 10 learning rates: 𝛾 = 1, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1.
AVA1
AVA2
AVA3
AVA4
AVA5
VAE
GAN
𝛾 = 1.0
0.2298
0.2301
0.0642
0.0766
0.2301
0.0583
0.2298
𝛾 = 0.9
0.2307
0.2294
0.0546
0.0594
0.2293
0.0681
0.2283
𝛾 = 0.8
0.2309
0.2316
0.0596
0.0546
0.2301
0.0587
0.2311
𝛾 = 0.7
0.2316
0.2305
0.0629
0.0631
0.2305
0.0665
0.2311
𝛾 = 0.6
0.2309
0.2317
0.0555
0.0657
0.2318
0.0623
0.2315
𝛾 = 0.5
0.2318
0.2319
0.0591
0.0598
0.2313
0.0610
0.2311
𝛾 = 0.4
0.2322
0.2329
0.0629
0.0732
0.2322
0.0568
0.2312
𝛾 = 0.3
0.2318
0.2321
0.0741
0.0655
0.2326
0.0651
0.2325
𝛾 = 0.2
0.2300
0.2312
0.0740
0.0929
0.2302
0.0735
0.2315
𝛾 = 0.1
0.2103
0.2105
0.1230
0.1217
0.2114
0.1238
0.2107
Table 1
BM regarding learning rates from 1 down to 0.1.
Table 2 shows the BM values of AVAs, VAE, and GAN with nine learning rates: 𝛾 = 0.09, 0.08, 0.07, 0.06, 0.05, 0.04, 0.03, 0.02, 0.01.
AVA1
AVA2
AVA3
AVA4
AVA5
VAE
GAN
𝛾 = 0.09
0.2038
0.2015
0.1319
0.1328
0.2026
0.1338
0.2031
𝛾 = 0.08
0.1924
0.1938
0.1417
0.1446
0.1978
0.1435
0.1916
𝛾 = 0.07
0.1842
0.1826
0.1566
0.1574
0.1834
0.1555
0.1818
𝛾 = 0.06
0.1685
0.1772
0.1662
0.1659
0.1785
0.1676
0.1699
𝛾 = 0.05
0.1664
0.1617
0.1792
0.1785
0.1621
0.1805
0.1628
𝛾 = 0.04
0.1675
0.1655
0.1918
0.1906
0.1662
0.1924
0.1665
𝛾 = 0.03
0.1845
0.1832
0.2017
0.2014
0.1855
0.2021
0.1857
𝛾 = 0.02
0.2047
0.2032
0.2098
0.2098
0.2028
0.2099
0.2046
𝛾 = 0.01
0.2147
0.2146
0.2147
0.2147
0.2146
0.2147
0.2148
Table 2
BM regarding learning rates from 0.09 down to 0.01.
Table 3 shows BM means, BM maxima, BM minima, and BM standard deviations (SDs) of AVAs, VAE, and GAN.
AVA1
AVA2
AVA3
AVA4
AVA5
VAE
GAN
Mean
0.2093
0.2092
0.1202
0.1225
0.2096
0.1207
0.2089
Maximum
0.2322
0.2329
0.2147
0.2147
0.2326
0.2147
0.2325
Minimum
0.1664
0.1617
0.0546
0.0546
0.1621
0.0568
0.1628
SD
0.0249
0.0251
0.0606
0.0586
0.0244
0.0606
0.0252
Table 3
Evaluation of AVAs, VAE, and GAN.
Note that VAE and GAN represent a pole of similarity quality and a pole of balance quality, respectively. From the experimental results shown in Table 3, AVA5 is the best DGM because it gains the highest BM mean (0.2096), which is also larger than the BM mean (0.2089) of the pole GAN. It is easy to explain this result because AVA5 is the one that improves both the decoding task and the encoding task when it embeds both the decoder discriminator and the encoder discriminator as well as both the leaning decoder and the leaning encoder. Moreover, both AVA1 and AVA2 are better than GAN because their BM means (0.2093, 0.2092) are larger than the BM mean (0.2089) of GAN. If the similarity quality is considered, AVA3 is the best DGM because it gains the lowest BM mean (0.1202), which is also smaller than the BM mean (0.1207) of the pole VAE. It is easy to explain this result because AVA3 is the one that improves the encoding task when it embeds the encoder discriminator. Moreover, AVA1, which is a fair AVA because it embeds the decoder discriminator but does not support the leaning decoder, is better than the pole GAN whereas AVA3, which is a fair AVA because it embeds the encoder discriminator but does not support the leaning encoder, is better than the pole VAE. This result is important because the best AVA5 is not a fair one because it supports both the leaning decoder and the leaning encoder. Therefore, about the BM mean, which is the most important metric, all AVA variants are better than traditional DGMs such as VAE and GAN with regard to both similarity quality and balance quality.
Although the BM mean is the most important metric, it is necessary to check other metrics related to extreme values that are BM maximum and BM minimum, where BM maximum implies the best balance quality and BM minimum implies the best similarity quality. Note from experimental results shown in Table 3 that the decoder improvement with AVA1 and AVA2 aims to improve balance quality with high BM and the encoder improvement with AVA3 and AVA4 aims to improve similarity quality with low BM whereas AVA5 improves both the decoder and the encoder. AVA2 and AVA5 are better DGMs about the extreme balance quality because their BM maxima (0.2329, 0.2326) are larger than the BM maximum (0.2325) of GAN. Similarly, AVA3 and AVA4 are better DGMs about the extreme similarity quality because their BM minima (0.0546, 0.0546) are smaller than the BM minimum (0.0568) of VAE. Therefore, about BM extreme values, AVA variants are better than traditional DGMs such as VAE and GAN with regard to both similarity quality and balance quality.
Because the two poles VAE and GAN are stabler than AVAs in theory as each AVA includes functions from VAE and GAN so that each AVA is more complicated than VAE and GAN, it is necessary to check the SD of BM, which reflects the stability of DGMs. The smaller the SD, the stabler the DGM. AVA1 and AVA2 are stabler than GAN when their SDs (0.0249, 0.0251) are smaller than the SD (0.0252) of GAN. AVA3 and AVA4 are slightly stabler than VAE when their SDs (0.0606, 0.0586) are smaller than or equal to the SD (0.0606) of VAE. Moreover, AVA5 is the best one about the stability quality when its SD (0.0244) is the smallest. Therefore, AVA variants are stabler than traditional DGMs such as VAE and GAN.
Figure 6 depicts BM means, BM maxima, BM minima, and BM standard deviations of AVAs, VAE, and GAN by charts.
Figure 6.
Evaluation of AVAs, VAE, and GAN.
It is concluded that the combination of GAN and VAE, which produces AVA in this research, results in better encoding and decoding performance of the DGM when metrics such as BM means, BM maxima, BM minima, and BM standard deviations of AVAs are better with regard to contexts of balance quality and similarity quality. Moreover, AVA5, which is full of functions including the decoder discriminator, decoder leaning, encoder discrimination, and encoder leaning, produces the best results with the highest balance quality given the largest BM mean (0.2096) and the highest stability given the smallest SD (0.0244).
Conclusions
It is certain that AVA is better than the traditional VAE and GAN due to the support of Kullback–Leibler divergence that establishes the encoder as well as the built-in discriminator function of GAN that assesses the reliability of data. It is possible to think that VAE and GAN are solid models in both theory and practice when their mathematical foundation cannot be changed or transformed. However, it is still possible to improve them by modifications or combinations as well as applying them to specific tools where their strong points are brought into play. In applications related to raster data like images, VAE has a drawback of consuming much memory because probabilistic distribution represents the entire image whereas some other DGMs focus on representing the product of many conditional probabilistic distributions for pixels. Although this approach for modeling pixels by the recurrent neural network may consume less memory, it is significantly useful to fill in or recover smaller damaged areas in a bigger image. In the future, we will try to apply the pixel approach to AVA; for instance, AVA processes a big image block by block and then every block is modeled by a conditional probability distribution with a recurrent neural network as well as a long short-term memory network.
Conflict of interest
The authors declare no conflict of interest.
References
1.
RuthottoL, HaberE. An Introduction to Deep Generative Modeling [Internet]. arXiv; 2021. Available from: https://arxiv.org/10.48550/arXiv.2103.05180.
2.
LarsenAB, SønderbySK, LarochelleH, WintherO. Autoencoding beyond pixels using a learned similarity metric [Internet]. In: International Conference on Machine Learning, vol. 48, New York: JMLR; 2016. p. 1558–1566. Available from: http://proceedings.mlr.press/v48/larsen16.pdf.
3.
MeschederL, NowozinS, GeigerA. Adversarial variational Bayes: unifying variational autoencoders and generative adversarial networks [Internet]. In: Proceedings of the 34th International Conference on Machine, vol. 70, Sydney: PMLR; 2017. p. 2391–2400. Available from: http://proceedings.mlr.press/v70/mescheder17a/mescheder17a.pdf.
4.
RoscaM, LakshminarayananB, Warde-FarleyD, MohamedS. Variational Approaches for Auto-encoding Generative Adversarial Networks [Internet]. arXiv; 2017. Available from: https://arxiv.org/abs/1706.04987.
5.
AhmadB, SunJ, YouQ, PaladeV, MaoZ. Brain tumor classification using a combination of variational autoencoders and generative adversarial networks. Biomedicines. 2022;10(2):1–19. doi:10.3390/biomedicines10020223.
6.
MiolaneN, PoitevinF, LiY-T. Estimation of orientation and camera parameters from cryo-electron microscopy images with variational autoencoders and generative adversarial [Internet]. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New Orleans: IEEE; 2020. p. 970–971. Available from: http://openaccess.thecvf.com/content_CVPRW_2020/papers/w57/Miolane_Estimation_of_Orientation_and_Camera_Parameters_From_Cryo-Electron_Microscopy_Images_CVPRW_2020_paper.pdf.
7.
DingY, KangW, FengJ, PengB, YangA. Credit card fraud detection based on improved variational autoencoder generative adversarial network. In: AbbottD, editor. IEEE Access, vol. 11, 2023. p. 83680–83691. doi:10.1109/ACCESS.2023.3302339.
