Generative Models are a form of unsupervised learning models and the outcome goal is to generate new data points by understanding underlying distributions of data points in the dataset. After the deep learning revolution, Generative models are very popular and widely used in applications and research projects.

As we know, in unsupervised learning, we only have features data points, no labels are provided and our goal is to understand the structure of data. In the same way, generative models are used to generating new data points, look similar to one in the training data, using past data points.

Note that if a generative model is generating sample data which exactly similar to training data, then these models are memorizing and not useful. So newly generated data should have diverse samples beyond what we already have in training data.

Assuming Training Data X = {x1,x2,x3,x4,x5….xn} comes from underlying distributions pd(x) and a generative model samples data from learnt pm(x) distribution. So goal is to achieve minimum distance between pd(x) and pm(x).

KL-Divergence is used as a distance function to calculate the difference between two distributions.

Types of Generative Models

There are broadly two types of generative models.

Fully Visible Models

In these models, no new feature variables or latent variables are introduced. Only the available features are used as observations and generate new data. e.g For an Image input, each pixel in the image is an observation and a new image is generated using these pixels’ values.

Latent Variables Models

These models introduce new hidden variables, known as latent variables as well, which represents the overall structure of the training data. These latent variables are used to generate new data points. Further, Latent Variables are categorised in two types:

Explicit Model

In these models, we explicitly define and learn the likelihood of data by imposing a sample distribution. For e.g, impose a Gaussian Model and learn the Mean and Covariance of the distribution.

Such a model assigns a likelihood value to each image and very useful to find outliers. But the quality of newly generated images is not so well and the sampling rate is also slow. Variational Autoencoders (VAEs) is a very popular generative method in this category.

Implicit Model

We don’t impose any distributions so there is no need to learn the parameters of distribution and the goal is to learn latent variables directly from the training data and generate new data points which look similar to the training data.

There is no likelihood estimation in these models so these are extremely fast and generate high-quality data. Generative Adversarial Networks (GANs) are very popular in this category.

Applications of Generative Models

Generative modes can be very useful in following applications.

  • Image Super-Resolution – Convert a low-resolution image to high resolution
  • Image Colorization – Colorized sketches and old black and White Images.
  • Cross-Domain Image Translation – Convert an image from one domain to another domain.
  • Generate Real Faces Dataset – For 3D modeling, generating real faces which actually don’t exist.
  • Augment Small Dataset – To generate new data points from a small dataset and use these augmented data in training.
  • Virtual Try-on Applications – This can be used in a mix-reality application. e.g how a pair of sunglasses will look on a face.
  • Outlier Detection – Using likelihood estimation, an outlier can be found if the likelihood value is very low for that data point.