Autoencoders and variational autoencoders: compressed representations
Autoencoders are self-supervised neural networks that compress input data into a compact code and then reconstruct the original input. Variational autoencoders learn a continuous probabilistic latent space, enabling generation of new samples. This article covers architecture, training, and common variants such as denoising autoencoders.
What is an autoencoder?
An autoencoder is a neural network architecture trained to compress input data into its essential features and then reconstruct the original input from that compressed representation. It uses unsupervised learning to discover latent variables that hold the most information about the data distribution. The original input serves as ground truth, so the model is self-supervised, distinct from encoder-decoder models that produce different outputs.
Common applications of autoencoders include feature extraction, data compression, image denoising, anomaly detection, and facial recognition, as described in the IBM overview of autoencoders. Certain variants, such as variational autoencoders, extend the architecture for generative tasks.
How autoencoders work
An autoencoder consists of an encoder network, a bottleneck layer (also called the code), and a decoder network. The encoder compresses the input through dimensionality reduction, progressively reducing the number of nodes. The bottleneck holds the most compressed representation, and the decoder expands it back to the original input size. The Deep Learning textbook by Goodfellow, Bengio, and Courville devotes a chapter to autoencoders and describes this architecture.
During training, the network adjusts its weights to minimize the reconstruction error, which is the difference between the decoded output and the original input. This process typically uses gradient-based optimization algorithms. The design of the bottleneck dimension is crucial: a bottleneck smaller than the input forces the model to learn a compressed representation, while a larger one may allow the model to copy input directly.
Autoencoders are trained without labeled data, making them useful for representation learning. After training, the decoder can sometimes be discarded, leaving the encoder to produce feature embeddings for downstream tasks.
Variational autoencoders
Variational autoencoders (VAEs) extend the standard autoencoder by encoding a continuous, probabilistic representation of the latent space. IBM's page on variational autoencoders notes that this enables the model not only to reconstruct input but also to generate new data samples that resemble the training data.
In the paper 'Auto-Encoding Variational Bayes' by Diederik P. Kingma and Max Welling, the authors introduce a stochastic variational inference and learning algorithm that scales to large datasets, and they show that a reparameterization of the variational lower bound yields an estimator that can be optimized using standard stochastic gradient methods, as described in the arXiv abstract.
VAEs are applied to image generation, anomaly detection, and generating novel drug molecules. They are part of the broader class of generative models.
Denoising autoencoders and other variants
Denoising autoencoders train the network to reconstruct a clean input from a corrupted version. This approach was shown to yield significantly lower classification error when stacked to form deep networks, as reported in the Journal of Machine Learning Research paper by Vincent et al.
Other variants include sparse autoencoders, which impose sparsity constraints on activations, and contractive autoencoders, which penalize the Jacobian of the encoder. Undercomplete autoencoders, with a bottleneck smaller than the input, are the basic form.
Stacked denoising autoencoders learn hierarchical representations, and the higher-level features have been shown to boost the performance of subsequent classifiers such as support vector machines.
| Feature | Standard autoencoder | Variational autoencoder |
|---|---|---|
| Latent space | Discrete, fixed | Continuous, probabilistic |
| Training objective | Minimize reconstruction error | Maximize evidence lower bound (ELBO) |
| Output | Reconstruction of input | Reconstruction or new samples |
| Common use | Denoising, anomaly detection | Image generation, drug discovery |
Frequently asked questions
What is the difference between an autoencoder and a variational autoencoder?
A standard autoencoder learns a discrete, fixed latent representation for each input, while a variational autoencoder learns a continuous probability distribution over the latent space, allowing sampling to generate new data.
How is an autoencoder trained?
An autoencoder is trained by minimizing the reconstruction error between the decoded output and the original input. This is done through backpropagation and gradient descent, without needing labeled data.
What is latent space?
Latent space is the set of latent variables that summarize the most essential information of the input data. It is the compressed representation produced by the encoder.
What is the reconstruction error?
Reconstruction error is the difference between the autoencoder's output and the original input. The training objective is to minimize this error.
What are denoising autoencoders?
Denoising autoencoders are trained to reconstruct a clean input from a corrupted version, making them able to recover clean data from noisy input and useful for learning invariant features.