Embeddings: how neural networks turn words and items into vectors
Embeddings turn words and items into dense numerical vectors that capture meaning. They avoid the problems of one-hot encoding by reducing dimensionality and preserving semantic relationships. Models like word2vec and GloVe learn these vectors from large corpora.
What are embeddings?
Vector embeddings are numerical representations of data points that express different types of data, such as words or images, as arrays of numbers that machine learning models can process. This conversion is necessary because AI models operate through mathematical logic and require numerical input, as explained by IBM's vector embedding overview.
The resulting vectors are placed in a multi-dimensional space where similar data points are located near each other. This property allows algorithms to compare, transform, and combine data through vector operations, making embeddings a fundamental building block for many deep learning applications.
From one-hot encoding to dense vectors
One-hot encoding is a simple way to represent categorical data, but it creates large input vectors. For a dataset of 5,000 meal items, each vector has a length of 5,000, leading to a huge number of weights in a neural network, as described in Google's Machine Learning Crash Course.
One-hot vectors also lack meaningful relationships between items. The vectors for hot dogs and shawarmas are no more similar than those for hot dogs and salads, because the cosine similarity between any two different one-hot vectors is zero.
Dense embeddings solve these issues by providing lower-dimensional representations that capture semantic similarities. This reduces the number of weights, the amount of data needed for training, and the computational and memory requirements of the model.
Learning embeddings with word2vec and GloVe
The word2vec tool, introduced in the paper Efficient Estimation of Word Representations in Vector Space, describes two new model architectures for learning word vectors from huge corpora. The paper reports high accuracy at low computational cost, with models trained in less than a day on a 1.6 billion word data set.
Word2vec contains two models: skip-gram and continuous bag of words (CBOW). Both are self-supervised and rely on predicting words from their surrounding context in a corpus, as detailed in Dive into Deep Learning's word2vec chapter.
Another approach, GloVe, is an unsupervised learning algorithm that trains on aggregated global word-word co-occurrence statistics. Its vectors exhibit linear substructures, such as vector differences capturing analogies, according to Stanford's GloVe project page.
Using embeddings in practice
In PyTorch, the nn.Embedding module is a simple lookup table that stores embeddings of a fixed dictionary and size. It takes indices as input and returns the corresponding embedding vectors, as documented in the PyTorch Embedding reference.
Embeddings enable a variety of applications, including image recognition, natural language processing, and recommendation systems. For example, word embeddings help NLP software understand context and relationships, and graph embeddings support network analysis, according to AWS's machine learning guide.
| Attribute | One-hot encoding | Dense embedding |
|---|---|---|
| Vector length | Equals vocabulary size | Fixed small dimension |
| Semantic similarity | None; cosine similarity is 0 | Captures relationships via vector distance |
| Model weights | M x N for M vocabulary and N first-layer nodes | Fewer weights due to lower dimension |
| Memory and computation | High | Lower |
Frequently asked questions
What is an embedding?
An embedding is a numerical representation of a data point, such as a word or image, in the form of a vector. It captures the data's meaning and relationships in a way that machine learning models can process.
Why are one-hot vectors a bad choice for words?
One-hot vectors are sparse and high-dimensional, require many weights, and have no semantic similarity. The cosine similarity between any two different one-hot vectors is zero, so they cannot express relationships between words.
How does word2vec learn embeddings?
Word2vec uses either the skip-gram model, which predicts surrounding words from a center word, or the continuous bag of words model, which predicts a center word from its surrounding words. Both are self-supervised and trained on large corpora.
What is the main difference between GloVe and word2vec?
Word2vec learns from local context windows during training, while GloVe uses aggregated global word-word co-occurrence statistics from the entire corpus to learn its representations.
Can pre-trained embeddings be reused?
Yes. Many libraries provide pre-trained embeddings, such as the GloVe vectors available from Stanford. In PyTorch, the from_pretrained method can load existing tensors into an Embedding module.