◆ Overview
Thu Vu delivers a comprehensive beginner's guide to text embeddings — the numerical representations that allow computers to understand and work with language. She traces the full evolution from naive one-hot encoding through TF-IDF to modern contextual embeddings (ELMo, BERT, GPT), explaining the intuition behind each approach. The video covers the mathematical foundations (embedding spaces, cosine similarity, dimensionality reduction) and practical usage (pre-trained models, OpenAI API, the MTEB leaderboard). A perfect foundation for anyone entering the world of NLP, RAG systems, or LLM applications.
1 What Are Text Embeddings?
Text embeddings are a way of representing words or documents as numerical vectors that capture their meaning. This converts text data into a format computers can understand and process mathematically.
Embeddings broadly fall into two categories:
- Word/Token embeddings — represent a single word or token as a vector
- Document embeddings — represent an entire sentence, paragraph, or document as a single vector
Both types can be used for various NLP applications and are created using similar principles and techniques. The key insight: there are many ways to represent words as numbers, ranging from very simple to highly sophisticated.
2 Frequency-Based Methods
Thu walks through the earliest and simplest approaches to text representation:
One-Hot Encoding
- List all unique words in your vocabulary
- Create a vector as long as the vocabulary size
- Fill with zeros except for the cell corresponding to the target word (set to 1)
- Problem: creates extremely long, sparse vectors — one word = one vector of size N (vocabulary size)
Bag of Words
- To represent a sentence, count how many times each word occurs
- Squeeze the entire sentence into one vector of word counts
N-grams
- Similar to bag of words but takes groups of 2-3 words instead of individual words
- Counts occurrence of word pairs/triples in a sentence
TF-IDF (Term Frequency — Inverse Document Frequency)
- Tracks how often a word occurs in a specific document and how often it appears across all documents
- Differentiates commonly used words ("the", "like", "you") from words that carry specific meaning for a document
- Words unique to a document get higher weight; common words get lower weight
3 Shortcomings of Frequency-Based Methods
Thu identifies four critical limitations:
- No word order awareness — "Her dog bit the owner" and "The dog bit her owner" have identical bag-of-words representations despite meaning very different things
- No context awareness — the word "fine" in "I'm fine" has the exact same representation as "fine" in "I got a fine," even though they have completely different meanings
- Extremely sparse vectors — mostly zeros. A vocabulary of 100,000 words requires a 100,000-dimensional vector to represent a single word or sentence
- Cannot capture semantic meaning — "Meal was bad" and "Food is awful" would have completely different representations despite having nearly identical meaning
4 The Big Idea Behind Embeddings
Thu introduces embeddings with a philosophical quote from the Chinese philosopher Zhuangzi:
"Nets are for fish. Once you get the fish, you can forget the net. Words are for meaning. Once you get the meaning, you can forget the words."
Words are just a medium to convey meaning. The goal of embeddings is to help computers get the meaning so they can "forget the words."
Real-World Applications
- Google Search — maps queries and documents into a common embedding space, matching words with similar meaning even when the exact term doesn't appear
- Machine translation — Google Translate relies on embeddings to translate between languages
- Text analysis & clustering — find patterns in large volumes of text
- RAG chatbots — use embeddings to interpret user inputs, look up relevant information, and generate responses
The Distributional Hypothesis (1950s)
The foundational idea dates back to the 1950s: words that occur in similar distributions — whose neighboring words are similar — have similar meanings. In other words, similar words are words used in the same context.
Thu illustrates with "ong choy" (a Cantonese loanword): if you see it used with "garlic," "rice," "sautéed," and "delicious" — the same words used with "spinach," "chard," and "collard greens" — you can infer it's a leafy green vegetable. This is exactly how embedding models learn meaning from context.
5 Dense Vectors & Dimensions
The goal of embeddings is to represent words in a dense vector where similar words are closer to each other in embedding space. Breaking down the jargon:
What "Dense" Means
- Unlike one-hot encoding (mostly zeros), dense vectors have meaningful values in every dimension
- The embedding vector is shorter than the vocabulary size — typically 50 to a few thousand dimensions
- Example: OpenAI's
text-embedding-ada-002converts text into a vector of 1,536 numbers
Semantic Similarity
- "Tea" and "Coffee" — used in similar contexts (drinks, breakfast) → vectors are close together
- "Tea" and "Pasta" — different contexts → vectors are far apart
- The specific values in the vectors don't have inherent meaning on their own — the relationships between vectors are what matters
6 What Is Embedding Space?
Embedding space is the mathematical representation of where your embeddings live. Thu builds intuition through dimensionality:
- 1D embeddings — each word is a single number on a number line. Distance between points = similarity between words
- 2D embeddings — each word is two numbers plotted on a plane. You can see spatial relationships
- 3D embeddings — three numbers, visualized in 3D space
- 32D or 1,536D embeddings — the same logic applies, but you can't visualize it. The math still works: you calculate distances to measure similarity
Measuring Distance
- Cosine similarity / cosine distance — measures the angle between two vectors
- Euclidean distance — measures the straight-line distance between two points
- Existing libraries handle these calculations — you don't need to implement them yourself
Visualization with Dimensionality Reduction
To visualize high-dimensional embeddings, use algorithms like t-SNE or UMAP to reduce 32D+ data down to 2D. The result: you can see clusters of semantically related words (king/queen/sovereign/kingdom, or cat/dog/pet/bird/animal).
7 How Embeddings Are Created — The Lifecycle
Word embeddings are learned by training a machine learning or deep learning model on a large corpus (a big collection of text). The full lifecycle:
- Input text — the raw text data
- Tokenization — text is broken into smaller pieces called tokens (words, characters, numbers, or punctuation marks)
- Token IDs — each token is assigned a unique number. For a vocabulary of 16 tokens, IDs range from 0 to 15
- Initial representation — token IDs can be represented as one-hot encodings (sparse, no meaning)
- Embedding creation — the critical step: creating richer representations that capture meaning and relationships between tokens
8 Word2Vec — Skip-gram in Detail
Thu dives deep into Word2Vec, one of the most popular traditional embedding models. It has two flavors:
Continuous Bag of Words (CBOW)
- Take groups of N words from a corpus (e.g., 3 words at a time)
- Use the surrounding two words as input
- Train a neural network to predict the middle word
Skip-gram (Detailed Walkthrough)
- The exact opposite of CBOW: take the middle word and try to guess the surrounding words
- Create a shallow neural network with one hidden layer
- Hidden layer size = embedding dimension (the size of vectors you'll get)
- Output layer size = vocabulary size
- The model predicts the probability that two words are neighbors in the corpus
Self-Supervised Training
For a target word like "apricot":
- Positive examples — actual neighboring words from the corpus (garlic, jam, pie)
- Negative examples — random "noise" words from the vocabulary that have nothing to do with "apricot" (car, telephone)
- Both inputs and outputs come from the text data itself — no manual labeling needed
The Clever Trick
9 Static vs Contextual Embeddings
Word2Vec and similar shallow neural network methods create static embeddings:
- Only one single embedding per unique word
- The word "bank" gets the same vector whether it means a financial institution or a river bank
- This is a major problem — language is inherently contextual
Contextual Embeddings
Modern methods produce contextual embeddings where the same word gets different vectors in different contexts:
- ELMo — one of the first contextual embedding methods
- BERT — bidirectional transformer encoder
- GPT — generative pre-trained transformer
How Transformers Create Embeddings
- They don't just embed individual words — they process entire sequences of text
- Embeddings are refined by considering the context of every word in the sequence
- This is done through self-attention — the model weighs the importance of all words in a sentence when creating each word's embedding
10 Using Pre-trained Embedding Models
Two approaches to using embeddings in your projects:
Option 1: Train Your Own
- Libraries exist for every model discussed (Word2Vec, BERT, etc.) — just push your data through
- Drawback: needs lots of training data and compute time
- Use case: when you need embeddings tailored to a very specific domain
Option 2: Pre-trained Models (Recommended)
- Open-source — research groups release free embedding models publicly
- Commercial APIs — OpenAI and Mistral offer embedding models via paid APIs. Anthropic does not offer an embedding model
- Quality varies — different models perform differently on different task types
MTEB Leaderboard
Thu points to the Massive Text Embedding Benchmark (MTEB) leaderboard — a ranking of embedding models by performance across different task types. An essential reference when choosing which model to use for your application.
Practical Demo: OpenAI API
Thu demonstrates using OpenAI's text-embedding-ada-002:
- Embedding the word "cat" → a vector of 1,536 numbers
- Calculating distance between "Amsterdam" and "coffee shop" → relatively close (related concepts)
- Calculating distance between "Paris" and "coffee shop" → farther apart (less related)
- This demonstrates how embeddings capture real-world semantic relationships
🎯 Key Takeaways
🔑 Key Takeaways
- Embeddings convert text to meaningful numbers — dense vectors that capture semantic meaning, not just word identity
- Frequency-based methods (one-hot, bag of words, TF-IDF) are limited — they ignore word order, context, and semantics, and produce sparse vectors
- The distributional hypothesis is foundational — words that appear in similar contexts have similar meanings (1950s idea that still powers modern NLP)
- Dense vectors are more efficient than sparse — 1,536 dimensions vs. 100,000+ for one-hot encoding
- Embedding space = where vectors live — similar words are close, dissimilar words are far apart. Distance metrics (cosine similarity, Euclidean) measure relationships
- Word2Vec learns embeddings as a byproduct — train a neural network on a pretext task (predict neighboring words), then extract the hidden layer weights as embeddings
- Static embeddings give one vector per word — Word2Vec, GloVe. The word "bank" always has the same embedding regardless of context
- Contextual embeddings are the modern standard — ELMo, BERT, GPT produce different embeddings for the same word in different contexts via self-attention
- Pre-trained models are usually the right choice — unless you need domain-specific embeddings, use existing models (OpenAI, Mistral, open-source)
- Use the MTEB leaderboard — to choose the best embedding model for your specific task type
- Embeddings power real applications — search engines, translation, RAG chatbots, text clustering, and recommendation systems