Embeddings

Lab: Word2Vec’s Static Embedding Problem Word2Vec and GloVe produce one vector per word. That is their strength — compact, reusable — and their fundamental weakness: a word with multiple meanings is forced into a single point in space. This notebook uses tiny synthetic vectors to show the problem before we move to contextual models. import numpy as np def cosine(a, b): return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))) # Hand-designed 4-dim vectors. 'bank' is forced to sit between two meanings. embeddings = { "river": np.array([1.0, 0.0, 0.0, 0.0]), "stream": np.array([0.9, 0.1, 0.0, 0.0]), "money": np.array([0.0, 1.0, 0.0, 0.0]), "cash": np.array([0.1, 0.9, 0.0, 0.0]), "bank": np.array([0.5, 0.5, 0.0, 0.0]), # one static compromise "cat": np.array([0.0, 0.0, 1.0, 0.1]), "dog": np.array([0.0, 0.0, 0.9, 0.2]), } 1. Polysemy: one vector cannot be in two places at once In a real corpus, bank appears near river and near money. Word2Vec averages the pressure and lands somewhere in between. ...

Attention: Turning Token Vectors into Context Vectors The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time. But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first. ...

Embeddings: From One-Hot Vectors to Learned Representations Everything in the previous notes assumed the network’s input was already a list of numbers — x1 = 1, x2 = 2, pixel intensities, whatever. But most interesting data is not numeric. “cat”, “dog”, “bank”, user IDs, product IDs, words of a sentence. Neural networks cannot multiply the string "cat" by a weight matrix. Somewhere between the raw symbol and the first linear layer, a translation to numbers must happen, and the way we do it — the embedding layer — turns out to be one of the most consequential ideas in modern deep learning. ...