Lab: Word2Vec's Static Embedding Problem

Mon, 21 Sep 2026 00:00:00 +0530

Lab: Word2Vec’s Static Embedding Problem

Word2Vec and GloVe produce one vector per word. That is their strength — compact, reusable — and their fundamental weakness: a word with multiple meanings is forced into a single point in space.

This notebook uses tiny synthetic vectors to show the problem before we move to contextual models.

import numpy as np

def cosine(a, b):
 return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# Hand-designed 4-dim vectors. 'bank' is forced to sit between two meanings.
embeddings = {
 "river": np.array([1.0, 0.0, 0.0, 0.0]),
 "stream": np.array([0.9, 0.1, 0.0, 0.0]),
 "money": np.array([0.0, 1.0, 0.0, 0.0]),
 "cash": np.array([0.1, 0.9, 0.0, 0.0]),
 "bank": np.array([0.5, 0.5, 0.0, 0.0]), # one static compromise
 "cat": np.array([0.0, 0.0, 1.0, 0.1]),
 "dog": np.array([0.0, 0.0, 0.9, 0.2]),
}

1. Polysemy: one vector cannot be in two places at once

In a real corpus, bank appears near river and near money. Word2Vec averages the pressure and lands somewhere in between.

Lab on Sanketh's Blog

Lab: Word2Vec's Static Embedding Problem

Lab: Word2Vec’s Static Embedding Problem

1. Polysemy: one vector cannot be in two places at once