A learned lookup table in a neural network that maps discrete tokens (like words or subwords) to continuous vector representations called embeddings.
Definition:
An embedding matrix is a 2D matrix of shape
[vocabulary_size times embedding_dimension]
where:
- Each row corresponds to a unique token in the vocabulary.
- Each row vector is the embedding for that token.
In LLMs:
- The embedding matrix is used to convert input tokens (e.g., token ID 27) into a dense vector (e.g., a 768-dimensional float array).
- These vectors capture semantic and syntactic properties of the tokens.
An embedding matrix stores the vector representations of all tokens in the vocabulary and is essential for transforming discrete language into numerical form that LLMs can process.
