Lessons
- 01 The whole machineThe full loop in one machine: read the tape, pay attention, score every token, pick one, print it, and read again.
- 02 TokenisationHow text is cut into tokens. A press fuses the most common pair of tiles, round after round, until words become single pieces.
- 03 EmbeddingsHow a token becomes numbers. A lookup table hands every token its own list, and words that behave alike end up with lists that are alike.
- 04 Positional encodingHow a token learns where it stands. Three geared hands turn at three speeds along the sentence, and their readings are added to each token.
- 05 Self-attentionHow one position decides which earlier tokens matter: a query, a key on every token, and a set of weights that add up to one.
- 06 Multi-head attentionWhy a layer runs several attention heads at once. Three heads read one sentence with three different questions, and their answers are mixed into one list.
- 07 SamplingHow one token is drawn from the odds. Temperature, top-k and top-p all act on a single row of plungers.
In preparation
These lessons are planned and not yet built.
- The feed-forward layer
- Residual stream and layer norm
- Logits and softmax
- Training: loss and backpropagation
- Optimisers: gradient descent and Adam
- Context window and the KV cache