The maths you need to start understanding LLMs

Summary

Archives Categories Blogroll This article is the second of three "state of play" posts that explain how Large Language Models work, aimed at readers with the level of understanding I had in mid-2022: techies with no deep AI knowledge. It grows out of part 19 in my series working through Sebastian Raschka's book "Build a Large Language Model (from Scratch)". You can read the first post in the series here. Actually coming up with ideas like GPT-based LLMs and doing serious AI research requires serious maths. But the good news is that if you just want to understand how they work, while it does require some maths, if you studied it at high-school at any time since the 1960s, you did all of the groundwork then: vectors, matrices, and so on. One thing to note -- what I'm covering here is what you need to know to understand inference -- that is, using an existing AI, rather than the training process used to create them. That's also not much beyond high-school maths, but I'll be writing about that later on. So, with that caveat, let's dig in! Vectors and high-dimensional spaces In the last post I used the word "vector" in the way it's normally used by software engineers -- pretty much as a synonym of "an array of numbers". But a vector of length n is more than that; it's a distance and direction in n-dimensional space, or (equivalently) it can be taken as a point -- you start at the origin, and then follow the vector from there to the point in question. In 2-d space, the vector (2,−3) means "two units to the right, and three down", or the point that is located if you move that way from the origin. In 3-d, (5,1,−7) means "five right, one up, and seven away from the viewer" (or in some schools of thought, seven toward the viewer), or the point there. With more dimensions, it becomes pretty much impossible to visualise, but conceptually it's the same. We use vectors to mean things in LLMs. For example, the vectors of logits that come out of the LLM (see the last post) represen...