Embeddings Are Underrated

Summary

Machine learning (ML) has the potential to advance the state of the art in technical writing. No, I’m not talking about text generation models like Claude, Gemini, LLaMa, GPT, etc. The ML technology that might end up having the biggest impact on technical writing is embeddings. Embeddings aren't exactly new, but they have become much more widely accessible in the last couple years. What embeddings offer to technical writers is Building intuition about embeddings Here’s an overview of how you use embeddings and how they work. It’s geared towards technical writers who are learning about embeddings for the first time. Input and output Someone asks you to “make some embeddings”. What do you input? You input text.1 You don’t need to provide the same amount of text every time. E.g. sometimes your input is a single paragraph while at other times it’s a few sections, an entire document, or even multiple documents. What do you get back? If you provide a single word as the input, the output will be an array of numbers like this: [-0.02387, -0.0353, 0.0456] Now suppose your input is an entire set of documents. The output turns into this: [0.0451, -0.0154, 0.0020] One input was drastically smaller than the other, yet they both produced an array of 3 numbers. Curiouser and curiouser. (When you work with real embeddings, the arrays will have hundreds or thousands of numbers, not 3. More on that later.) Here’s the first key insight. Because we always get back the same amount of numbers no matter how big or small the input text, we now have a way to mathematically compare any two pieces of arbitrary text to each other. But what do those numbers MEAN? 1 Some embedding models are “multimodal”, meaning you can also provide images, videos, and audio as input. This post focuses on text since that’s the medium that we work with the most as technical writers. Haven’t seen a multimodal model support taste, touch, or smell yet! First, how to literally make the embeddings The big service pro...