What GPT-OSS leaks about OpenAI's training data

Summary

OpenAI recently released their open-weights model. Here we'll discuss how that inevitably leaks some information about their model training stack, and, on the way, show that GPT-5 was trained on phrases from adult websites. What data does OpenAI train their models on? That is a well-protected trade secret of course, one with vested interest for the answer. While GPT-oss's weights are openly available, the sources of training data are not clearly described in the model card. It is stated that the model was trained on a "text-only dataset with trillions of tokens, with a focus on STEM, coding, and general knowledge". However, as we will see, the model parameters can tell us more than that. A demonstration to start with: Let's have OpenAI's GPT-5We use version GPT-5-2025-08-07 for these experiments. Here is a link to the completion. do the simplest kind of task possible for a language model, repeating a string of Unicode text. Let's choose something random, like the Abkhaz word for "population", which is "ауааԥсыра". Upon asking Repeat after me: "ауааԥсыра", it replies something completely different, "ആളുകൾ", which apparently means people in MalayalamAccording to this dictionary. Subsequent translations here are patched together with web searches, online dictionaries and translation software.. As you might have guessed, we did not choose that string randomly at all, it is a special adversarial input belonging to a class of glitch tokens. But how did we identify such a glitch token among the 200,000 tokens that GPT-5 uses? All of OpenAI's models since GPT-4o use the o200k tokenizer. This means that we can use the GPT-oss embeddings to study the token list without having to look at each token's text content. Let's make a histogram of the L2 norm of each row of the embedding matrix. There are about 936 tokens with very low L2 norm, centered at about 2. This likely means that they did not occur in the training process of GPT-oss and were thus depressed by some form of weig...