Llama from scratch (2023)

Summary

Llama from scratch (or how to implement a paper without crying) 09 Aug, 2023 Llama from scratchI want to provide some tips from my experience implementing a paper. I'm going to cover my tips so far from implementing a dramatically scaled-down version of Llama for training TinyShakespeare. This post is heavily inspired by Karpathy's Makemore series, which I highly recommend. I'm only going to loosely follow the layout of their paper; while the formatting and order of sections makes sense for publication, we're going to be implementing the paper. I'll also be skipping over some of the more obvious steps, like setting up a virtual environment and installing dependencies. A preview of what we're going to end up with: print(generate(llama, MASTER_CONFIG, 500)[0]) ZELBETH: Sey solmenter! 'tis tonguerered if berryishdd, and What his stabe, you, and, but all I pilJefals, mode with, Vurint as steolated have loven OlD the queen'd refore Are been, good plmp: Proforne, wift'es swleen, was no bunderes'd a a quain beath! Tybell is my gateer stalk smen'd as be matious dazest brink thou lord Enves were cIUll, afe and whwas seath This a is, an tale hoice his his onety Meall-tearn not murkawn, fase bettizen'd her, To belacquesterer? baxewed wupl usweggs yet tall An TakeawaysAlways work iteratively: start small, stay certain, and build up.My approach for implementing papers is: Make all of the helper functions required to test your model quantitatively (data splits, training, plotting the loss). Before you even look at the paper, pick a small, simple, and fast model that you've done in the past. Then make a helper function to evaluate the model qualitatively. Start by picking apart different components of the paper, and then implementing them one-by-one, training and evaluating as you go. Make sure your layers do what you think. Use .shape religiously. assert and plt.imshow are also your friends. Work out the results without matrix multiplication first, and then use the torch function...