The author implemented a language model entirely in NumPy, manually deriving all backward passes without using autograd and limiting the vocabulary to 24 characters. The model consisted of a few weight matrices and relied on custom gradient calculations. Three of its features passed every test but were functionally inert.
Read original
medium