This is my attempt at making an LLM with just python and numPy. However, that's really hard, and so I've started with a small character-level name generator. Go through all of the tabs until you get to the chatbot at the end.
A few notes:
Sometimes the model may output something random, like nothing at all or just a letter. So keep pressing the button until a sensical name pops up.
Overall, the finished product is rather unimpressive (unless you have a similar taste in names to Elon); I highly suggest you check out the Github Repository, which is a little more interesting.
I got a lot of inspiration (and the names dataset) from Andrej Karpathy's Nueral Networks series. It was really helpful and fun to watch so you should check it out.
badGPT v2
This version still generates names except it's a wavenet model. This means that the context is gradually combined over several layers, as apposed to all at once like the previous model.
A few notes:
It gets 8 characters of context instead of 3, folded together two at a time (8 → 4 → 2 → 1), so every layer only ever compares neighbours. It has 45,597 parameters.
This model performs markedly worse than the last one, unfortunately, despite the fancier architecture.
badGPT v3
This is an improvement on the name generator, but it still does essentially the same thing, make more of stuff (except instead of names, it's shakespear!). The architecture is very different from the bigram and wavenet models, as it includes self attention.
A few notes:
Is shakespear really dead? Yes, clearly. But pay attention to how it gets the general structure right, and the words have mostly complete syllables, they just haven't come together quite right.
This model includes 56,769 parameters, 4 transformer blocks of 4 attention heads each, and 64 characters of context. For scale, GPT-3 had about three million times as many parameters.
This model was trained on 1.1 million characters of shakespear.txt, which took roughly 20 hours on my CPU. It finished at 1.55 train / 1.76 validation loss, down from 4.14 at the start.
In terms of progress towards building a chatGPT-like thing, this is about half of it. It completes documents, just not question-answer documents. To get to something that could answer questions, I would have to train it on a dataset of question-answer examples.
badGPT v4
This has the same transformer as the shakespear model, except it reads tokens instead of characters. I used byte pair encodings, so common words like " the" take up one token instead of 4. Instead of training on shakespear, it's trained on web articles, like what real LLMs are trained off of, so it sounds a little less archaic.
A few notes:
This model has 119,169 parameters (GPT-2, which this model is based off of, has 1.5 billion. Current (September 2026) models, like Claude's Fable, have trillions). It has 4 transformer blocks of 4 attention heads each, and 64 tokens of context. A token averages a bit over two characters here, so the same 64 tokens of context means it can include up to 150 characters of context instead of 64.
Trained on a slice of FineWeb-Edu, so it writes like an explainer article, or would if it was large enough to write coherent sentences.
This model, like GPT-2, wasn't trained on any specific question answer documents, like later models have been. So it's not really a chatbot, in the modern sense, but it's as close as I can get.
It was trained with an explicit end-of-text token, so unlike the other three which stop at a predetermined max token count, it can decide it's finished and stop on its own.