Giles' blog

20 years of blogging

RMSNorm

Fun with low-rank vocab matrices (and a bonus test loss reduction?)

Why do OpenAI's GPT-2 weights beat mine? Part five: data quality

Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090

Putting my JAX-trained models on the Hugging Face Hub

Why do OpenAI's GPT-2 weights beat mine? Part four: digging into dropout

Adding diagrams to my static site generator with D2

Use the built-in GELU, don't roll your own!

A quick(ish) Chinchilla check

How I use AI on this blog

Why do OpenAI's GPT-2 weights beat mine? Part three: testing overtraining

Why do OpenAI's GPT-2 weights beat mine? Part two: the bugfix

Why do OpenAI's GPT-2 weights beat mine?

Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090

Building intuition about LLM parameter counts

poppy the training box, part 1: the beginnings

Writing an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX)

Writing an LLM from scratch, part 34a -- building a JAX training loop for an LLM training run

Thoughts on Role Confusion

Flax debugging: making a hash of things

10Gb/s Ethernet: switching to a Broadcom SFP+ module

JAX: commitment issues

JAX backends and devices

Using Safetensors with Flax

On first looking into JAX

10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ module

10Gb/s Ethernet: what I actually did to get it working in my home

10Gb Ethernet: what I had to (re)learn

Writing an LLM from scratch, part 33 -- what I learned from finally getting round to the appendices

Writing an LLM from scratch, part 32m -- Interventions: conclusion

Writing an LLM from scratch, part 32l -- Interventions: updated instruction fine-tuning results

How an LLM becomes more coherent as we train it

Writing an LLM from scratch, part 32k -- Interventions: training a better model locally with gradient accumulation

Writing an LLM from scratch, part 32j -- Interventions: trying to train a better model in the cloud

Writing an LLM from scratch, part 32i -- Interventions: what is in the noise?

Writing an LLM from scratch, part 32h -- Interventions: full fat float32

Automating starting Lambda Labs instances

Writing an LLM from scratch, part 32g -- Interventions: weight tying

Writing an LLM from scratch, part 32f -- Interventions: weight decay

Writing an LLM from scratch, part 32e -- Interventions: the learning rate

Writing an LLM from scratch, part 32d -- Interventions: adding attention bias

Writing an LLM from scratch, part 32c -- Interventions: removing dropout

Writing an LLM from scratch, part 32b -- Interventions: gradient clipping

Writing an LLM from scratch, part 32a -- Interventions: training a baseline model

Getting a custom PyTorch LLM onto the Hugging Face Hub (Transformers: AutoModel, pipeline, and Trainer)

Writing an LLM from scratch, part 31 -- the models are now on Hugging Face

Writing an LLM from scratch, part 30 -- digging into the LLM-as-a-judge results

Writing an LLM from scratch, part 30 -- digging into the LLM-as-a-judge results

Writing an LLM from scratch, part 29 -- using DistributedDataParallel to train a base model from scratch in the cloud

Writing an LLM from scratch, part 28 -- training a base model from scratch on an RTX 3090

Why smart instruction-following makes prompt injection easier

Writing an LLM from scratch, part 27 -- what's left, and what's next?

Writing an LLM from scratch, part 26 -- evaluating the fine-tuned model

Writing an LLM from scratch, part 25 -- instruction fine-tuning

Writing an LLM from scratch, part 24 -- the transcript hack

A classifier using Qwen3

Retro Language Models: Rebuilding Karpathy’s RNN in PyTorch

Writing an LLM from scratch, part 23 -- fine-tuning for classification

Writing an LLM from scratch, part 22 -- finally training our LLM!

Revisiting Karpathy’s 'The Unreasonable Effectiveness of Recurrent Neural Networks'

Writing an LLM from scratch, part 21 -- perplexed by perplexity

Writing an LLM from scratch, part 20 -- starting training, and cross entropy loss

How do LLMs work?

An addendum to 'the maths you need to start understanding LLMs'

The maths you need to start understanding LLMs

What AI chatbots are actually doing under the hood

Writing an LLM from scratch, part 19 -- wrapping up Chapter 4

Writing an LLM from scratch, part 18 -- residuals, shortcut connections, and the Talmud

The fixed length bottleneck and the feed forward network

Writing an LLM from scratch, part 17 -- the feed-forward network

Writing an LLM from scratch, part 16 -- layer normalisation