Duncan (@duncanmudulo) header
Duncan (@duncanmudulo) profile picture

1 valid bit, 1 modify bit, 1 refrence bit and 5 permission bits. Infra Nerd · Kotlin|Swift|Rust · It works on my machine

Kampala, Uganda
Joined July 2022
Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)Photo by Duncan (@duncanmudulo)
COMEBACK = COMPLETE! ❤️‍🔥
2,427
11,391
3,322
61,601
1,712,572
Duncan retweeted
Wow. 🏆
18,799
106,526
7,421
1,365,378
15,296,160
Ukraine's President Zelensky tells CNN there is nothing personal between him and Trump despite a bitter spat that sent relations plummeting. cnn.it/4hZ1V6j
197
153
20
1,005
675,797
We're the first team to beat Arsenal 3-0 in any competition since... us, in May 2023. 😅👏
1,899
16,761
1,740
158,031
2,949,444
🌎🇦🇷 2 World Cup games, 5 goals for Lionel Messi.
2,748
11,850
688
135,385
3,041,853
🏆 BACK 2 BACK 🏆
2,445
18,068
1,292
119,307
4,051,882
Crystal Palace, Champions of Europe ❤️💙
3,357
23,389
2,325
188,429
7,559,139
BRUNO FERNANDES. RECORD-BREAKER.
4,582
75,438
5,169
307,480
11,147,028
NASSR. Your Saudi League Champion 🏆
1,767
18,417
633
98,872
1,667,613
ASTON VILLA HAVE WON THE EUROPA LEAGUE!!!!! 🏆
2,037
11,594
1,165
88,681
1,769,555
Duncan retweeted
The Arsenal. Your Premier League champions.
27,602
201,638
35,303
579,805
39,678,734
🚨🌕 OUSMANE DEMBÉLÉ WINS THE BALLON D’OR 2025! 🤩
4,964
15,441
2,355
160,152
4,764,789
FIFA CLUB WORLD CUP WINNERS!!! 🏆
7,362
62,155
9,591
227,921
13,132,994
🚨🏆 Portugal win the Nations League!
2,996
23,630
1,148
265,864
6,004,782
I support @realDonaldTrump and @elonmusk and they should make peace for the benefit of our great country. We are much stronger together than apart.
5,236
8,696
1,046
92,571
5,622,548
HISTORY MADE 📸
1,810
16,156
1,033
152,848
3,920,902
WE'VE DONE IT!!!!! 🏆
3,068
15,133
3,089
111,237
4,060,419
Jacob Kiplimo shattered the half marathon world record by 48 seconds in Barcelona 😲🏃‍♂️
102
911
122
3,735
291,774
UGANDA'S JOSHUA CHEPTEGEI SETS A NEW OLYMPIC RECORD TO WIN GOLD IN THE MEN'S 10,000M HE SNATCHED IT RIGHT AT THE DEATH 🤯
287
1,514
151
8,396
271,174
📽️ New 4 hour (lol) video lecture on YouTube: "Let’s reproduce GPT-2 (124M)" youtu.be/l8pRSuU81PU The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model: - first we build the GPT-2 network - then we optimize it to train very fast - then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers - then we bring up model evaluation, and - then cross our fingers and go to sleep. In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step. github.com/karpathy/build-na… Chapters. On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail: 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 parameter sharing wte and lm_head 01:13:47 model initialization: std 0.02, residual init 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms 01:39:38 float16, gradient scalers, bfloat16, 300ms 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms 02:00:18 flash attention, 96ms 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping 02:21:06 learning rate scheduler: warmup + cosine decay 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms 02:34:09 gradient accumulation 02:46:52 distributed data parallel (DDP) 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU) 03:23:10 validation data split, validation loss, sampling revive 03:28:23 evaluation: HellaSwag, starting the run 03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro 03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA 03:59:39 summary, phew, build-nanogpt github repo
391
2,147
414
15,214
1,529,096