Dry (@DryLdn) header
Dry (@DryLdn) profile picture
Martinique
Joined October 2012
Photo by Dry (@DryLdn)Photo by Dry (@DryLdn)Photo by Dry (@DryLdn)
Using an hologram fan to visualize 360° ancient relics. teapawt.org/i/status/1780689172631…
510
12,720
942
125,168
12,579,758
Remember Devin? Apparently demo's fake. Paul is sad. 😥
129
345
225
4,089
894,406
A few new CUDA hacker friends joined the effort and now llm.c is only 2X slower than PyTorch (fp32, forward pass) compared to 4 days ago, when it was at 4.2X slower 📈 The biggest improvements were: - turn on TF32 (NVIDIA TensorFLoat-32) instead of FP32 for matmuls. This is a new mathmode in GPUs starting with Ampere+. This is a very nice, ~free optimization that sacrifices a little bit of precision for a large increase in performance, by running the matmuls on tensor cores, while chopping off the mantissa to only 10 bits (the least significant 19 bits of the float get lost). So the inputs, outputs and internal accumulates remain in fp32, but the multiplies are lower precision. Equivalent to PyTorch `torch.set_float32_matmul_precision('high')` - call cuBLASLt API instead of cuBLAS for the sGEMM (fp32 matrix multiply), as this allows you to also fuse the bias into the matmul and deletes the need for a separate add_bias kernel, which caused a silly round trip to global memory for one addition. - a more efficient attention kernel that uses 1) cooperative_groups reductions that look much cleaner and I only just learned about (they are not covered by the CUDA PMP book...), 2) the online softmax algorithm used in flash attention, 3) fused attention scaling factor multiply, 4) "built in" autoregressive mask bounds. (big thanks to ademeure, ngc92, lancerts on GitHub for writing / helping with these kernels!) Finally, ChatGPT created this amazing chart to illustrate our progress. 4 days ago we were 4.6X slower, today we are 2X slower. So we are going to beat PyTorch imminently 😂 Now (personally) going to focus on the backward pass, so we have the full training loop in CUDA.
97
347
28
4,146
1,419,504
Seems like a busy game of hungry hungry hippo happened this weekend
4
3
1
51
6,164
Human evolution process
54
195
23
687
102,311
Le 8 Mai 1902, la Ville de Saint-Pierre, ancienne capitale de la Martinique, disparaissait suite à une éruption Volcanique 🌋🙏🏾
237
6
542
29,289
#Aérien : @CorsairFr s’est associé au chef guadeloupéen Jimmy Bibrac pour ses repas en business au départ des #Antilles 🍽️🛫Une façon de mettre en avant "la #gastronomie créole" et le "terroir antillais" ➡️https://teapawt.org/t.co/g84RIy06Hq
1
45
10
94
11,624
JUST IN: #Bitcoin whitepaper is hidden on every Apple MacBook computer running recent versions of macOS software.
578
3,025
640
12,948
2,913,978
Old joke about agnostic technologists building artificial super intelligence to find out if there’s a God. They finally finish & ask the question. AI replies: “There is now, mfs!!”
14,379
18,596
2,776
196,097
57,966,656
There are two rules in life: 1) Never give out all the information.
336
3,700
219
25,115
3,604,188
L'exercice international de préparation aux tsunamis #CaribeWave2023 se déroule aujourd'hui dans toute la Caraïbe à partir de 10h, avec comme scénario, l'effondrement d'un pan de la Montagne Pelée causant une vague destructrice. Mais savez-vous quoi faire en cas de tsunami ?
22
18
2,278
🇫🇷 FLASH - "Si les Français étaient vraiment en colère, je n'aurais pas été réélu il y a un an", affirme Emmanuel #Macron. (France 2) #Macron13h #reformedesretraites
2,252
3,287
6,428
26,405
5,280,443
Are you?
996
1,786
462
21,222
1,575,656
24,173
42,902
5,158
381,325
53,620,988
“I used to be in crypto, but now I got interested in AI"
28,315
19,064
9,004
238,385
51,013,276
There was no VAR check for that penalty call. Wow
1,210
2,344
320
21,116
1,778,189
Never seen them give one man so many penalties in one competition, this shit is rigged
2,366
7,183
751
55,073
4,011,454
mistX v2 has launched! See below why mistX.io could become your go-to DEX in 2022. THREAD 👇
7
48
10
148
Les Antilles, l'Afrique et la musique viennent de perdre l'un de ses plus grand Ambassadeur. Jacob grâce à ton art, tu as rapproché les Antilles à l'Afrique. Dakar où tu as vécu te pleure. Adieu l'ami.
138
1,608
156
6,208
Covid-19 : la situation sanitaire se dégrade à nouveau https://teapawt.org/t.co/WX3a5cl4vl
4
21
9
10