Infrastructure to run ML & GPU algorithms in cloud-native sandboxes
Joined April 2025
- Tweets362
- Following1
- Followers11.3K
- Likes763
Gradient checking is the most underrated debugging skill in deep learning.
1. Use a tiny input, a few parameters and float64. Not your full model.
2. Compute the gradient with your backward pass.
3. Compute it numerically too
4. Compare them with relative error, not the plain difference.
5. Check one parameter at a time to find which layer is wrong.
Build Micrograd from Scratch has 25 exercises, including gradient checks.
tensortonic.com/study-plans/…
Implement FlashAttention from scratch on TensorTonic in PyTorch and Triton.
This project makes you write each piece yourself: safe softmax, online softmax over chunks, merging two partial results, the tiled forward pass, skipping causal tiles, and counting reads and writes.
link - tensortonic.com/projects
ML engineer interview in 2016:
- Explain linear regression
- What's overfitting
- Train a model in sklearn
- Get the offer
ML engineer interview in 2026:
Implement attention from scratch.
Write backprop without PyTorch.
Code softmax that doesn’t overflow.
Explain KV cache.
Implement LayerNorm.
Debug a tensor shape mismatch live.
Build a mini transformer in 45 mins.
Every one of these is a problem you can practice on TensorTonic tensortonic.com
yesterday we released RL projects on TensorTonic.
you build the RL environment, write the agent, and implement Q-learning yourself until the agent learns to reach the goal.
tensortonic.com
We just released a new Multimodal AI project. Train your own tiny CLIP from scratch-
> build the scaled similarity matrix
> write the symmetric contrastive loss
> measure retrieval with recall at K
> classify zero-shot with prompts
> add hard negatives to fix the shortcut
> write the pairwise sigmoid loss
tensortonic.com
8 computer vision concepts worth understanding by implementing them yourself on TensorTonic:
> 2D Convolution - slide a kernel across an image, and work out the output size from padding and stride.
> Max Pooling - shrink a feature map by keeping only the strongest activation in each window.
> Batch Normalization - normalize activations across the batch, then rescale them with learned gamma and beta.
> Intersection over Union - score how well two boxes overlap by dividing the shared area by the combined area.
> Non-Max Suppression - remove duplicate detections by keeping the top-scoring box and dropping the ones that overlap it too much.
> Focal Loss - a cross-entropy variant that down-weights easy examples so the hard, rare ones drive the gradient.
> Dice Loss - train segmentation models on mask overlap, so a tiny object isn’t drowned out by a huge background.
> Receptive Field - compute how much of the input image each output unit can actually see through stacked layers.
link in comments.
almost 400K people have watched Stanford's CS336 on building language models from scratch.
but if you want to know whether you can actually build one, implement each piece yourself on TensorTonic with provided gpus to fine tune your model
tensortonic.com
Every LLM is 8 layers of ideas stacked on top of each other which you can implement on tensortonic:
Layer 1: Gradients
Write a tiny autograd engine. Value nodes, expression graphs, a neuron.
loss.backward() stops being a black box.
Layer 2: Tokens
Train BPE yourself and apply the merges by hand.
Layer 3: The block
RMSNorm, SwiGLU, RoPE, causal grouped-query attention.
Layer 4: Training at scale
FLOP estimates, mixed-precision memory, gradient accumulation.
Layer 5: Attention variants
MHA → MQA → GQA → MLA, then FlashAttention.
Layer 6: Kernels
Softmax, GELU and reductions in CUDA, then again in Triton.
Layer 7: Fine-tuning
QLoRA on Qwen3-4B with real medical Q&A.
Layer 8: Serving
Build your own Qwen3 inference server.
All 8 are graded, runnable problems on TensorTonic.
link in comments.