Topic
Fine-Tuning & Adaptation
Adapting pre-trained models to your task without burning a cluster — LoRA and its variants, quantized training, and the tradeoffs that decide quality.
The Llama 3 Herd of Models: What the Paper Actually Says
Llama 3's 405B benchmark numbers are fine. The paper is actually about something more useful: what decisions you make when you can train on 15T tokens across 16K H100s, and which of those decisions transfer to your deployment.
Switch Transformers: What the Sparse MoE Scaling Paper Actually Says
Every modern large model — Mixtral, DeepSeek, Gemini — routes tokens through sparse experts. The design decisions in all of them trace back to one 2021 Google Brain paper. The paper is worth reading because the failure modes live in the routing logic, not the math.
T5: What the Text-to-Text Paper Actually Says
Every instruction-tuned model today owes something to T5's core idea: every NLP task is just sequence-to-sequence. But the paper's real contribution is a systematic ablation of what actually helps in transfer learning — and several of the answers are counterintuitive.
DeepSeek-V3: What the Frontier-on-a-Budget Paper Actually Says
DeepSeek-V3 trained a 671B-parameter frontier model for ~$5.5M. The paper is less about model quality and more about whether the training stack itself is the bottleneck — and how to engineer around it.
Toolformer: What the Paper Actually Says
A 6.7B model beats GPT-3 175B on math by learning to use a calculator. Toolformer's self-supervised training pipeline is the interesting part — and it's more constrained than the demos imply.
Ring Attention: What the Near-Infinite Context Paper Actually Says
Extending context beyond what fits on one GPU isn't just a memory problem — it's a communication design problem. Ring Attention sequences the K/V data through a ring of devices and hides the transfers behind computation. Here's what that actually costs in production.
MegaScale: What ByteDance's 12,288-GPU Training Paper Actually Says
At 12,288 GPUs, you see roughly one hardware failure per day. Standard training frameworks have no answer for this. MegaScale is ByteDance's account of what actually breaks when you scale LLM training to 10,000+ GPUs — and the algorithm-system co-design changes that kept MFU above 55%.
LoRA: What the Low-Rank Adaptation Paper Actually Says
Full fine-tuning GPT-3 requires roughly 1.4 TB of optimizer state. LoRA gets trainable parameters down to ~4.7M with comparable quality — by exploiting a property of pre-trained models that most engineers know about but don't fully reason from.
S-LoRA: What the Paper Actually Says About Serving Thousands of LoRA Adapters
LoRA makes fine-tuning cheap. Serving thousands of LoRA adapters from a single GPU cluster is a different problem entirely — one that requires rethinking memory management, batching, and tensor parallelism from scratch.
Llama 2: What the Open-Source RLHF Paper Actually Says
By turn 15, your agent is ignoring its system prompt. Llama 2 documented the fix — Ghost Attention — but also the full iterative RLHF pipeline with rejection sampling that outperforms PPO alone. Here's what the paper actually says.
BitNet b1.58: What the 1-bit LLM Paper Actually Says
A 70B BitNet model fits in 7GB instead of 140GB — and the math says output quality matches FP16 at scale. The catch: you can't convert existing models. Here's what the paper actually proves, and why the hardware story matters more than the math.
QLoRA: What the Paper Actually Says About Fine-Tuning on Consumer Hardware
Full fine-tuning a 65B model requires ~780 GB of GPU memory. QLoRA gets it to 48 GB — a single A100 — without meaningfully degrading quality. The three mechanisms that make this possible are more interesting than the headline number.