Topic
Paper Breakdowns
The “What the Paper Actually Says” series — close readings of the systems and ML papers that matter, focused on the mechanism, the numbers, and when not to use the idea.
Cassandra: What the Paper Actually Says
We had a Cassandra cluster where DELETE operations made reads progressively slower until queries timed out. Adding more disk space made it worse. The root cause is described precisely in the 2009 paper — but only if you understand that Cassandra cannot actually delete data.
EAGLE: Speculative Decoding with Feature-Level Prediction — What the Paper Actually Says
Standard speculative decoding tops out at 2–2.4x speedup because token-level draft prediction is hard. EAGLE sidesteps this by predicting at the feature level instead — the second-to-top-layer hidden states — which are far more predictable than discrete tokens. The result is 3–4x with a draft head that's 0.3% the size of the target model.
The Llama 3 Herd of Models: What the Paper Actually Says
Llama 3's 405B benchmark numbers are fine. The paper is actually about something more useful: what decisions you make when you can train on 15T tokens across 16K H100s, and which of those decisions transfer to your deployment.
LLM.int8(): What the 8-bit Matrix Multiplication Paper Actually Says
INT8 quantization works perfectly for vision models. For LLMs above ~7B parameters it silently destroys quality — unless you understand why outlier features emerge at scale and how mixed-precision decomposition works around them.
Mooncake: What the KV-Cache-Centric Disaggregated Serving Paper Actually Says
DistServe disaggregates prefill from decode. SGLang caches KV cache per-instance. Mooncake goes further: it treats the KV cache as a shared, distributed resource that the scheduler routes around — turning a fleet of 50 isolated caches into one coherent pool.
Pregel: What the Large-Scale Graph Processing Paper Actually Says
PageRank in MapReduce is O(iterations × full dataset reloads). Pregel fixes this by keeping the graph in memory across iterations and replacing disk I/O with message passing. The 'think like a vertex' model is the insight — BSP is the implementation.
Switch Transformers: What the Sparse MoE Scaling Paper Actually Says
Every modern large model — Mixtral, DeepSeek, Gemini — routes tokens through sparse experts. The design decisions in all of them trace back to one 2021 Google Brain paper. The paper is worth reading because the failure modes live in the routing logic, not the math.
Titans: What the Test-Time Memorization Paper Actually Says
Attention is quadratic. SSMs compress everything into a fixed state. Titans takes a third path: a neural memory module whose weights are the memory, updated via gradient descent at inference time. Here's the mechanism, what the benchmark numbers actually say, and why you shouldn't deploy this without thinking carefully about inference cost.
T5: What the Text-to-Text Paper Actually Says
Every instruction-tuned model today owes something to T5's core idea: every NLP task is just sequence-to-sequence. But the paper's real contribution is a systematic ablation of what actually helps in transfer learning — and several of the answers are counterintuitive.
SGLang and RadixAttention: What the Paper Actually Says
Multi-call LLM programs recompute the same KV cache thousands of times per day. RadixAttention fixes this by maintaining a global radix tree of cached KV blocks — turning redundant prefill into a lookup. Here's the mechanism, the numbers, and where it falls apart.
SARATHI: What the Chunked-Prefill Paper Actually Says
Continuous batching fixed GPU utilization but created a new problem: long prefills stall every in-flight decode for seconds. SARATHI splits prefill into chunks and interleaves them with decode, eliminating the stall without disaggregating your cluster. Here's the mechanism, the math, and where it breaks down.
Mixture of Depths: What the Paper Actually Says
Every transformer layer processes every token, even when 90% of that work does nothing useful. Mixture of Depths lets the model skip layers for tokens that don't need them — and the compute budget stays completely predictable.
MapReduce: What the Google Paper Actually Says
The 2004 Google paper that gave us Hadoop — and everything that replaced it — is worth reading not for the map/reduce abstraction itself, but for the fault tolerance model and the straggler insight. The failure modes are still the failure modes.
Mamba: What the Selective State Space Paper Actually Says
Transformers scale quadratically with sequence length and carry a growing KV cache that never shrinks. Mamba proposes a different trade: compress context into a fixed-size state, process tokens recurrently at inference, and do it faster than attention at long sequences. Here's the mechanism, what it actually buys you, and where it quietly fails.
Kafka: What the Original Paper Actually Says
The original Kafka paper from 2011 had no replication. A broker failure made all unconsumed messages permanently unavailable. The paper treats this as a limitation to fix later, not a deal-breaker. Understanding why explains more about Kafka's design philosophy than any architecture diagram.
H2O: Heavy-Hitter Oracle for KV Cache Eviction — What the Paper Actually Says
StreamingLLM keeps the first tokens and a sliding window — a positional bet. H2O makes a different bet: keep the tokens that actually received attention, not the ones that happened to appear early. The paper shows you can cut KV cache by 5× with a greedy per-step eviction policy that costs almost nothing to implement.
DeepSeek-V3: What the Frontier-on-a-Budget Paper Actually Says
DeepSeek-V3 trained a 671B-parameter frontier model for ~$5.5M. The paper is less about model quality and more about whether the training stack itself is the bottleneck — and how to engineer around it.
Dapper: What Google's Distributed Tracing Paper Actually Says
Every distributed tracing tool you use — Jaeger, Zipkin, OpenTelemetry — descends from one design decision Google made in 2010: sample at the trace root, not per-span. The paper explains why, and the failure modes it didn't fully solve.
Toolformer: What the Paper Actually Says
A 6.7B model beats GPT-3 175B on math by learning to use a calculator. Toolformer's self-supervised training pipeline is the interesting part — and it's more constrained than the demos imply.
MegaScale: What ByteDance's 12,288-GPU Training Paper Actually Says
At 12,288 GPUs, you see roughly one hardware failure per day. Standard training frameworks have no answer for this. MegaScale is ByteDance's account of what actually breaks when you scale LLM training to 10,000+ GPUs — and the algorithm-system co-design changes that kept MFU above 55%.
LoRA: What the Low-Rank Adaptation Paper Actually Says
Full fine-tuning GPT-3 requires roughly 1.4 TB of optimizer state. LoRA gets trainable parameters down to ~4.7M with comparable quality — by exploiting a property of pre-trained models that most engineers know about but don't fully reason from.
HNSW: What the Vector Similarity Search Paper Actually Says
Every vector database runs HNSW under the hood. Most engineers tune M and ef_construction without understanding what they're actually trading off. The paper rewards a close read.
GFS: What the Google File System Paper Actually Says
GFS isn't just 'distributed POSIX.' The paper is an exercise in deliberately violating POSIX to make large-scale sequential workloads tractable on unreliable hardware. The consistency model it chose — intentionally weaker than what you'd expect — defines everything downstream, including HDFS.
Generative Agents: What the Paper Actually Says
The Generative Agents paper isn't just a simulation demo. It's a formal architecture for agent memory: stream, retrieval, reflection, and planning. Every agent framework you're using borrowed from it.
FrugalGPT: What the Paper Actually Says
GPT-4-level quality at 4× lower cost isn't wishful thinking — FrugalGPT shows how LLM cascades route queries to cheap models when they're sufficient and expensive models when they're not. Here's the algorithm, the tradeoffs, and where it breaks.
DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving
Every GPU running LLM serving is doing two fundamentally different jobs simultaneously — and that interference is silently destroying your SLOs. DistServe separates prefill and decode onto different machines. Here's the mechanism, the cost, and when it's actually worth it.
Bigtable: What the Distributed Storage Paper Actually Says
Bigtable's data model is three lines: a sorted map indexed by (row, column, timestamp). Everything that makes it powerful — and everything that makes it dangerous — follows from those three lines.
Zanzibar: What Google's Authorization Paper Actually Says
Google serves 10M authorization checks per second with 3ms p95 latency. The paper behind it — Zanzibar — introduces a consistency mechanism called zookies that nobody talks about, and a data model that's quietly become the industry standard for permission systems.
StreamingLLM: What the Attention Sink Paper Actually Says
Sliding your KV cache window shouldn't crash model quality — but it does. The reason is attention sinks: models learn to dump excess softmax mass on the first few tokens. StreamingLLM keeps those sinks and recovers stable infinite-length generation.
Spanner: What Google's Globally-Distributed Database Paper Actually Says
Spanner achieves external consistency across a globally-distributed database using atomic clocks and GPS receivers. The insight isn't exotic hardware — it's that bounded clock uncertainty lets you make a specific, provable guarantee about transaction ordering that 'eventually consistent' databases can't offer.
Raft: What the Understandable Consensus Algorithm Paper Actually Says
Raft isn't just Paxos made simpler. It's a specific set of design decisions — each one a tradeoff — that power etcd, CockroachDB, and TiKV. The paper is worth reading because the failure modes live in the decisions, not the math.
Multi-Token Prediction: What the Meta FAIR Paper Actually Says
Speculative decoding solves LLM generation latency, but requires a separate draft model — which you have to deploy, tune, and keep in sync. Multi-Token Prediction solves the same problem at training time by adding prediction heads to the main model. Here's the mechanism and what it costs.
MapReduce: What the Paper Actually Says
MapReduce isn't just 'map then reduce.' The paper is a fault-tolerance paper disguised as a programming model. The actual insight is about how to make commodity hardware reliable enough to process terabytes — and the tradeoffs that follow from that choice.
Llama 2: What the Open-Source RLHF Paper Actually Says
By turn 15, your agent is ignoring its system prompt. Llama 2 documented the fix — Ghost Attention — but also the full iterative RLHF pipeline with rejection sampling that outperforms PPO alone. Here's what the paper actually says.
BitNet b1.58: What the 1-bit LLM Paper Actually Says
A 70B BitNet model fits in 7GB instead of 140GB — and the math says output quality matches FP16 at scale. The catch: you can't convert existing models. Here's what the paper actually proves, and why the hardware story matters more than the math.
Tree of Thoughts: What the Paper Actually Says About LLM Search
Chain-of-thought is one path through a reasoning problem. Tree of Thoughts is a search over many paths. The NeurIPS 2023 paper shows 74% vs 4% on certain benchmarks. The production story is harder.
Self-RAG: What Adaptive Retrieval Actually Means in Production
Standard RAG retrieves on every request, whether or not the model needs it. Self-RAG teaches the model to decide — and to grade its own output. The mechanism is more interesting than the leaderboard numbers.
ReAct: What the Reasoning + Acting Paper Actually Says
Every agent framework you've used is built on ReAct's thought/action/observation loop. The paper is more specific — and more fragile — than the abstractions suggest.
QLoRA: What the Paper Actually Says About Fine-Tuning on Consumer Hardware
Full fine-tuning a 65B model requires ~780 GB of GPU memory. QLoRA gets it to 48 GB — a single A100 — without meaningfully degrading quality. The three mechanisms that make this possible are more interesting than the headline number.
MLA: What the Multi-Head Latent Attention Paper Actually Says
GQA cuts your KV cache 8x. At 128K context, you still run out of memory. Multi-Head Latent Attention — the architecture inside DeepSeek-V2, V3, and R1 — compresses KV cache via low-rank projection instead of head reduction, achieving 57x compression with near-zero quality loss. Here's what that actually means for inference.
Mistral 7B: What the Sliding Window Attention Paper Actually Says
A 7B model that outperforms Llama 2 13B everywhere and matches 34B on math. Mistral's sliding window attention and rolling KV cache aren't just architectural novelties — they're the answer to a specific memory problem that kills you at scale.