Tag
Production
49 posts tagged Production.
LIMA: What the Superficial Alignment Paper Actually Says
A 1,000-example SFT run on LLaMA-65B outperforms models trained with 52,000 examples and beats text-davinci-003 in human evals. LIMA's core claim: your model already knows how to be helpful — alignment is just format learning.
Mistral 7B: What the Sliding Window Attention Paper Actually Says
A 7B model that outperforms Llama 2 13B everywhere and matches 34B on math. Mistral's sliding window attention and rolling KV cache aren't just architectural novelties — they're the answer to a specific memory problem that kills you at scale.
MLA: What the Multi-Head Latent Attention Paper Actually Says
GQA cuts your KV cache 8x. At 128K context, you still run out of memory. Multi-Head Latent Attention — the architecture inside DeepSeek-V2, V3, and R1 — compresses KV cache via low-rank projection instead of head reduction, achieving 57x compression with near-zero quality loss. Here's what that actually means for inference.
QLoRA: What the Paper Actually Says About Fine-Tuning on Consumer Hardware
Full fine-tuning a 65B model requires ~780 GB of GPU memory. QLoRA gets it to 48 GB — a single A100 — without meaningfully degrading quality. The three mechanisms that make this possible are more interesting than the headline number.
ReAct: What the Reasoning + Acting Paper Actually Says
Every agent framework you've used is built on ReAct's thought/action/observation loop. The paper is more specific — and more fragile — than the abstractions suggest.
Self-RAG: What Adaptive Retrieval Actually Means in Production
Standard RAG retrieves on every request, whether or not the model needs it. Self-RAG teaches the model to decide — and to grade its own output. The mechanism is more interesting than the leaderboard numbers.
Tree of Thoughts: What the Paper Actually Says About LLM Search
Chain-of-thought is one path through a reasoning problem. Tree of Thoughts is a search over many paths. The NeurIPS 2023 paper shows 74% vs 4% on certain benchmarks. The production story is harder.
BitNet b1.58: What the 1-bit LLM Paper Actually Says
A 70B BitNet model fits in 7GB instead of 140GB — and the math says output quality matches FP16 at scale. The catch: you can't convert existing models. Here's what the paper actually proves, and why the hardware story matters more than the math.
Llama 2: What the Open-Source RLHF Paper Actually Says
By turn 15, your agent is ignoring its system prompt. Llama 2 documented the fix — Ghost Attention — but also the full iterative RLHF pipeline with rejection sampling that outperforms PPO alone. Here's what the paper actually says.
MapReduce: What the Paper Actually Says
MapReduce isn't just 'map then reduce.' The paper is a fault-tolerance paper disguised as a programming model. The actual insight is about how to make commodity hardware reliable enough to process terabytes — and the tradeoffs that follow from that choice.
Multi-Token Prediction: What the Meta FAIR Paper Actually Says
Speculative decoding solves LLM generation latency, but requires a separate draft model — which you have to deploy, tune, and keep in sync. Multi-Token Prediction solves the same problem at training time by adding prediction heads to the main model. Here's the mechanism and what it costs.
Raft: What the Understandable Consensus Algorithm Paper Actually Says
Raft isn't just Paxos made simpler. It's a specific set of design decisions — each one a tradeoff — that power etcd, CockroachDB, and TiKV. The paper is worth reading because the failure modes live in the decisions, not the math.
S-LoRA: What the Paper Actually Says About Serving Thousands of LoRA Adapters
LoRA makes fine-tuning cheap. Serving thousands of LoRA adapters from a single GPU cluster is a different problem entirely — one that requires rethinking memory management, batching, and tensor parallelism from scratch.
Spanner: What Google's Globally-Distributed Database Paper Actually Says
Spanner achieves external consistency across a globally-distributed database using atomic clocks and GPS receivers. The insight isn't exotic hardware — it's that bounded clock uncertainty lets you make a specific, provable guarantee about transaction ordering that 'eventually consistent' databases can't offer.
StreamingLLM: What the Attention Sink Paper Actually Says
Sliding your KV cache window shouldn't crash model quality — but it does. The reason is attention sinks: models learn to dump excess softmax mass on the first few tokens. StreamingLLM keeps those sinks and recovers stable infinite-length generation.
Zanzibar: What Google's Authorization Paper Actually Says
Google serves 10M authorization checks per second with 3ms p95 latency. The paper behind it — Zanzibar — introduces a consistency mechanism called zookies that nobody talks about, and a data model that's quietly become the industry standard for permission systems.
Bigtable: What the Distributed Storage Paper Actually Says
Bigtable's data model is three lines: a sorted map indexed by (row, column, timestamp). Everything that makes it powerful — and everything that makes it dangerous — follows from those three lines.
DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving
Every GPU running LLM serving is doing two fundamentally different jobs simultaneously — and that interference is silently destroying your SLOs. DistServe separates prefill and decode onto different machines. Here's the mechanism, the cost, and when it's actually worth it.
FrugalGPT: What the Paper Actually Says
GPT-4-level quality at 4× lower cost isn't wishful thinking — FrugalGPT shows how LLM cascades route queries to cheap models when they're sufficient and expensive models when they're not. Here's the algorithm, the tradeoffs, and where it breaks.
Generative Agents: What the Paper Actually Says
The Generative Agents paper isn't just a simulation demo. It's a formal architecture for agent memory: stream, retrieval, reflection, and planning. Every agent framework you're using borrowed from it.
GFS: What the Google File System Paper Actually Says
GFS isn't just 'distributed POSIX.' The paper is an exercise in deliberately violating POSIX to make large-scale sequential workloads tractable on unreliable hardware. The consistency model it chose — intentionally weaker than what you'd expect — defines everything downstream, including HDFS.
HNSW: What the Vector Similarity Search Paper Actually Says
Every vector database runs HNSW under the hood. Most engineers tune M and ef_construction without understanding what they're actually trading off. The paper rewards a close read.
What Every Backend Engineer Should Know About Attention
RNNs forced you to wait for token 100 before processing token 101. Transformers parallelize the whole sequence. Here's why that matters for production systems.
LoRA: What the Low-Rank Adaptation Paper Actually Says
Full fine-tuning GPT-3 requires roughly 1.4 TB of optimizer state. LoRA gets trainable parameters down to ~4.7M with comparable quality — by exploiting a property of pre-trained models that most engineers know about but don't fully reason from.
MegaScale: What ByteDance's 12,288-GPU Training Paper Actually Says
At 12,288 GPUs, you see roughly one hardware failure per day. Standard training frameworks have no answer for this. MegaScale is ByteDance's account of what actually breaks when you scale LLM training to 10,000+ GPUs — and the algorithm-system co-design changes that kept MFU above 55%.
Ring Attention: What the Near-Infinite Context Paper Actually Says
Extending context beyond what fits on one GPU isn't just a memory problem — it's a communication design problem. Ring Attention sequences the K/V data through a ring of devices and hides the transfers behind computation. Here's what that actually costs in production.
RAG in Production: The Failure Modes Nobody Puts in the Pitch Deck
RAG is in every AI pitch deck. The failure modes rarely are — retrieval collapse, frozen encoders, approximate MIPS. Here's what actually breaks it in production.
Toolformer: What the Paper Actually Says
A 6.7B model beats GPT-3 175B on math by learning to use a calculator. Toolformer's self-supervised training pipeline is the interesting part — and it's more constrained than the demos imply.
Dapper: What Google's Distributed Tracing Paper Actually Says
Every distributed tracing tool you use — Jaeger, Zipkin, OpenTelemetry — descends from one design decision Google made in 2010: sample at the trace root, not per-span. The paper explains why, and the failure modes it didn't fully solve.
DeepSeek-V3: What the Frontier-on-a-Budget Paper Actually Says
DeepSeek-V3 trained a 671B-parameter frontier model for ~$5.5M. The paper is less about model quality and more about whether the training stack itself is the bottleneck — and how to engineer around it.
H2O: Heavy-Hitter Oracle for KV Cache Eviction — What the Paper Actually Says
StreamingLLM keeps the first tokens and a sliding window — a positional bet. H2O makes a different bet: keep the tokens that actually received attention, not the ones that happened to appear early. The paper shows you can cut KV cache by 5× with a greedy per-step eviction policy that costs almost nothing to implement.
Kafka: What the Original Paper Actually Says
The original Kafka paper from 2011 had no replication. A broker failure made all unconsumed messages permanently unavailable. The paper treats this as a limitation to fix later, not a deal-breaker. Understanding why explains more about Kafka's design philosophy than any architecture diagram.
Mixture of Experts: What the Architecture Actually Does to Your Inference Budget
Mixtral has 46.7B parameters but activates only 12.9B per token. MoE is how you get GPT-4-tier capacity at GPT-3.5-tier compute — and why your inference stack needs to treat it differently.
Mamba: What the Selective State Space Paper Actually Says
Transformers scale quadratically with sequence length and carry a growing KV cache that never shrinks. Mamba proposes a different trade: compress context into a fixed-size state, process tokens recurrently at inference, and do it faster than attention at long sequences. Here's the mechanism, what it actually buys you, and where it quietly fails.
MapReduce: What the Google Paper Actually Says
The 2004 Google paper that gave us Hadoop — and everything that replaced it — is worth reading not for the map/reduce abstraction itself, but for the fault tolerance model and the straggler insight. The failure modes are still the failure modes.
Mixture of Depths: What the Paper Actually Says
Every transformer layer processes every token, even when 90% of that work does nothing useful. Mixture of Depths lets the model skip layers for tokens that don't need them — and the compute budget stays completely predictable.
SARATHI: What the Chunked-Prefill Paper Actually Says
Continuous batching fixed GPU utilization but created a new problem: long prefills stall every in-flight decode for seconds. SARATHI splits prefill into chunks and interleaves them with decode, eliminating the stall without disaggregating your cluster. Here's the mechanism, the math, and where it breaks down.
Scaling laws are not just about research budgets
Loss follows a power law across seven orders of magnitude of compute. Kaplan et al.'s scaling laws are a decision framework — not just research trivia.
SGLang and RadixAttention: What the Paper Actually Says
Multi-call LLM programs recompute the same KV cache thousands of times per day. RadixAttention fixes this by maintaining a global radix tree of cached KV blocks — turning redundant prefill into a lookup. Here's the mechanism, the numbers, and where it falls apart.
T5: What the Text-to-Text Paper Actually Says
Every instruction-tuned model today owes something to T5's core idea: every NLP task is just sequence-to-sequence. But the paper's real contribution is a systematic ablation of what actually helps in transfer learning — and several of the answers are counterintuitive.
Titans: What the Test-Time Memorization Paper Actually Says
Attention is quadratic. SSMs compress everything into a fixed state. Titans takes a third path: a neural memory module whose weights are the memory, updated via gradient descent at inference time. Here's the mechanism, what the benchmark numbers actually say, and why you shouldn't deploy this without thinking carefully about inference cost.
Self-correcting agents in production
The labeling bottleneck is real. Constitutional AI's principle-based approach teaches agents to critique themselves — fewer human labels, faster iteration, new tradeoffs.
Switch Transformers: What the Sparse MoE Scaling Paper Actually Says
Every modern large model — Mixtral, DeepSeek, Gemini — routes tokens through sparse experts. The design decisions in all of them trace back to one 2021 Google Brain paper. The paper is worth reading because the failure modes live in the routing logic, not the math.
Pregel: What the Large-Scale Graph Processing Paper Actually Says
PageRank in MapReduce is O(iterations × full dataset reloads). Pregel fixes this by keeping the graph in memory across iterations and replacing disk I/O with message passing. The 'think like a vertex' model is the insight — BSP is the implementation.
Mooncake: What the KV-Cache-Centric Disaggregated Serving Paper Actually Says
DistServe disaggregates prefill from decode. SGLang caches KV cache per-instance. Mooncake goes further: it treats the KV cache as a shared, distributed resource that the scheduler routes around — turning a fleet of 50 isolated caches into one coherent pool.
LLM.int8(): What the 8-bit Matrix Multiplication Paper Actually Says
INT8 quantization works perfectly for vision models. For LLMs above ~7B parameters it silently destroys quality — unless you understand why outlier features emerge at scale and how mixed-precision decomposition works around them.
The Llama 3 Herd of Models: What the Paper Actually Says
Llama 3's 405B benchmark numbers are fine. The paper is actually about something more useful: what decisions you make when you can train on 15T tokens across 16K H100s, and which of those decisions transfer to your deployment.
EAGLE: Speculative Decoding with Feature-Level Prediction — What the Paper Actually Says
Standard speculative decoding tops out at 2–2.4x speedup because token-level draft prediction is hard. EAGLE sidesteps this by predicting at the feature level instead — the second-to-top-layer hidden states — which are far more predictable than discrete tokens. The result is 3–4x with a draft head that's 0.3% the size of the target model.
Cassandra: What the Paper Actually Says
We had a Cassandra cluster where DELETE operations made reads progressively slower until queries timed out. Adding more disk space made it worse. The root cause is described precisely in the 2009 paper — but only if you understand that Cassandra cannot actually delete data.