Tag
Systems
30 posts tagged Systems.
How I think about agent memory
Most LLM agents are amnesiacs. The fix isn't a bigger context window — it's a memory system with four layers, explainable retrieval, and a feedback loop.
StreamingLLM: What the Attention Sink Paper Actually Says
Sliding your KV cache window shouldn't crash model quality — but it does. The reason is attention sinks: models learn to dump excess softmax mass on the first few tokens. StreamingLLM keeps those sinks and recovers stable infinite-length generation.
Zanzibar: What Google's Authorization Paper Actually Says
Google serves 10M authorization checks per second with 3ms p95 latency. The paper behind it — Zanzibar — introduces a consistency mechanism called zookies that nobody talks about, and a data model that's quietly become the industry standard for permission systems.
How I think about agent observability
Traditional APM was built for web requests, not for agents that loop, retry, branch, and spend dollars per call. Here's what agent observability actually needs — and why I built Peekr.
Bigtable: What the Distributed Storage Paper Actually Says
Bigtable's data model is three lines: a sorted map indexed by (row, column, timestamp). Everything that makes it powerful — and everything that makes it dangerous — follows from those three lines.
DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving
Every GPU running LLM serving is doing two fundamentally different jobs simultaneously — and that interference is silently destroying your SLOs. DistServe separates prefill and decode onto different machines. Here's the mechanism, the cost, and when it's actually worth it.
FrugalGPT: What the Paper Actually Says
GPT-4-level quality at 4× lower cost isn't wishful thinking — FrugalGPT shows how LLM cascades route queries to cheap models when they're sufficient and expensive models when they're not. Here's the algorithm, the tradeoffs, and where it breaks.
GFS: What the Google File System Paper Actually Says
GFS isn't just 'distributed POSIX.' The paper is an exercise in deliberately violating POSIX to make large-scale sequential workloads tractable on unreliable hardware. The consistency model it chose — intentionally weaker than what you'd expect — defines everything downstream, including HDFS.
HNSW: What the Vector Similarity Search Paper Actually Says
Every vector database runs HNSW under the hood. Most engineers tune M and ef_construction without understanding what they're actually trading off. The paper rewards a close read.
LoRA: What the Low-Rank Adaptation Paper Actually Says
Full fine-tuning GPT-3 requires roughly 1.4 TB of optimizer state. LoRA gets trainable parameters down to ~4.7M with comparable quality — by exploiting a property of pre-trained models that most engineers know about but don't fully reason from.
RAG in Production: The Failure Modes Nobody Puts in the Pitch Deck
RAG is in every AI pitch deck. The failure modes rarely are — retrieval collapse, frozen encoders, approximate MIPS. Here's what actually breaks it in production.
Dapper: What Google's Distributed Tracing Paper Actually Says
Every distributed tracing tool you use — Jaeger, Zipkin, OpenTelemetry — descends from one design decision Google made in 2010: sample at the trace root, not per-span. The paper explains why, and the failure modes it didn't fully solve.
DeepSeek-V3: What the Frontier-on-a-Budget Paper Actually Says
DeepSeek-V3 trained a 671B-parameter frontier model for ~$5.5M. The paper is less about model quality and more about whether the training stack itself is the bottleneck — and how to engineer around it.
H2O: Heavy-Hitter Oracle for KV Cache Eviction — What the Paper Actually Says
StreamingLLM keeps the first tokens and a sliding window — a positional bet. H2O makes a different bet: keep the tokens that actually received attention, not the ones that happened to appear early. The paper shows you can cut KV cache by 5× with a greedy per-step eviction policy that costs almost nothing to implement.
Kafka: What the Original Paper Actually Says
The original Kafka paper from 2011 had no replication. A broker failure made all unconsumed messages permanently unavailable. The paper treats this as a limitation to fix later, not a deal-breaker. Understanding why explains more about Kafka's design philosophy than any architecture diagram.
Mamba: What the Selective State Space Paper Actually Says
Transformers scale quadratically with sequence length and carry a growing KV cache that never shrinks. Mamba proposes a different trade: compress context into a fixed-size state, process tokens recurrently at inference, and do it faster than attention at long sequences. Here's the mechanism, what it actually buys you, and where it quietly fails.
MapReduce: What the Google Paper Actually Says
The 2004 Google paper that gave us Hadoop — and everything that replaced it — is worth reading not for the map/reduce abstraction itself, but for the fault tolerance model and the straggler insight. The failure modes are still the failure modes.
Mixture of Depths: What the Paper Actually Says
Every transformer layer processes every token, even when 90% of that work does nothing useful. Mixture of Depths lets the model skip layers for tokens that don't need them — and the compute budget stays completely predictable.
SARATHI: What the Chunked-Prefill Paper Actually Says
Continuous batching fixed GPU utilization but created a new problem: long prefills stall every in-flight decode for seconds. SARATHI splits prefill into chunks and interleaves them with decode, eliminating the stall without disaggregating your cluster. Here's the mechanism, the math, and where it breaks down.
Scaling laws are not just about research budgets
Loss follows a power law across seven orders of magnitude of compute. Kaplan et al.'s scaling laws are a decision framework — not just research trivia.
SGLang and RadixAttention: What the Paper Actually Says
Multi-call LLM programs recompute the same KV cache thousands of times per day. RadixAttention fixes this by maintaining a global radix tree of cached KV blocks — turning redundant prefill into a lookup. Here's the mechanism, the numbers, and where it falls apart.
Titans: What the Test-Time Memorization Paper Actually Says
Attention is quadratic. SSMs compress everything into a fixed state. Titans takes a third path: a neural memory module whose weights are the memory, updated via gradient descent at inference time. Here's the mechanism, what the benchmark numbers actually say, and why you shouldn't deploy this without thinking carefully about inference cost.
Self-correcting agents in production
The labeling bottleneck is real. Constitutional AI's principle-based approach teaches agents to critique themselves — fewer human labels, faster iteration, new tradeoffs.
Switch Transformers: What the Sparse MoE Scaling Paper Actually Says
Every modern large model — Mixtral, DeepSeek, Gemini — routes tokens through sparse experts. The design decisions in all of them trace back to one 2021 Google Brain paper. The paper is worth reading because the failure modes live in the routing logic, not the math.
Pregel: What the Large-Scale Graph Processing Paper Actually Says
PageRank in MapReduce is O(iterations × full dataset reloads). Pregel fixes this by keeping the graph in memory across iterations and replacing disk I/O with message passing. The 'think like a vertex' model is the insight — BSP is the implementation.
Mooncake: What the KV-Cache-Centric Disaggregated Serving Paper Actually Says
DistServe disaggregates prefill from decode. SGLang caches KV cache per-instance. Mooncake goes further: it treats the KV cache as a shared, distributed resource that the scheduler routes around — turning a fleet of 50 isolated caches into one coherent pool.
LLM.int8(): What the 8-bit Matrix Multiplication Paper Actually Says
INT8 quantization works perfectly for vision models. For LLMs above ~7B parameters it silently destroys quality — unless you understand why outlier features emerge at scale and how mixed-precision decomposition works around them.
The Llama 3 Herd of Models: What the Paper Actually Says
Llama 3's 405B benchmark numbers are fine. The paper is actually about something more useful: what decisions you make when you can train on 15T tokens across 16K H100s, and which of those decisions transfer to your deployment.
EAGLE: Speculative Decoding with Feature-Level Prediction — What the Paper Actually Says
Standard speculative decoding tops out at 2–2.4x speedup because token-level draft prediction is hard. EAGLE sidesteps this by predicting at the feature level instead — the second-to-top-layer hidden states — which are far more predictable than discrete tokens. The result is 3–4x with a draft head that's 0.3% the size of the target model.
Cassandra: What the Paper Actually Says
We had a Cassandra cluster where DELETE operations made reads progressively slower until queries timed out. Adding more disk space made it worse. The root cause is described precisely in the 2009 paper — but only if you understand that Cassandra cannot actually delete data.