Writing
Engineering writing
Engineering writing on distributed systems and applied AI — production LLM infrastructure, agent memory and observability, search platforms, and system design from Ashwani Jha.
ZeRO: How Microsoft Trained 100B-Parameter Models Without Model Parallelism
Data parallel training silently copies the entire optimizer state to every GPU. For a 7.5B-parameter model with Adam, that's 120 GB per device before you process a single batch. ZeRO eliminates that redundancy without changing the math.
How I think about agent memory
Most LLM agents are amnesiacs. The fix isn't a bigger context window — it's a memory system with four layers, explainable retrieval, and a feedback loop.
How I think about agent observability
Traditional APM was built for web requests, not for agents that loop, retry, branch, and spend dollars per call. Here's what agent observability actually needs — and why I built Peekr.
The AI agent observability stack — what to measure, what to use
Most teams reach for one observability tool and call it solved. Different tools measure different things — traces, hallucinations, cost, drift. Here's a map of the space and where to start.
Building a Brain: How I Gave My AI Assistant a Cognitive Architecture
Most AI assistants are stateless. I wanted one that remembers, learns, and adapts — so I built Friday a brain.
What Every Backend Engineer Should Know About Attention
RNNs forced you to wait for token 100 before processing token 101. Transformers parallelize the whole sequence. Here's why that matters for production systems.
RAG in Production: The Failure Modes Nobody Puts in the Pitch Deck
RAG is in every AI pitch deck. The failure modes rarely are — retrieval collapse, frozen encoders, approximate MIPS. Here's what actually breaks it in production.
Mixture of Experts: What the Architecture Actually Does to Your Inference Budget
Mixtral has 46.7B parameters but activates only 12.9B per token. MoE is how you get GPT-4-tier capacity at GPT-3.5-tier compute — and why your inference stack needs to treat it differently.
Scaling laws are not just about research budgets
Loss follows a power law across seven orders of magnitude of compute. Kaplan et al.'s scaling laws are a decision framework — not just research trivia.
Self-correcting agents in production
The labeling bottleneck is real. Constitutional AI's principle-based approach teaches agents to critique themselves — fewer human labels, faster iteration, new tradeoffs.
Paper breakdowns — the series
All 57 breakdowns →Close readings of the systems and ML papers behind the tools everyone uses — the mechanism, the numbers, and when not to reach for the idea.
- Speculative Decoding: What the Inference Speed Paper Actually Says
- AWQ: What the Activation-Aware Weight Quantization Paper Actually Says
- Chain-of-Thought Prompting: What the Wei et al. Paper Actually Says
- Continuous Batching: What the Orca Paper Actually Says
- DeepSeek-R1: What Reinforcement Learning with Verifiable Rewards Actually Does
- GQA: What the Grouped Query Attention Paper Actually Says
- Mamba: What the Selective State Space Paper Actually Says
- Megatron-LM: What the Tensor Parallelism Paper Actually Says