Tag
Architecture
3 posts tagged Architecture.
What Every Backend Engineer Should Know About Attention
RNNs forced you to wait for token 100 before processing token 101. Transformers parallelize the whole sequence. Here's why that matters for production systems.
Mixture of Experts: What the Architecture Actually Does to Your Inference Budget
Mixtral has 46.7B parameters but activates only 12.9B per token. MoE is how you get GPT-4-tier capacity at GPT-3.5-tier compute — and why your inference stack needs to treat it differently.
Titans: What the Test-Time Memorization Paper Actually Says
Attention is quadratic. SSMs compress everything into a fixed state. Titans takes a third path: a neural memory module whose weights are the memory, updated via gradient descent at inference time. Here's the mechanism, what the benchmark numbers actually say, and why you shouldn't deploy this without thinking carefully about inference cost.