Tag
Alignment
3 posts tagged Alignment.
RLHF: What the InstructGPT Paper Actually Says
A 1.3B InstructGPT model beat 175B GPT-3 on human preference evals. The method that achieved this—RLHF—is also the technique that introduces a reward model inference bottleneck, PPO instability at scale, and a documented regression on academic benchmarks. Here's what the paper actually says.
LIMA: What the Superficial Alignment Paper Actually Says
A 1,000-example SFT run on LLaMA-65B outperforms models trained with 52,000 examples and beats text-davinci-003 in human evals. LIMA's core claim: your model already knows how to be helpful — alignment is just format learning.
Llama 2: What the Open-Source RLHF Paper Actually Says
By turn 15, your agent is ignoring its system prompt. Llama 2 documented the fix — Ghost Attention — but also the full iterative RLHF pipeline with rejection sampling that outperforms PPO alone. Here's what the paper actually says.