![Use a small model to generate a 'draft' output, then use a l...](https://readwise-assets.s3.amazonaws.com/static/images/article4.6bc1851654a0.png)
Josh BeckmanUse a small model to generate a ‘draft’ output, then use a larger and smarter model to score the ‘draft’, then use a rejection sampling scheme to accept the tokens which are agreed by the small and large models.
In tests, they find that a draft model can give them speedups ranging between 1.92X (on a summarization benchmark called XSum) and 2.46X on a code generation task called HumanEval.
✉️Josh BeckmanFwd: Import AI 317: DeepMind Speeds Up Language Model Sampling; Voice Cloning Tech Gets Abused; More Scaling Laws for RL