Speculative decoding
Speeding up text generation by letting a small model guess several tokens ahead and a large model check them at once.
Generating one token at a time leaves the GPU underused. In speculative decoding, a small, fast draft model proposes several next tokens, and the large model verifies them all in a single pass, keeping those it agrees with. When guesses are often right, output comes two to three times faster with the same result the large model alone would give. It is widely used by inference platforms and built into common serving software.