← All projects

Suffix Speculative Decoding for LLM Serving

Integrated suffix-based speculative decoding into the Hugging Face generation stack — a draft-model-free proposer reusing repeated context patterns for 1.74x faster decoding — plus a unified verification interface for suffix and tree-based speculation, achieving 1.30x speedup on tree-spec workloads.

PyTorch LLM Inference Hugging Face Transformers
Source on GitHub →
ML SYSTEMS / INFERENCE02 / SYSTEM STUDY
Reuse context. Verify together.Repeated context is indexed into a suffix tree. Branching token candidates feed one target-model verification pass with a KV cache. Accepted tokens extend the context and rejected tokens are discarded.CONTEXT / CANDIDATE PROPOSALTARGET-MODEL VERIFICATIONthecatsatthecat·satranonbytoinLLMKV cacheONE FORWARD PASSsatona×accept prefix / discard rest
Reuse context. Verify together.No separate draft model

Overview

An open-source contribution to Hugging Face’s transformers generation stack, adding suffix-based speculative decoding: a proposer that speeds up autoregressive generation without needing a separate draft model, by reusing repeated patterns already present in the context. I also designed a unified verification interface so that both suffix-based and tree-based speculative candidates can be checked in a single target-model forward pass.

Continue to update… soon.