The race to improve large language model (LLM) inference speed has led to significant innovation in speculative decoding. As AI models grow larger and serve increasingly diverse workloads, reducing latency while maintaining output quality has become one of the industry’s most important challenges.
Tencent has now introduced AngelSpec, an open-source, PyTorch-native training framework designed specifically for speculative-decoding draft models. The framework supports both autoregressive Multi-Token Prediction (MTP) and the block-parallel DFlash family, bringing multiple speculative decoding approaches under a single training infrastructure.
Unlike many previous speculative-decoding systems that focus on creating one generalized drafter optimized for benchmark averages, AngelSpec takes a different path. Tencent recognizes that real-world workloads are highly heterogeneous. Conversations, coding tasks, mathematical reasoning, and enterprise workflows all exhibit different prediction characteristics. A single drafting strategy rarely performs optimally across every scenario.
By treating workload diversity as a core design principle, AngelSpec introduces specialized training approaches, architecture designs, and verification strategies tailored to different use cases.
In this article, we explore how AngelSpec works, why Tencent developed separate drafting approaches, the innovations behind MTP and DFly, benchmark performance results, framework architecture, and what this release means for the future of speculative decoding.
What Is AngelSpec?
AngelSpec is Tencent’s open-source framework for training speculative-decoding draft models.
The framework is built on top of TorchSpec and provides a unified environment for training, evaluating, and deploying multiple speculative decoding architectures.
The release includes support for:
- Multi-Token Prediction (MTP)
- DFly
- DFlash
- DFlare
- Eagle3
- DSpark
- Additional speculative decoding variants
Instead of requiring separate pipelines for each architecture, AngelSpec allows organizations to train multiple draft models through a configuration-driven workflow.
This significantly simplifies experimentation while improving scalability for research and production environments.
Understanding Speculative Decoding
Before exploring AngelSpec’s innovations, it’s important to understand speculative decoding itself.
Speculative decoding is a technique used to accelerate language model inference without changing model outputs.
The process works as follows:
- A lightweight draft model predicts several future tokens.
- The target model verifies those predictions in a single forward pass.
- Rejection sampling ensures correctness.
- Accepted tokens are committed.
- Rejected tokens are regenerated.
Because verification preserves the target model’s distribution, speculative decoding remains lossless.
The primary goal is to generate more tokens per verification cycle while minimizing computational overhead.
The Two Factors That Determine Speedup
Speculative decoding performance depends on two key variables.
Acceptance Rate
The percentage of proposed draft tokens accepted by the target model.
Higher acceptance means fewer corrections and greater acceleration.
Draft–Verify Cycle Time
The total time required to generate and verify proposed tokens.
Longer verification sequences can increase computational cost.
The challenge is that these two metrics often move in opposite directions.
Increasing proposal depth may generate more tokens, but it can also reduce acceptance rates.
Finding the right balance is critical.
Why a Universal Drafter Often Fails
Most speculative decoding research attempts to create a single draft model that performs well across multiple benchmark datasets.
However, real-world traffic rarely resembles benchmark mixtures.
Different domains exhibit very different token prediction patterns.
A universal drafter optimized for average performance often becomes suboptimal everywhere.
Tencent’s AngelSpec addresses this problem directly.
Rather than forcing one drafting strategy across all workloads, the framework introduces specialized drafters optimized for specific task categories.
Open-Ended Conversation vs Structured Reasoning
The distinction becomes clear when comparing conversational AI and programming tasks.
Conversational Workloads
Open-ended conversations contain high entropy.
Many responses can be considered valid.
For example:
- Chatbots
- Virtual assistants
- Customer support
- Creative writing
Multiple continuations may satisfy the user’s intent.
As proposal depth increases, acceptance rates decline rapidly because there are many plausible next tokens.
In this environment, shorter draft sequences perform better.
This makes Multi-Token Prediction (MTP) particularly effective.
Code and Mathematical Reasoning
Programming and mathematical tasks behave differently.
These domains contain strong structural constraints.
Examples include:
- Syntax rules
- Variable names
- Function structures
- Equations
- Step-by-step reasoning
Future tokens become significantly more predictable.
This creates longer stretches where speculative proposals can be accepted successfully.
Block-parallel drafting performs exceptionally well in these environments.
Why AngelSpec Uses Two Drafting Strategies
Rather than forcing a compromise, AngelSpec ships with two complementary drafters.
MTP Models
Optimized for:
- Diverse conversations
- Open-ended interactions
- General language generation
Training data emphasizes conversational diversity and semantic flexibility.
Block-Diffusion Models
Optimized for:
- Programming tasks
- Mathematical reasoning
- Structured outputs
Training data includes code-focused and math-focused datasets.
This specialization allows each architecture to excel in its intended domain.
The MTP Training Challenge
Tencent identified an important limitation in the original Hy3 implementation.
The base Hy3 model uses:
- A single MTP layer
- No recurrent self-conditioned unrolling
While the MTP block can be reused during inference, training never exposed the model to long self-generated prediction chains.
This creates a train-inference mismatch.
Errors compound as prediction depth increases.
Acceptance rates for deeper positions decline substantially compared to earlier predictions.
AngelSpec’s Shared-Parameter Multi-Depth Training
To solve this problem, Tencent introduced a shared-parameter multi-depth architecture.
The system maintains:
- Multiple logical prediction depths
- One shared physical MTP block
During training:
- The MTP block is autoregressively unrolled.
- Predictions feed subsequent invocations.
- Self-generated outputs become future inputs.
This better matches inference-time behavior.
Training-Time Test (TTT) Principle
AngelSpec incorporates concepts from EAGLE-3 through Training-Time Test (TTT).
Instead of feeding ground-truth tokens into subsequent depths:
- Depth k generates a prediction.
- Depth k+1 receives the argmax prediction.
- Supervision remains depth-specific.
This trains the model under realistic conditions where previous predictions may contain errors.
The result is significantly improved robustness.
Frozen Target Models and Rollout Training
Two additional design decisions contribute heavily to AngelSpec’s improvements.
Frozen Backbone
The target backbone remains frozen.
MTP inputs from the backbone are detached.
This allows the drafter to improve proposals without altering the target distribution that verification must preserve.
Target-Model Rollout
Instead of using original reference data directly, responses are generated by the frozen target model itself.
This provides:
- Realistic token choices
- Accurate hidden-state trajectories
- Authentic uncertainty patterns
As a result, training more closely mirrors production environments.
MTP Performance Improvements
The impact of these changes appears clearly in acceptance metrics.
At temperature zero:
| Metric | Original | AngelSpec |
|---|---|---|
| Mean Acceptance | 52.8% | 66.4% |
| Mean Accepted Length | 2.58 | 2.99 |
The first prediction position remains nearly unchanged:
- 0.799 → 0.814
However, deeper positions improve dramatically.
Examples include:
GSM8K
- p3 acceptance: 0.290 → 0.706
HumanEval
- p3 acceptance: 0.387 → 0.757
These gains show that AngelSpec successfully addresses depth-related degradation.
Introducing DFly
Beyond MTP, Tencent introduces DFly, a next-generation block-diffusion architecture.
DFly builds upon DFlash while introducing two major architectural improvements.
Hybrid Target-Conditioning Backbone
DFlash combines hidden states from multiple target layers into a single context representation.
While effective, this approach limits specialization.
Every draft layer receives the same context.
DFly adopts a hybrid approach.
It combines:
Shared Semantic Foundation
A fully connected branch generates common semantic features.
Layer-Specific Refinement
Each draft layer learns its own fusion weights.
This allows specialized views of the target hierarchy.
The additional computation remains minimal.
Only D × T scalar weights are introduced.
Softmax coefficients can even be precomputed after training.
Predecessor-Conditioned Autoregressive Head
The second innovation addresses suffix acceptance decay.
Traditional parallel backbones predict block positions independently.
They cannot observe which continuations were selected earlier.
This weakens later predictions.
DFly solves the issue using a lightweight sequential head.
The architecture works as follows:
- Parallel backbone predicts candidate positions.
- Small autoregressive head processes outputs left-to-right.
- Prefix-conditioned distributions are generated.
The expensive backbone remains parallel.
Only the lightweight head introduces sequential computation.
This delivers better acceptance without sacrificing efficiency.
DFly Benchmark Results on Qwen3-8B
Performance results show significant gains.
Average Mean Accepted Length:
| Model | Mean Accepted Length |
|---|---|
| DFly | 5.41 |
| DSpark | 5.32 |
| DFlash | 4.57 |
| MTP | 3.24 |
DFly achieved the strongest results across all five evaluated math and code benchmarks.
DSpark retained a slight advantage on MT-Bench:
- DSpark: 3.77
- DFly: 3.67
This aligns with DFly’s optimization toward coding and mathematical workloads.
DFly Results on Hy3-A21B
Performance improvements become even more pronounced on Hy3-A21B.
Average Mean Accepted Length:
| Model | Mean Accepted Length |
|---|---|
| DFly | 4.79 |
| DFlash | 3.69 |
| MTP | 3.00 |
Relative improvements:
- +29.8% vs DFlash
- +59.7% vs MTP
DFly improved results across all six reported benchmarks.
Understanding the Ablation Study
Tencent’s cumulative ablation analysis reveals where these gains originate.
Results under greedy decoding:
| Configuration | Accepted Length |
|---|---|
| DFlash Backbone | 3.77 |
| DFly Backbone | 4.40 |
| + Markov Head | 4.56 |
| + Hidden Correction | 4.60 |
| + Code/Math Data | 4.75 |
Each architectural enhancement contributes incrementally.
Together, they produce substantial performance improvements.
Expanded Code and Math Training Data
AngelSpec expands training data significantly.
Additional prompts include:
- 500K coding examples
- 200K mathematical examples
Data sources include:
To prevent benchmark contamination:
Any prompt sharing a contiguous 16-token span with evaluation samples is removed before response generation.
This ensures cleaner and more reliable evaluation.
AngelSpec Framework Architecture
AngelSpec extends TorchSpec with several important infrastructure improvements.
The framework follows a disaggregated design.
Inference engines:
- Run frozen target models
- Stream hidden states
- Communicate through Mooncake-backed RDMA storage
Training workers receive hidden states directly.
Importantly, this is accomplished using public vLLM APIs without modifying engine source code.
Key Framework Features
Parallel TTT Rollout
Training-Time Test rollout executes across entire sequences in parallel.
FlexAttention and logsumexp merging preserve memory efficiency close to standard causal passes.
Long-Context Training
AngelSpec supports context lengths up to:
- 128,000 tokens
Ulysses sequence parallelism enables scalable long-context training.
Document-Aware Sequence Packing
Three isolation mechanisms prevent leakage:
- Attention document gate
- Depth-shift document gate
- Document-local position encoding
Unit tests verify complete cross-document isolation.
Real Serving Evaluation
A dedicated evaluation server periodically performs genuine speculative decoding.
Metrics include:
- Mean accepted length
- Position-specific acceptance
- Real serving behavior
This provides more realistic measurements than offline benchmarks.
Pluggable Architecture
AngelSpec introduces modular integration points.
Target Plugins
vLLM runtime plugins operate without source modifications.
Objective Plugins
Training objectives can be selected through configuration.
Optimizer Plugins
Muon can be used as an alternative to AdamW.
This flexibility simplifies experimentation.
Backend Support
AngelSpec currently supports multiple serving environments.
First-Class Support
Community Support
This broad compatibility increases adoption potential across research and production ecosystems.
Key Takeaways
Tencent’s AngelSpec represents a significant advancement in speculative decoding research.
The framework introduces a unified environment capable of training multiple draft architectures while recognizing that different workloads require different drafting strategies.
Some of the most important highlights include:
- Seven checkpoints available on Hugging Face and ModelScope
- Support for six draft architectures through one pipeline
- DFly achieves 4.79 mean accepted length on Hy3-A21B
- DFly outperforms DFlash and MTP across major benchmarks
- Up to 2.40× inference speedup over autoregressive decoding on HY3-295B-A21B
- D-cut reaches 981 tokens per second at concurrency 64
- Throughput increases by 15.7% compared to standard DFly while sacrificing only 2.8% acceptance
Final Thoughts
Tencent’s release of AngelSpec highlights a growing realization within the AI community: speculative decoding is not a one-size-fits-all problem. Different workloads demand different drafting strategies, and maximizing inference performance requires architectures specifically designed for those workloads.
By combining Multi-Token Prediction, DFly, DFlash, DSpark, Eagle3, and other approaches under a unified framework, AngelSpec provides researchers and practitioners with a powerful platform for exploring next-generation decoding methods.
The framework’s innovations—including Training-Time Test rollout, hybrid target conditioning, predecessor-conditioned autoregressive heads, long-context training support, and real-serving evaluation—demonstrate how speculative decoding continues to evolve beyond simple acceleration techniques.
As language models become larger and AI applications serve increasingly diverse workloads, frameworks like AngelSpec may play a crucial role in delivering faster, more efficient, and more scalable inference systems without sacrificing output quality.
Discover more from AiTechtonic - AI & Informative News
Subscribe to get the latest posts sent to your email.