فصل تسخیرشدگی برای عشق و جادوگران

Introduction Architecture Basic Architecture Multi-Head Latent Attention DeepSeekMoE with Auxiliary-Loss-Free Load Balancing Multi-Token Prediction Infrastructures Compute Clusters Training Framework DualPipe and Computation-Communication Overlap Efficient Implementation of Cross-Node All-to-All Communication Extremely Memory Saving with Minimal Overhead FP8 Training Mixed Precision Framework Improved Precision from Quantization and Multiplication Low-Precision Storage and Communication Inference and Deployment Prefilling Decoding Suggestions on Hardware Design Communication Hardware Compute Hardware Pre-Training Data Construction Hyper-Parameters Long Context Extension Evaluations Evaluation Benchmarks Evaluation Results Discussion Ablation Studies for Multi-Token Prediction Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy Batch-Wise Load Balance VS. Sequence-Wise Load Balance Post-Training Supervised Fine-Tuning Reinforcement Learning Reward Model Group Relative Policy Optimization Evaluations Evaluation Settings Standard Evaluation Open-Ended Evaluation DeepSeek-V3 as a Generative Reward Model Discussion Distillation from DeepSeek-R1 Self-Rewarding Multi-Token Prediction Evaluation Conclusion, Limitations, and Future Directions Contributions and Acknowledgments Ablation Studies for Low-Precision Training FP8 v.s. BF16 Training Discussion About Block-Wise Quantization Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models