训练deepseek蒸馏的7B时，loss在每个epoch开始时翻倍

[Y56611]

Reminder

• I have read the above rules and searched the existing issues.

System Info

[2025-03-07 09:01:46,835] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)

• llamafactory version: 0.9.2.dev0
• Platform: Linux-5.4.119-19-0009.11-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version: 2.3.0a0+ebedce2 (GPU)
• Transformers version: 4.49.0
• Datasets version: 3.2.0
• Accelerate version: 1.2.1
• PEFT version: 0.12.0
• TRL version: 0.9.6
• GPU type: NVIDIA A800-SXM4-80GB
• GPU number: 8
• GPU memory: 79.33GB
• DeepSpeed version: 0.16.2

Reproduction

Put your message here.

Others

No response

[Y56611]

{'loss': 0.7723, 'grad_norm': 0.3804401225661987, 'learning_rate': 3.7866236009990253e-06, 'epoch': 1.97}
{'loss': 0.7755, 'grad_norm': 0.3685096806372812, 'learning_rate': 3.77567508265225e-06, 'epoch': 1.98}
{'loss': 0.7701, 'grad_norm': 0.4337734736678332, 'learning_rate': 3.764693384223671e-06, 'epoch': 1.99}
{'loss': 0.7858, 'grad_norm': 0.4056827759511268, 'learning_rate': 3.753678791345311e-06, 'epoch': 1.99}
{'loss': 1.5318, 'grad_norm': 0.6894233935022746, 'learning_rate': 3.7426315905047694e-06, 'epoch': 2.01}
{'loss': 0.755, 'grad_norm': 0.4109267906585638, 'learning_rate': 3.731552069037774e-06, 'epoch': 2.01}
{'loss': 0.7521, 'grad_norm': 0.3946745772581362, 'learning_rate': 3.7204405151207036e-06, 'epoch': 2.02}
{'loss': 0.7519, 'grad_norm': 0.38307276099678733, 'learning_rate': 3.7092972177630998e-06, 'epoch': 2.03}
{'loss': 0.765, 'grad_norm': 0.4016667872509622, 'learning_rate': 3.6981224668001427e-06, 'epoch': 2.04}
{'loss': 0.7556, 'grad_norm': 0.3905259113566424, 'learning_rate': 3.686916552885115e-06, 'epoch': 2.04}
log的片段