Qwen2 MoE manual head_dim

[yunju63]

Feature request

transformers/src/transformers/models/qwen2_moe/modeling_qwen2_moe.py

Line 317 in 81aa9b2

self.head_dim = self.hidden_size // self.num_heads

For qwen2 moe, head_dim is now forced to be hidden_size // num_heads.

Motivation

manual head_dim setting support in llama, mistal, mixtral modeling

Your contribution

PR

[Rocketknight1]

hi @yunju63, won't the reshape in the attention layer fail if we set self.head_dim to any value other than self.hidden_size // self.num_heads?