Updating Add qwen3 (PR 2903) to use HF weights

[greenrazer]

Builds off of @maximizemaxwell 's PR, who wrote it before the weights were released.

This PR:

• Loads the weights
• Fixes modeling code
• Adds Multiple Qwen3 models to the qwen example.

Issues 2025/4/28

• Qwen3 doesn't work with qwen example code by default
• Qwen3 modeling code needs slight rework

[EricLBuehler]

@greenrazer looks like the MoE model has the same architecture as the dense one and the MoE block from the V2!

[greenrazer]

I couldn't test the MoE model completely because it couldn't run on my computer. So if anyone has a beefier rig that'd be great :).

Also the Model never terminates. I tried adding an EOS token to <im_end>. It's probably an issue with rotary embeddings like last time. I'll have to take a look later.

[EricLBuehler]

I couldn't test the MoE model completely because it couldn't run on my computer. So if anyone has a beefier rig that'd be great :).

Will test!

[EricLBuehler]

I couldn't test the MoE model completely because it couldn't run on my computer. So if anyone has a beefier rig that'd be great :).

Will test!

@greenrazer seems that it is indeed broken for the MoE, while the dense model roughly works. Let me know if there is any information you need.

[greenrazer]

I couldn't test the MoE model completely because it couldn't run on my computer. So if anyone has a beefier rig that'd be great :).

Will test!

@greenrazer seems that it is indeed broken for the MoE, while the dense model roughly works. Let me know if there is any information you need.

Thank you @EricLBuehler!

Since the MoE model isn’t currently working and I don’t have the resources to debug it efficiently, I’m planning to remove it from this PR and move it to a draft instead. That way, someone else can build on it if they’d like, or I can revisit it later, especially if smaller MoE variants are released.

Edit:

PR here #2934