Skip to content

Add Qwen3 MoE #2934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 31, 2025
Merged

Add Qwen3 MoE #2934

merged 5 commits into from
May 31, 2025

Conversation

greenrazer
Copy link
Collaborator

@greenrazer greenrazer commented Apr 30, 2025

This PR implements the Qwen3 Mixture of Experts models (like Qwen3-30B-A3B)

Doesn't work, depends on #2930 (Merged)

@greenrazer
Copy link
Collaborator Author

Tested this on a Mac with more memory.

@greenrazer greenrazer marked this pull request as ready for review May 5, 2025 23:19
@greenrazer
Copy link
Collaborator Author

This PR is ready for review. @LaurentMazare

Copy link
Collaborator

@LaurentMazare LaurentMazare left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for adding this. Would you mind just adding an example of how to call it in the readme of the example (as well as the generated output).

@LaurentMazare
Copy link
Collaborator

Thanks!

@LaurentMazare LaurentMazare merged commit 0224a74 into huggingface:main May 31, 2025
4 of 9 checks passed
@greenrazer greenrazer deleted the add-qwen-moe branch June 4, 2025 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants