Skip to content

Add support to convert deepseek-r1 from fp8 to bf16 on CPU #1629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 25, 2025

Conversation

Lumosis
Copy link
Collaborator

@Lumosis Lumosis commented Apr 24, 2025

Description

Converts weights from FP8 to BF16 for a HF model on CPU.

The script is mainly ported from https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py. The original script is GPU-based.

Example cmd:

python3 -m MaxText.fp8_to_bf16 --input-fp8-hf-path <path/to/fp8/ckpt>
--output-bf16-hf-path <local/path/to/save/new/bf16/ckpt>

For deepseek-R1, the conversion process takes 1~2 hrs and the memory usage is below ~25 GB.

Tests

I compared the model files generated using my script with those generated using the original script by running diff command:

root@ranlihao-m1-ultramem-160:/mnt/disks/persist# diff models/deepseek_bf16 models/deepseek_bf16_ref/hf-r1-671b-bf16/

root@ranlihao-m1-ultramem-160:/mnt/disks/persist# 

There is no difference.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed.

Copy link
Collaborator

@gagika gagika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few comments

@Lumosis Lumosis force-pushed the lihao/fp8_to_bf16 branch from 0b58367 to e6a07ef Compare April 24, 2025 23:07
Copy link
Collaborator

@gagika gagika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@Lumosis Lumosis force-pushed the lihao/fp8_to_bf16 branch from e6a07ef to ed5dfbc Compare April 25, 2025 22:09
@copybara-service copybara-service bot merged commit 2ca55ae into main Apr 25, 2025
13 checks passed
@copybara-service copybara-service bot deleted the lihao/fp8_to_bf16 branch April 25, 2025 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants