Skip to content

Transfer learning broken for models trained before #182 #249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
icedoom888 opened this issue Apr 10, 2025 · 6 comments
Open

Transfer learning broken for models trained before #182 #249

icedoom888 opened this issue Apr 10, 2025 · 6 comments
Labels
bug Something isn't working

Comments

@icedoom888
Copy link
Contributor

What happened?

With the modification of the attention mechanism and layers introduced in #182.
Transfer leanring from previous checkpoints is broken:
node_dst_mlp weights and biases shapes are incompatible and will not be loaded from the pretrained checkpoints

What are the steps to reproduce the bug?

Run transfer leanring from any checkpoint trained on versions before the merge of #182.

Version

graphs: v0.5.0

Platform (OS and architecture)

Linux balfrin-ln002 5.14.21-150400.24.81_12.0.87-cray_shasta_c #1 SMP Sun Dec 17 12:59:08 UTC 2023 (e30c7c1) x86_64 x86_64 x86_64 GNU/Linux

Relevant log output

[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.encoder.proc.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([163842, 4])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([184430, 4])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.encoder.proc.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([184430, 4])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.encoder.proc.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.encoder.proc.node_dst_mlp.0.bias
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.encoder.proc.node_dst_mlp.0.bias
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.encoder.proc.node_dst_mlp.0.bias
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.0.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.0.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.0.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.0.node_dst_mlp.0.bias
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.0.node_dst_mlp.0.bias
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.0.node_dst_mlp.0.bias
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.1.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.1.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.1.node_dst_mlp.0.weight
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.1.node_dst_mlp.0.bias
[2025-04-10 11:44:08,002][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.1.node_dst_mlp.0.bias
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.1.node_dst_mlp.0.bias
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.2.node_dst_mlp.0.weight
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.2.node_dst_mlp.0.weight
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.2.node_dst_mlp.0.weight
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.2.node_dst_mlp.0.bias
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.2.node_dst_mlp.0.bias
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.2.node_dst_mlp.0.bias
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.node_attributes.latlons_data
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.3.node_dst_mlp.0.weight
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.3.node_dst_mlp.0.weight
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.3.node_dst_mlp.0.weight
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([542080, 4])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([730653, 4])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.node_attributes.latlons_hidden
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.3.node_dst_mlp.0.bias
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.3.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([163842, 4])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([184430, 4])
[2025-04-10 11:44:08,003][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.3.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.4.node_dst_mlp.0.weight
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.encoder.proc.node_dst_mlp.0.weight
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.encoder.proc.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.4.node_dst_mlp.0.weight
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.4.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.4.node_dst_mlp.0.weight
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.4.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.4.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.0.node_dst_mlp.0.weight
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.5.node_dst_mlp.0.weight
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.5.node_dst_mlp.0.weight
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.5.node_dst_mlp.0.weight
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.5.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.5.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.0.node_dst_mlp.0.bias
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,004][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.5.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.6.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.6.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.1.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.6.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.6.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.6.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.1.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.6.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.7.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.7.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.2.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.7.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.7.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.7.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.7.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.2.node_dst_mlp.0.bias
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.0.node_dst_mlp.0.weight
[2025-04-10 11:44:08,005][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.0.node_dst_mlp.0.weight
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.0.node_dst_mlp.0.weight
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.3.node_dst_mlp.0.weight
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.0.node_dst_mlp.0.bias
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.0.node_dst_mlp.0.bias
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.3.node_dst_mlp.0.bias
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.0.node_dst_mlp.0.bias
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.1.node_dst_mlp.0.weight
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.4.node_dst_mlp.0.weight
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.1.node_dst_mlp.0.weight
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.1.node_dst_mlp.0.weight
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.1.node_dst_mlp.0.bias
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.4.node_dst_mlp.0.bias
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.1.node_dst_mlp.0.bias
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.1.node_dst_mlp.0.bias
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,006][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.2.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.5.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.2.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.2.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.5.node_dst_mlp.0.bias
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.2.node_dst_mlp.0.bias
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.2.node_dst_mlp.0.bias
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.2.node_dst_mlp.0.bias
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.3.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.6.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.3.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.3.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.3.node_dst_mlp.0.bias
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.6.node_dst_mlp.0.bias
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.3.node_dst_mlp.0.bias
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.3.node_dst_mlp.0.bias
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.4.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.7.node_dst_mlp.0.weight
[2025-04-10 11:44:08,007][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.4.node_dst_mlp.0.weight
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.4.node_dst_mlp.0.weight
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.4.node_dst_mlp.0.bias
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.0.blocks.7.node_dst_mlp.0.bias
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.4.node_dst_mlp.0.bias
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.4.node_dst_mlp.0.bias
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.5.node_dst_mlp.0.weight
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.0.node_dst_mlp.0.weight
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.5.node_dst_mlp.0.weight
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.5.node_dst_mlp.0.weight
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.5.node_dst_mlp.0.bias
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.0.node_dst_mlp.0.bias
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.5.node_dst_mlp.0.bias
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.5.node_dst_mlp.0.bias
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.6.node_dst_mlp.0.weight
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.6.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,008][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.6.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.6.node_dst_mlp.0.bias
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.1.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.1.node_dst_mlp.0.bias
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.6.node_dst_mlp.0.bias
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.6.node_dst_mlp.0.bias
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.7.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.2.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.7.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.7.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.7.node_dst_mlp.0.bias
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.2.node_dst_mlp.0.bias
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.7.node_dst_mlp.0.bias
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.7.node_dst_mlp.0.bias
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.decoder.proc.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.3.node_dst_mlp.0.weight
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.decoder.proc.node_dst_mlp.0.weight
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.decoder.proc.node_dst_mlp.0.weight
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,009][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.decoder.proc.node_dst_mlp.0.bias
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.3.node_dst_mlp.0.bias
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.decoder.proc.node_dst_mlp.0.bias
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.decoder.proc.node_dst_mlp.0.bias
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: loss.node_weights
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([542080])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: loss.node_weights
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([542080])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: loss.node_weights
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([542080])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([730653])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: metrics.0.node_weights
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.4.node_dst_mlp.0.weight
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([730653])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: metrics.0.node_weights
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([542080])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([730653])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: metrics.0.node_weights
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([542080])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.4.node_dst_mlp.0.bias
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([730653])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([730653])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([542080])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([730653])
[2025-04-10 11:44:08,010][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.5.node_dst_mlp.0.weight
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.5.node_dst_mlp.0.bias
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.6.node_dst_mlp.0.weight
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.6.node_dst_mlp.0.bias
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.7.node_dst_mlp.0.weight
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,011][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.processor.proc.1.blocks.7.node_dst_mlp.0.bias
[2025-04-10 11:44:08,012][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,012][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096])
[2025-04-10 11:44:08,012][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.decoder.proc.node_dst_mlp.0.weight
[2025-04-10 11:44:08,012][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])
[2025-04-10 11:44:08,012][anemoi.training.utils.checkpoint][INFO] - Model shape: torch.Size([4096, 1024])
[2025-04-10 11:44:08,012][anemoi.training.utils.checkpoint][INFO] - Skipping loading parameter: model.model.decoder.proc.node_dst_mlp.0.bias
[2025-04-10 11:44:08,012][anemoi.training.utils.checkpoint][INFO] - Checkpoint shape: torch.Size([1024])

Accompanying data

No response

Organisation

No response
@icedoom888 icedoom888 added the bug Something isn't working label Apr 10, 2025
@mchantry
Copy link
Member

Hi @icedoom888 thanks for opening the issue.
Can I clarify that this occurs when attempting to transfer learn on a model first training pre #182 using the latest version of main?

@icedoom888
Copy link
Contributor Author

icedoom888 commented Apr 22, 2025 •
edited
Loading

Hey @mchantry, yes.
In #182 there is a new structure for the MLP that handles node distance (edge features) that is different from before.
You can see how those weights have different shapes in the log above.
If we load an old checkpoint (trained before #182), all weights related to node distance will not be initialised correctly and will have random values, effectively breaking the transfer learning functionality.

@einrone
Copy link
Contributor

einrone commented Apr 29, 2025

I had some similar problems with multi-domain setup when layer_kernels where introduced. Layers such as layer_norm1 was renamed to layer_norm_attention_src. So using the original transfer_learning_loading function, these layer were not detected, hence random weights. So I added a "patch" to fix this issue. So I introduced mapping_weights. In the config you specify :

mapping_weights:
  new_name: old_name

in my case I had to map this:

mapping_weights: 
    layer_norm_attention_src: layer_norm1
    layer_norm_attention_dest: layer_norm2
    layer_norm_attention: layer_norm1
    layer_norm_mlp: layer_norm2

which works fine, but not an ideal solution if there is alot of layers missing or mismatch in names etc.. However would this patch be of interest? Would also love to see if this can be generalized in a way, happy to add some help if needed.

@dietervdb-meteo dietervdb-meteo mentioned this issue May 8, 2025
Open
@JesperDramsch
Copy link
Member

This is something to consider in #248. Possibly some renaming functionality, but this might be out of scope. Would love your input on the roadmap though as users of the transfer learning / model freezing capabilities.

@einrone
Copy link
Contributor

einrone commented May 13, 2025

Indeed, this is a more like a quick fix however quite keen on ideas to improve "my fix". However looking at #248 I think this would be more like a ideal solution. Especially if someone consider to perform model distillation. So this is great 💃

@JesperDramsch
Copy link
Member

Oh yeah, I mentioned in #309 that #248 is a long-term thing, so whatever you need to do to get unblocked in the meantime is good in my opinion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Anemoi-dev
Status: No status
Milestone
No milestone
Development

No branches or pull requests
4 participants
@JesperDramsch @icedoom888 @mchantry @einrone