Discussions

• 1 You must be logged in to vote

  🙏

  [QUESTION] Will the data get re-shuffled if the sequence length is modified during training?

  SefaZeng asked Aug 13, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION] Does it support Knowledge Distillation?

  mushan09 asked Aug 9, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  [QUESTION] Why and When dose matmul call different kernels？

  hxdtest asked Aug 12, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  [QUESTION] Sample idx, bin files in public domain for trying out pretrain_gpt.py?

  sambar1729 asked Jun 26, 2024 in Q&A · Unanswered

  

  

  

  

  2
• 1 You must be logged in to vote

  🙏

  [QUESTION] Does Megatron-LM supports Flash Attention for BERT and T5 Pretraining?

  Leo-T-Zang asked Jul 2, 2024 in Q&A · Unanswered

  

  

  

  

  3
• 1 You must be logged in to vote

  🙏

  [QUESTION]Splitting large document and bucketing

  shafiqabedin asked Aug 7, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION] Megatron-LM DistributedOptimizer or NeMo MegatronDistributedFusedAdam Optimizer?

  TJ-Solergibert asked Aug 6, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION] Checkpoint storage format

  syx11237744 asked Aug 6, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION]

  suzewei asked Aug 6, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Is "Bucketization" an option for training

  shafiqabedin asked Aug 1, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION] Question about resume with distributed optimizer

  stale No activity in 60 days on issue or PR

  WailordHe asked Jun 1, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  [QUESTION] Can fp8 and pipeline parallelism be used together?

  exnx asked Jul 9, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  [QUESTION] Why is reset_attention_mask=False by default?

  dtamayo-nlp asked Jul 26, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION] One possible typo in docs/source/distrib_optimizer.md

  wplf asked Jul 26, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Differnt Tokenizer

  dustinwloring1988 asked Jul 25, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION] About Optimizer & Params Offload

  shh2000 asked Jul 24, 2024 in Q&A · Unanswered

  

  

  1
• 2 You must be logged in to vote

  🙏

  [QUESTION] Calculations regarding calculate_per_token_loss parameter

  clarence-lee-sheng asked Jul 19, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION] Has standalone_embedding_stage been supported yet in core?

  JiwenJ asked Jun 26, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  [QUESTION] add_position_embedding=False in checkpoint_args during Llama3 8B training

  NEU-rzh asked Jul 17, 2024 in Q&A · Unanswered

  

  

  

  

  2
• 1 You must be logged in to vote

  🙏

  [QUESTION] Why not use tensor parallel APIs of pytorch

  stale No activity in 60 days on issue or PR

  GuWei007 asked May 16, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  Question with forward_backward_pipelining_without_interleaving in Megatron-LM Pipeline

  stale No activity in 60 days on issue or PR

  Hongjie1Chu asked May 17, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  [QUESTION] how to profile bubble time in pipeline parallelism?

  stale No activity in 60 days on issue or PR

  starstream asked May 15, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  [QUESTION] How does tensor_parallel coop with q/k_layernorm

  stale No activity in 60 days on issue or PR

  cryoco asked May 10, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  [QUESTION]

  woson asked Jul 8, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  function missing

  ywb2018 asked Jul 8, 2024 in Q&A · Unanswered

  

  

  0