Code

Complete ESM2 pretraining#112

Code

Merged

sichu2023 merged 93 commits intov2-mainNVIDIA/bionemo-framework:v2-mainfrom

esm2-pretrainNVIDIA/bionemo-framework:esm2-pretrainCopy head branch name to clipboard

Aug 29, 2024

Commits

Commits on Aug 29, 2024

• mount data

  

  farhadrgh

  authored and

  sichu2023

  committed
  692079a

  Copy full SHA for 692079a

• scale quary before rope

  

  farhadrgh

  authored and

  sichu2023

  committed
  532a6fa

  Copy full SHA for 532a6fa

• scale quary before rope

  

  farhadrgh

  authored and

  sichu2023

  committed
  71f6427

  Copy full SHA for 71f6427

• ruff

  Show description for 0f423be
  

  farhadrgh

  authored and

  sichu2023

  committed
  0f423be

  Copy full SHA for 0f423be

• doc str

  Show description for eb2926e
  

  farhadrgh

  authored and

  sichu2023

  committed
  eb2926e

  Copy full SHA for eb2926e

• logits test

  Show description for 34ab0f3
  

  farhadrgh

  authored and

  sichu2023

  committed
  34ab0f3

  Copy full SHA for 34ab0f3

• add embedding and logits tests

  Show description for 68493d5
  

  farhadrgh

  authored and

  sichu2023

  committed
  68493d5

  Copy full SHA for 68493d5

• move tests to test_model

  Show description for b4693d7
  

  farhadrgh

  authored and

  sichu2023

  committed
  b4693d7

  Copy full SHA for b4693d7

• ESM2 bionemo1 options

  Show description for a317e16
  

  farhadrgh

  authored and

  sichu2023

  committed
  a317e16

  Copy full SHA for a317e16

• move doc str

  

  farhadrgh

  authored and

  sichu2023

  committed
  74dab95

  Copy full SHA for 74dab95

• ruff

  

  farhadrgh

  authored and

  sichu2023

  committed
  54c5236

  Copy full SHA for 54c5236

• fix typo

  

  farhadrgh

  authored and

  sichu2023

  committed
  c4104b0

  Copy full SHA for c4104b0

• move fixtures up

  

  farhadrgh

  authored and

  sichu2023

  committed
  5dcf303

  Copy full SHA for 5dcf303

• test loss gv

  

  farhadrgh

  authored and

  sichu2023

  committed
  f2f4bfa

  Copy full SHA for f2f4bfa

• add ESM2 pretrain script

  

  farhadrgh

  authored and

  sichu2023

  committed
  a2cc3af

  Copy full SHA for a2cc3af

• add doc str

  

  farhadrgh

  authored and

  sichu2023

  committed
  54e05d3

  Copy full SHA for 54e05d3

• update pretrain test

  

  farhadrgh

  authored and

  sichu2023

  committed
  bd6e264

  Copy full SHA for bd6e264

• add device

  

  farhadrgh

  authored and

  sichu2023

  committed
  c582abb

  Copy full SHA for c582abb

• rm async_save

  

  farhadrgh

  authored and

  sichu2023

  committed
  96c6ba2

  Copy full SHA for 96c6ba2

• remove comments

  

  farhadrgh

  authored and

  sichu2023

  committed
  dcb9f64

  Copy full SHA for dcb9f64

• change atol

  

  farhadrgh

  authored and

  sichu2023

  committed
  a8a7005

  Copy full SHA for a8a7005

• skip pytest

  

  farhadrgh

  authored and

  sichu2023

  committed
  0022523

  Copy full SHA for 0022523

• change atol

  

  farhadrgh

  authored and

  sichu2023

  committed
  c0a7890

  Copy full SHA for c0a7890

• rename to avoid ci error

  

  farhadrgh

  authored and

  sichu2023

  committed
  97b13a2

  Copy full SHA for 97b13a2

• fix import

  

  farhadrgh

  authored and

  sichu2023

  committed
  12422bd

  Copy full SHA for 12422bd

• test with dummy data

  

  farhadrgh

  authored and

  sichu2023

  committed
  20cef07

  Copy full SHA for 20cef07

• skip pretrain test

  

  farhadrgh

  authored and

  sichu2023

  committed
  4d811b0

  Copy full SHA for 4d811b0

• compute hf reference

  

  farhadrgh

  authored and

  sichu2023

  committed
  dc64871

  Copy full SHA for dc64871

• update hf reference

  

  farhadrgh

  authored and

  sichu2023

  committed
  41e2cb0

  Copy full SHA for 41e2cb0

• update

  

  farhadrgh

  authored and

  sichu2023

  committed
  e97f40d

  Copy full SHA for e97f40d

• rm skip

  

  farhadrgh

  authored and

  sichu2023

  committed
  909b0fd

  Copy full SHA for 909b0fd

• tokenizer serde

  

  farhadrgh

  authored and

  sichu2023

  committed
  990de8c

  Copy full SHA for 990de8c

• fix typo

  

  farhadrgh

  authored and

  sichu2023

  committed
  b77a4dc

  Copy full SHA for b77a4dc

• fix tests

  

  farhadrgh

  authored and

  sichu2023

  committed
  b9a0528

  Copy full SHA for b9a0528

• doc str

  

  farhadrgh

  authored and

  sichu2023

  committed
  0b2869e

  Copy full SHA for 0b2869e

• skip pretrain tests

  

  farhadrgh

  authored and

  sichu2023

  committed
  5da634d

  Copy full SHA for 5da634d

• update doc str

  

  farhadrgh

  authored and

  sichu2023

  committed
  b27e34b

  Copy full SHA for b27e34b

• doc str

  Show description for 12c6839
  

  farhadrgh

  authored and

  sichu2023

  committed
  12c6839

  Copy full SHA for 12c6839

• Add ESM2 Dataset and Datamodule

  Show description for ee61014
  

  pstjohn

  authored and

  sichu2023

  committed
  ee61014

  Copy full SHA for ee61014

• duplicate

  

  farhadrgh

  authored and

  sichu2023

  committed
  908ce03

  Copy full SHA for 908ce03

• add accumulate_grad_batches

  sichu2023

  committed
  ffafbe2

  Copy full SHA for ffafbe2

• increase num-dataset-workers for persistent_workers

  sichu2023

  committed
  60ba685

  Copy full SHA for 60ba685

• add esm2 hparam to argparse

  sichu2023

  committed
  8342dd8

  Copy full SHA for 8342dd8

• fix typo

  sichu2023

  committed
  2935a95

  Copy full SHA for 2935a95

• fix experiment name

  sichu2023

  committed
  0c52a0e

  Copy full SHA for 0c52a0e

• fix _random_crop

  sichu2023

  committed
  9d6f5a3

  Copy full SHA for 9d6f5a3

• fix typo

  sichu2023

  committed
  0461742

  Copy full SHA for 0461742

• disable cyclic sampler

  sichu2023

  committed
  bddf4e2

  Copy full SHA for bddf4e2

• add train and val dataset shuffling

  sichu2023

  committed
  683f1ea

  Copy full SHA for 683f1ea

• uncomment pretrain tests

  sichu2023

  committed
  abfd521

  Copy full SHA for abfd521

• support fractional value of limit_val_batches

  sichu2023

  committed
  2bc8bdc

  Copy full SHA for 2bc8bdc

• revert change on .devcontainer/devcontainer.json

  sichu2023

  committed
  2b44822

  Copy full SHA for 2b44822

• add fractional limit_val_batches testing

  sichu2023

  committed
  76f6c65

  Copy full SHA for 76f6c65

• fix ruff

  sichu2023

  committed
  f1d83f4

  Copy full SHA for f1d83f4

• update MegatronMixedPrecision

  sichu2023

  committed
  3db6434

  Copy full SHA for 3db6434

• update and skip fractional limit_val_batches main test

  sichu2023

  committed
  2d7a36b

  Copy full SHA for 2d7a36b

• update to skip io.track_io

  sichu2023

  committed
  0d77dff

  Copy full SHA for 0d77dff

• parameterize limit_val_batches main testwq

  sichu2023

  committed
  4b11b6a

  Copy full SHA for 4b11b6a

• wrap main with model parallel state in test

  sichu2023

  committed
  eb9e64f

  Copy full SHA for eb9e64f

• remove assertion

  sichu2023

  committed
  a5616ce

  Copy full SHA for a5616ce

• clean up callbacks

  sichu2023

  committed
  3a456d6

  Copy full SHA for 3a456d6

• move pretrain cli cwd to tempdir

  sichu2023

  committed
  fc32a90

  Copy full SHA for fc32a90

• ruff

  sichu2023

  committed
  6da31eb

  Copy full SHA for 6da31eb

• clean up ppl logging callback

  sichu2023

  committed
  bdb9567

  Copy full SHA for bdb9567

• share gbs calculation to geneformer

  sichu2023

  committed
  da2beef

  Copy full SHA for da2beef

• clean up geneformer test_pretrain.py

  sichu2023

  committed
  752ae55

  Copy full SHA for 752ae55

• limit main testing to argparse

  sichu2023

  committed
  d0ab66e

  Copy full SHA for d0ab66e

• sync testing changes on geneformer

  sichu2023

  committed
  be0f114

  Copy full SHA for be0f114

• support limit_val_batches = None

  sichu2023

  committed
  18dcb67

  Copy full SHA for 18dcb67

• add unittest to ensure consistant samples per epoch

  sichu2023

  committed
  2a2ce08

  Copy full SHA for 2a2ce08

• add unittest on limit_val_batches = None

  sichu2023

  committed
  1867704

  Copy full SHA for 1867704

• add unittest on float_or_int_or_none

  sichu2023

  committed
  8503b78

  Copy full SHA for 8503b78

• add infer_global_batch_size unittest

  sichu2023

  committed
  09c42e7

  Copy full SHA for 09c42e7

• ruff

  sichu2023

  committed
  6974d28

  Copy full SHA for 6974d28

• ruff

  sichu2023

  committed
  e675f94

  Copy full SHA for e675f94

• ruff

  sichu2023

  committed
  468697a

  Copy full SHA for 468697a

• fix dict order in tensor_dict_hash

  sichu2023

  committed
  1cdf113

  Copy full SHA for 1cdf113

• remove eval_iters from num_val_samples

  sichu2023

  committed
  4923056

  Copy full SHA for 4923056

• remove comment

  sichu2023

  committed
  72cdab8

  Copy full SHA for 72cdab8

• expand test_limit_val_batches_is_none

  sichu2023

  committed
  394582d

  Copy full SHA for 394582d

• fix unittest and add limit_val_samples = 0 edge case

  sichu2023

  committed
  b70b76b

  Copy full SHA for b70b76b

• Fix unittest and add limit_val_batches = 0 in esm

  sichu2023

  committed
  bb6ef40

  Copy full SHA for bb6ef40

• rename PRNGDatasetShuffler

  sichu2023

  committed
  2653033

  Copy full SHA for 2653033

• drop total_samples in create_valid_dataset

  sichu2023

  committed
  ab679a9

  Copy full SHA for ab679a9

• drop total_samples from create_train_dataset

  sichu2023

  committed
  bfc4769

  Copy full SHA for bfc4769

• ruff and extract infer_num_val_samples

  sichu2023

  committed
  cd37c0f

  Copy full SHA for cd37c0f

• transfer infer_num_samples to geneformer

  sichu2023

  committed
  ea7240f

  Copy full SHA for ea7240f

• ruff

  sichu2023

  committed
  ceac497

  Copy full SHA for ceac497

• use hosted tokenizer files

  sichu2023

  committed
  aaf3405

  Copy full SHA for aaf3405

• add back num_samples in esm dataset

  sichu2023

  committed
  8669a7d

  Copy full SHA for 8669a7d

• ruff

  sichu2023

  committed
  740f2b6

  Copy full SHA for 740f2b6

• fix valid shuffling

  sichu2023

  committed
  8c0daad

  Copy full SHA for 8c0daad

• num_train_samples in dataset shuffling

  sichu2023

  committed
  aaabf34

  Copy full SHA for aaabf34