Code

MTF dataset and packing#293

Code

Merged

thomasw21 merged 183 commits intomainbigscience-workshop/Megatron-DeepSpeed:mainfrom thomas/mtf_dataset_and_packingbigscience-workshop/Megatron-DeepSpeed:thomas/mtf_dataset_and_packingCopy head branch name to clipboardJul 2, 2022

+693-15

Commits

Commits on Jun 29, 2022

• added train script but with prefix manually declared

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  1fd8099

  Copy full SHA for 1fd8099

• made new dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  fe3f8c2

  Copy full SHA for fe3f8c2

• minor adjustments

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  31f2087

  Copy full SHA for 31f2087

• added capabilities for padding and prefix lm index

  

  lintangsutawika

  authored and

  thomasw21

  committed
  6ccacba

  Copy full SHA for 6ccacba

• added finetune script

  

  lintangsutawika

  authored and

  thomasw21

  committed
  155a8ef

  Copy full SHA for 155a8ef

• removed script

  

  lintangsutawika

  authored and

  thomasw21

  committed
  ec6c07e

  Copy full SHA for ec6c07e

• added adjustments and new dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  435d65f

  Copy full SHA for 435d65f

• try mlm dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  b1d7bbd

  Copy full SHA for b1d7bbd

• minor changes

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  714e5b7

  Copy full SHA for 714e5b7

• minor addition of import packages

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  dc15436

  Copy full SHA for dc15436

• minor error fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  e79ac16

  Copy full SHA for e79ac16

• minor error fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  9a90a2e

  Copy full SHA for 9a90a2e

• samples follow how gpt dataset is loaded

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  7e79b48

  Copy full SHA for 7e79b48

• added masked_lm_prob

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  3453dbd

  Copy full SHA for 3453dbd

• added mask id

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  d382d19

  Copy full SHA for d382d19

• added mask id

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  31fbf55

  Copy full SHA for 31fbf55

• added mask id

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  f571132

  Copy full SHA for f571132

• added mask id

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  5548a47

  Copy full SHA for 5548a47

• added fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  21237f3

  Copy full SHA for 21237f3

• added bos and eos token id

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  98c4635

  Copy full SHA for 98c4635

• no need for sentinal token

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  e1a75aa

  Copy full SHA for e1a75aa

• add aux functions

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  2fdd795

  Copy full SHA for 2fdd795

• add aux functions

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  154f39c

  Copy full SHA for 154f39c

• add aux functions

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  3765a81

  Copy full SHA for 3765a81

• add pad_id

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  2cd4174

  Copy full SHA for 2cd4174

• changed lm predictions to t5

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  b592ea3

  Copy full SHA for b592ea3

• changed lm predictions to t5

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  852faca

  Copy full SHA for 852faca

• changed lm predictions to t5

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  4333554

  Copy full SHA for 4333554

• changed lm predictions to t5

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  be73455

  Copy full SHA for be73455

• changed lm predictions to t5

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  163c966

  Copy full SHA for 163c966

• tokenizer add mask, cls, sep tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  56de89f

  Copy full SHA for 56de89f

• commit latest changes

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ca86fa8

  Copy full SHA for ca86fa8

• commit latest changes

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  5011d99

  Copy full SHA for 5011d99

• added sentinal tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  1b15263

  Copy full SHA for 1b15263

• added sentinal tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  1b16541

  Copy full SHA for 1b16541

• added sentinal tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  0603aac

  Copy full SHA for 0603aac

• added additional_special_tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  bd061d3

  Copy full SHA for bd061d3

• added additional_special_tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  aff88b9

  Copy full SHA for aff88b9

• check t5_input and output

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  aab4729

  Copy full SHA for aab4729

• check decoder in and decoder out

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  9448ef4

  Copy full SHA for 9448ef4

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  16ba4aa

  Copy full SHA for 16ba4aa

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  99ca9e8

  Copy full SHA for 99ca9e8

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  cdfecad

  Copy full SHA for cdfecad

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  e058688

  Copy full SHA for e058688

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  38ded72

  Copy full SHA for 38ded72

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  0f68be3

  Copy full SHA for 0f68be3

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  eb84844

  Copy full SHA for eb84844

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  a3af6bf

  Copy full SHA for a3af6bf

• made into input and output tokens

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  6ad61b6

  Copy full SHA for 6ad61b6

• added eos

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  9131fdd

  Copy full SHA for 9131fdd

• added eos

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  cb76cd3

  Copy full SHA for cb76cd3

• test text_token

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  531ee68

  Copy full SHA for 531ee68

• test text_token

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  a7d1158

  Copy full SHA for a7d1158

• test text_token

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  0008cfb

  Copy full SHA for 0008cfb

• test text_token

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  f1461a8

  Copy full SHA for f1461a8

• test text_token

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ada0f10

  Copy full SHA for ada0f10

• assigned array

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  298c9b7

  Copy full SHA for 298c9b7

• assigned array

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  d2bdff6

  Copy full SHA for d2bdff6

• assigned array

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  4ec8db3

  Copy full SHA for 4ec8db3

• hardcoded sequence length

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  10a2b6d

  Copy full SHA for 10a2b6d

• check again

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  a373a70

  Copy full SHA for a373a70

• show sentinal tokens

  

  lintangsutawika

  authored and

  thomasw21

  committed
  bdef71b

  Copy full SHA for bdef71b

• show sentinal tokens

  

  lintangsutawika

  authored and

  thomasw21

  committed
  262fd6c

  Copy full SHA for 262fd6c

• show sentinal tokens

  

  lintangsutawika

  authored and

  thomasw21

  committed
  68a6a93

  Copy full SHA for 68a6a93

• show sentinal tokens

  

  lintangsutawika

  authored and

  thomasw21

  committed
  1c00d4b

  Copy full SHA for 1c00d4b

• add more special tokens

  

  lintangsutawika

  authored and

  thomasw21

  committed
  8b85f11

  Copy full SHA for 8b85f11

• changed how mlm data is loaded

  

  lintangsutawika

  authored and

  thomasw21

  committed
  85d204a

  Copy full SHA for 85d204a

• changed how mlm data is loaded

  

  lintangsutawika

  authored and

  thomasw21

  committed
  4c84274

  Copy full SHA for 4c84274

• changed how mlm data is loaded

  

  lintangsutawika

  authored and

  thomasw21

  committed
  084245e

  Copy full SHA for 084245e

• changed how mlm data is loaded

  

  lintangsutawika

  authored and

  thomasw21

  committed
  32af10e

  Copy full SHA for 32af10e

• changed how mlm data is loaded

  

  lintangsutawika

  authored and

  thomasw21

  committed
  b6e0e63

  Copy full SHA for b6e0e63

• added new script

  

  lintangsutawika

  authored and

  thomasw21

  committed
  2af2e4b

  Copy full SHA for 2af2e4b

• added new script

  

  lintangsutawika

  authored and

  thomasw21

  committed
  cc5968e

  Copy full SHA for cc5968e

• added new script

  

  lintangsutawika

  authored and

  thomasw21

  committed
  cf0b2a0

  Copy full SHA for cf0b2a0

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  fc150a0

  Copy full SHA for fc150a0

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  039f90f

  Copy full SHA for 039f90f

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  7364781

  Copy full SHA for 7364781

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  5b1100a

  Copy full SHA for 5b1100a

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  45102a9

  Copy full SHA for 45102a9

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  7b2ebbf

  Copy full SHA for 7b2ebbf

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  fe8b3dc

  Copy full SHA for fe8b3dc

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  f456725

  Copy full SHA for f456725

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  ae73d8c

  Copy full SHA for ae73d8c

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  fae6a0b

  Copy full SHA for fae6a0b

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  8185842

  Copy full SHA for 8185842

• try t5 dataset

  

  lintangsutawika

  authored and

  thomasw21

  committed
  9deef49

  Copy full SHA for 9deef49

• developing

  

  lintangsutawika

  authored and

  thomasw21

  committed
  1e78a4b

  Copy full SHA for 1e78a4b

• developing

  

  lintangsutawika

  authored and

  thomasw21

  committed
  9070929

  Copy full SHA for 9070929

• developing

  

  lintangsutawika

  authored and

  thomasw21

  committed
  56c69de

  Copy full SHA for 56c69de

• developing

  

  lintangsutawika

  authored and

  thomasw21

  committed
  d1ca914

  Copy full SHA for d1ca914

• developing

  

  lintangsutawika

  authored and

  thomasw21

  committed
  13af623

  Copy full SHA for 13af623

• developing

  

  lintangsutawika

  authored and

  thomasw21

  committed
  dbc555e

  Copy full SHA for dbc555e

• developing

  

  lintangsutawika

  authored and

  thomasw21

  committed
  12b209d

  Copy full SHA for 12b209d

• test to see output of get_ltor_masks_and_position_ids

  

  lintangsutawika

  authored and

  thomasw21

  committed
  698eff0

  Copy full SHA for 698eff0

• test to see output of get_ltor_masks_and_position_ids

  

  lintangsutawika

  authored and

  thomasw21

  committed
  dae3cc6

  Copy full SHA for dae3cc6

• add new script

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  5c109c3

  Copy full SHA for 5c109c3

• add new script

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  2fc9995

  Copy full SHA for 2fc9995

• add new script

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ee7af99

  Copy full SHA for ee7af99

• changed settings

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  b6701a8

  Copy full SHA for b6701a8

• changed settings

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  2283e58

  Copy full SHA for 2283e58

• tidy up

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  9d00a49

  Copy full SHA for 9d00a49

• changed tokenizer and position embedding

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  0298fde

  Copy full SHA for 0298fde

• modifying mlm to reflect original implementation

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  bde07f0

  Copy full SHA for bde07f0

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  4c0ca2e

  Copy full SHA for 4c0ca2e

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  0c05596

  Copy full SHA for 0c05596

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  30f6924

  Copy full SHA for 30f6924

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  84408ef

  Copy full SHA for 84408ef

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ad964c5

  Copy full SHA for ad964c5

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  45899e9

  Copy full SHA for 45899e9

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  0b94597

  Copy full SHA for 0b94597

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  2b54cc1

  Copy full SHA for 2b54cc1

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ec61627

  Copy full SHA for ec61627

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  4448d1d

  Copy full SHA for 4448d1d

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ecd148c

  Copy full SHA for ecd148c

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  a99f30f

  Copy full SHA for a99f30f

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  62d3e3e

  Copy full SHA for 62d3e3e

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  a160853

  Copy full SHA for a160853

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  fe205f7

  Copy full SHA for fe205f7

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  d39bdaf

  Copy full SHA for d39bdaf

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  2530d3e

  Copy full SHA for 2530d3e

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  5e93c47

  Copy full SHA for 5e93c47

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ad86799

  Copy full SHA for ad86799

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  82c8d93

  Copy full SHA for 82c8d93

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ebf3561

  Copy full SHA for ebf3561

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  811f975

  Copy full SHA for 811f975

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  de7dfc8

  Copy full SHA for de7dfc8

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  be2af77

  Copy full SHA for be2af77

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  5e7e18f

  Copy full SHA for 5e7e18f

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  24d4f25

  Copy full SHA for 24d4f25

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  5926be1

  Copy full SHA for 5926be1

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  0f18174

  Copy full SHA for 0f18174

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  58ce714

  Copy full SHA for 58ce714

• set correct seq len

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  05470d7

  Copy full SHA for 05470d7

• refined sampling method

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  51a23f2

  Copy full SHA for 51a23f2

• refined sampling method

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  43cb2f0

  Copy full SHA for 43cb2f0

• refined sampling method

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  901defc

  Copy full SHA for 901defc

• refined sampling method

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  3130d7d

  Copy full SHA for 3130d7d

• refined sampling method

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  18eb53d

  Copy full SHA for 18eb53d

• refined sampling method

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  652c545

  Copy full SHA for 652c545

• first commit, adding non causal mlm dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  5a49db8

  Copy full SHA for 5a49db8

• fixed mlm dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  81b918c

  Copy full SHA for 81b918c

• fixed mlm dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  95afc4f

  Copy full SHA for 95afc4f

• fixed mlm dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  c4514d8

  Copy full SHA for c4514d8

• fixed mlm dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  5cca5af

  Copy full SHA for 5cca5af

• fixed mlm dataset

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  ae95878

  Copy full SHA for ae95878

• minor changes

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  a03e59f

  Copy full SHA for a03e59f

• removed mlm related scripts

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  fa1e072

  Copy full SHA for fa1e072

• removed any scipts not related to dataset, revert arguments

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  e3ce0a7

  Copy full SHA for e3ce0a7

• added sampler and test

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  87e4055

  Copy full SHA for 87e4055

• added testing data

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  0ae7661

  Copy full SHA for 0ae7661

• adapted test loader

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  71fb5ae

  Copy full SHA for 71fb5ae

• Update megatron/data/non_causal_mtf_dataset.py

  Show description for be0cea2
  

  Lintang Sutawika

  and

  thomasw21

  committed
  be0cea2

  Copy full SHA for be0cea2

• removed unused files

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  9daa376

  Copy full SHA for 9daa376

• changed with impossible token

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  6b9e81a

  Copy full SHA for 6b9e81a

• enable loading multiple indexed_dataset for each field

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  7feec27

  Copy full SHA for 7feec27

• minor fix

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  f84f293

  Copy full SHA for f84f293

• data_prefix is set as dict

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  2778d8d

  Copy full SHA for 2778d8d

• removed sample_idx lines

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  61ac4b9

  Copy full SHA for 61ac4b9

• change line from sample_idx to doc_idx

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  62e3fb1

  Copy full SHA for 62e3fb1

• replace shuffling _build_index_mappings with random.sample of the doc_idx

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  cb79f09

  Copy full SHA for cb79f09

• minor changes

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  e9cf22a

  Copy full SHA for e9cf22a

• Cleanup artefacts

  

  Muennighoff

  authored and

  thomasw21

  committed
  acd87cd

  Copy full SHA for acd87cd

• Add packed preprocessing

  

  Muennighoff

  authored and

  thomasw21

  committed
  019ed7c

  Copy full SHA for 019ed7c

• Use seq_length arg

  

  Muennighoff

  authored and

  thomasw21

  committed
  7619f7a

  Copy full SHA for 7619f7a

• Add sources & docstrings

  

  Muennighoff

  authored and

  thomasw21

  committed
  219209a

  Copy full SHA for 219209a

• added training process for t0

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  67424d6

  Copy full SHA for 67424d6

• Update pretrain_t0.py

  

  Lintang Sutawika

  authored and

  thomasw21

  committed
  a7c424e

  Copy full SHA for a7c424e

• Remove a bunch of code that's not needed

  thomasw21

  committed
  51d6c40

  Copy full SHA for 51d6c40

Commits on Jun 30, 2022

• WIP

  thomasw21

  committed
  b4e374c

  Copy full SHA for b4e374c

• Cleanup

  thomasw21

  committed
  0d2fdfd

  Copy full SHA for 0d2fdfd

• Add back all configs

  thomasw21

  committed
  126fa34

  Copy full SHA for 126fa34

• Woops

  thomasw21

  committed
  83d2405

  Copy full SHA for 83d2405

• Fix tests

  thomasw21

  committed
  c93ed5c

  Copy full SHA for c93ed5c

• Rename testing files

  thomasw21

  committed
  528f5d3

  Copy full SHA for 528f5d3

• Do in-place operations

  thomasw21

  committed
  8bed302

  Copy full SHA for 8bed302

• Do in-place operations

  thomasw21

  committed
  bd2fede

  Copy full SHA for bd2fede

• Woops

  thomasw21

  committed
  8593e42

  Copy full SHA for 8593e42

• Fix typo

  thomasw21

  committed
  a1eb558

  Copy full SHA for a1eb558

• Add test that packing is done optimially via greedy algorithm

  thomasw21

  committed
  3bddafa

  Copy full SHA for 3bddafa

• Woops

  thomasw21

  committed
  45c9444

  Copy full SHA for 45c9444

• Update tests/test_dataloaders.py

  Show description for c74dbb7
  

  thomasw21

  and

  Muennighoff

  authored
  c74dbb7

  Copy full SHA for c74dbb7

• Last segment is either pad or another document that fit perfectly

  thomasw21

  committed
  a337563

  Copy full SHA for a337563

Commits on Jul 2, 2022

• Resolve conflict

  thomasw21

  committed
  6227232

  Copy full SHA for 6227232