Code

Implementing RWKV-LLM (#37)#209

Code

Draft

AvidEslami wants to merge 64 commits intoOpenAdaptAI:mainOpenAdaptAI/OpenAdapt:mainfrom AvidEslami:RWKVAvidEslami/puterbot:RWKVCopy head branch name to clipboard

+763

Commits

Commits on May 24, 2023

• test RWKV

  AvidEslami

  committed
  178f0d2

  Copy full SHA for 178f0d2

Commits on May 26, 2023

• switch to using modal

  AvidEslami

  committed
  863b250

  Copy full SHA for 863b250

• Merge branch 'MLDSAI:main' into RWKV

  AvidEslami

  authored
  0a8838b

  Copy full SHA for 0a8838b

• clean up test_code

  AvidEslami

  committed
  0ff9955

  Copy full SHA for 0ff9955

Commits on May 30, 2023

• removed timeout, import model from huggingface, works for 7B

  AvidEslami

  committed
  7300b7e

  Copy full SHA for 7300b7e

Commits on May 31, 2023

• large models load, testing performance and uses, will look into faster load and deployment

  AvidEslami

  committed
  d24beb7

  Copy full SHA for d24beb7

• undo

  AvidEslami

  committed
  1fadf2d

  Copy full SHA for 1fadf2d

• formatting

  AvidEslami

  committed
  2424a1e

  Copy full SHA for 2424a1e

• improvments made to prompt

  AvidEslami

  committed
  2418fb5

  Copy full SHA for 2418fb5

Commits on Jun 1, 2023

• improve prompt formatting

  AvidEslami

  committed
  f742dd5

  Copy full SHA for f742dd5

• fixed mounting to use tokenizer from repository, removed unnecessary comments

  AvidEslami

  committed
  e8ffc75

  Copy full SHA for e8ffc75

• change approach, now uses parameters such as temperature

  AvidEslami

  committed
  f0f1b0c

  Copy full SHA for f0f1b0c

Commits on Jun 2, 2023

• switched to downloading tokenizer on launch

  AvidEslami

  committed
  17e5be5

  Copy full SHA for 17e5be5

• added versions to requirements

  AvidEslami

  committed
  8cec909

  Copy full SHA for 8cec909

Commits on Jun 5, 2023

• try calling run_RWKV from seperate file

  AvidEslami

  committed
  797b68c

  Copy full SHA for 797b68c

• using deploy, ready to function as mixin

  AvidEslami

  committed
  d887271

  Copy full SHA for d887271

• change stub and create mixin

  AvidEslami

  committed
  c3ef228

  Copy full SHA for c3ef228

Commits on Jun 6, 2023

• simplified prompting

  AvidEslami

  committed
  0ec1f27

  Copy full SHA for 0ec1f27

• switch to loading parameters from config.py

  AvidEslami

  committed
  d756244

  Copy full SHA for d756244

Commits on Jun 7, 2023

• loads parameters

  AvidEslami

  committed
  6b32766

  Copy full SHA for 6b32766

• Merge branch 'MLDSAI:main' into RWKV

  AvidEslami

  authored
  4152cbd

  Copy full SHA for 4152cbd

Commits on Jun 8, 2023

• modified input to make it closer to real life scenarios

  AvidEslami

  committed
  b8f3a4b

  Copy full SHA for b8f3a4b

• minor formatting improvements

  AvidEslami

  committed
  53406fc

  Copy full SHA for 53406fc

Commits on Jun 9, 2023

• more model types and less tests

  AvidEslami

  committed
  c2d7af3

  Copy full SHA for c2d7af3

• user now chooses which model to use

  AvidEslami

  committed
  f670356

  Copy full SHA for f670356

Commits on Jun 12, 2023

• added new parameter, removed default input,instruction, and task description, created more structured test for use demo

  AvidEslami

  committed
  82e834f

  Copy full SHA for 82e834f

• fix mixin

  AvidEslami

  committed
  59f2bc0

  Copy full SHA for 59f2bc0

• fix mixin case

  AvidEslami

  committed
  cc3ed61

  Copy full SHA for cc3ed61

• allow users to run test_RWKV without modal

  AvidEslami

  committed
  8d5cb00

  Copy full SHA for 8d5cb00

Commits on Jun 15, 2023

• remove unnecessary comment

  AvidEslami

  committed
  0781737

  Copy full SHA for 0781737

Commits on Jun 18, 2023

• added parameter for running on cpu, adding smaller model, confirmed local funtionality

  AvidEslami

  committed
  702e065

  Copy full SHA for 702e065

Commits on Jun 19, 2023

• Merge branch 'main' into RWKV

  AvidEslami

  authored
  e13f26e

  Copy full SHA for e13f26e

• added missing coma

  AvidEslami

  committed
  a07364a

  Copy full SHA for a07364a

Commits on Jun 30, 2023

• Explored newest RWKV model, still needs a bit of tests

  AvidEslami

  committed
  ca04175

  Copy full SHA for ca04175

• reduce tests to 1

  AvidEslami

  committed
  8e3aee0

  Copy full SHA for 8e3aee0

Commits on Jul 1, 2023

• added model descriptions to config

  AvidEslami

  committed
  7e0e323

  Copy full SHA for 7e0e323

Commits on Jul 3, 2023

• switch RWKV to using tempfiles

  AvidEslami

  committed
  bf17fb9

  Copy full SHA for bf17fb9

Commits on Jul 7, 2023

• experimenting with prompt generation scripts

  AvidEslami

  committed
  415931d

  Copy full SHA for 415931d

• created evaluate template

  AvidEslami

  committed
  b2f7c88

  Copy full SHA for b2f7c88

• increased populations of lists

  AvidEslami

  committed
  80289ca

  Copy full SHA for 80289ca

• remove relevance of dataframe to linkedin update

  AvidEslami

  committed
  05afbf5

  Copy full SHA for 05afbf5

• filled evaluate, just needs to call model

  AvidEslami

  committed
  3dfba54

  Copy full SHA for 3dfba54

Commits on Jul 14, 2023

• random includes all signals, generate_dataset creates X number of prompts with response concatenated at the end

  AvidEslami

  committed
  0385fa1

  Copy full SHA for 0385fa1

• signals are rearranged, model will no longer memorize simply ids associated with each task

  AvidEslami

  committed
  6227f88

  Copy full SHA for 6227f88

• resolved merge conflicts

  AvidEslami

  committed
  ee13d5e

  Copy full SHA for ee13d5e

• Increase dataset size to 5000 (larger dataset is preferred vs. having more epochs), prevent final newline character

  AvidEslami

  committed
  75c83fa

  Copy full SHA for 75c83fa

Commits on Jul 18, 2023

• fixed dataset, doesn't have trailing commas at the end of each line now follows proper jsonl format

  AvidEslami

  committed
  6f1b68e

  Copy full SHA for 6f1b68e

• fixed spreadsheet prompt grammar

  AvidEslami

  committed
  8745cdc

  Copy full SHA for 8745cdc

Commits on Jul 19, 2023

• desired outputs are sorted, finetune is in progress, generate labelled datasets keeps prompts and outputs seperate

  AvidEslami

  committed
  44dc443

  Copy full SHA for 44dc443

Commits on Jul 20, 2023

• finetunes, but model not saved

  AvidEslami

  committed
  7bde832

  Copy full SHA for 7bde832

Commits on Jul 21, 2023

• config file is being saved, but doesn't seem to load tokenizer correctly due to seemingly random output

  AvidEslami

  committed
  ae3f527

  Copy full SHA for ae3f527

Commits on Jul 24, 2023

• added comparison test

  AvidEslami

  committed
  c49999d

  Copy full SHA for c49999d

Commits on Jul 26, 2023

• attempt with simpler finetune test?

  AvidEslami

  committed
  ccd41b1

  Copy full SHA for ccd41b1

• Model can be reloaded after training though results are slightly different, runs on modal

  AvidEslami

  committed
  150b20e

  Copy full SHA for 150b20e

Commits on Jul 27, 2023

• Model finetunes and saves properly, upon reloading finetuned details remain

  AvidEslami

  committed
  e9c1106

  Copy full SHA for e9c1106

• cleaned finetune code

  AvidEslami

  committed
  81d18b3

  Copy full SHA for 81d18b3

Commits on Jul 28, 2023

• trying on larger rwkv models

  AvidEslami

  committed
  ce7a0d0

  Copy full SHA for ce7a0d0

Commits on Jul 31, 2023

• added modal structure to finetune test

  AvidEslami

  committed
  8bfb575

  Copy full SHA for 8bfb575

Commits on Aug 4, 2023

• added docstring + Black

  AvidEslami

  committed
  384449b

  Copy full SHA for 384449b

Commits on Aug 24, 2023

• minor grammar fix

  AvidEslami

  committed
  f01883f

  Copy full SHA for f01883f

Commits on Aug 25, 2023

• minor improvements to prompt

  AvidEslami

  committed
  8109f10

  Copy full SHA for 8109f10

Commits on Aug 28, 2023

• updated prompts script

  AvidEslami

  committed
  dba5b6d

  Copy full SHA for dba5b6d

• increase dataset size

  AvidEslami

  committed
  b5310b8

  Copy full SHA for b5310b8

Commits on Feb 4, 2024

• Removed RWKV world models for code clarity and changed test script to be more general (RWKV 5 is releasing soon and it is worth testing)

  AvidEslami

  committed
  2b833cb

  Copy full SHA for 2b833cb