Implementing RWKV-LLM (#37)

[AvidEslami]

This pull requests implements RWKV's RNN model using Modal. (Issue #37 ) (Reference)

Summary: Created RWKV.py, (will move to openadapt/strategies/mixins and restructure to accept inputs)

Run using the following command (will be changed once implemented as a mixin):

modal token new
modal run .\openadapt\RWKV\RWKV.py

Implementation:

• Starts a Modal application, gpu must be set to a100 in order to run Raven-14B (largest model)
• Downloads weights for the desired model from huggingface to the modal server
• Computes and returns output

TODO:

• Change structure to function as a mixin (take inputs such as prompts or task descriptions)
• Allow for modification of parameters via config file (Temperature, Top P, Presence Penalty, etc...)
• Deploy Modal application permanently to avoid startup times (Upon startup weights must be downloaded, to avoid this we can deploy the Modal app externally and make requests to it instead)

[abrichr]

As mentioned in Slack, I think it's time to start fine tuning 🤓

[AvidEslami]

Signals Finetune Update:

• The following sheet contains the outputs generated by the finetuned models link
• Any advice on the approach is appreciated, there are still several things to try:
  • Having a random numbers of signals, which could perhaps reduce the models chance of memorizing results
  • Changing prompt structure could improve understanding
  • Creating a more varied dataset (more signals / more tasks) help model catch relations
  • Finetuning Pile-14B for several epochs (will try), Pile-14B is the non finetuned version of Raven-14B

As usual please let me know if you have any suggestions!