demo_dataset_train_evaluate.ipynb

{"cells":[{"attachments":{},"cell_type":"markdown","metadata":{"id":"4rdX2Nqyv8w-"},"source":["## Demo notebook to run through main scripts\n","This goes through the following\n","\n","1.   Create Dataset (using a fraction of total dataset - takes about 5-10mins to download and run) - **THIS CAN BE SKIPPED**\n","2.   Finetune model on toxicity on very small dataset - takes about 5-10 mins to run - **THIS CAN BE SKIPPED**\n","3. **Evaluate** our fully fine-tuned model on small validation set\n","4. Play time - Test our LM with your own sentences\n","\n","\n","### Please connect to GPU before running\n","Activate session -> click RAM/Disk in the top right -> Change Runtime Type -> Activate GPU\n","\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":18212,"status":"ok","timestamp":1680535632721,"user":{"displayName":"Fred Clarke","userId":"10981377289460040773"},"user_tz":-60},"id":"MtwdQAdxv6gf","outputId":"fa8d69cb-f794-42c5-ddfc-fac859c2a7a2"},"outputs":[],"source":["from google.colab import drive\n","drive.mount('/content/drive')"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"RPwfIsDUwbk_"},"source":["### Change path to where folder shortcut is stored in your MrDrive"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":9,"status":"ok","timestamp":1680534305258,"user":{"displayName":"Fred and Tertia","userId":"04166755790669271432"},"user_tz":-60},"id":"WDwlaUl2v64Y","outputId":"44115608-7845-4d13-abf3-891c1e962b28"},"outputs":[],"source":["%cd /content/drive/MyDrive/group_19_demo"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"d00hLqgfGuSK"},"source":["### Install requirements\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":20573,"status":"ok","timestamp":1680534325826,"user":{"displayName":"Fred and Tertia","userId":"04166755790669271432"},"user_tz":-60},"id":"m3F-5rkavaoO","outputId":"22522a81-a68e-4dc5-8919-2f14670595e4"},"outputs":[],"source":["! pip install -r requirements.txt"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"PY2fD9eUwhd-"},"source":["### 1. Create very small dataset\n","This is creating a dataset only using 0.1% of total data available.\n","It downloads the full dataset first which is why it takes 5-10mins. This is assuming the tokenizer has already been created which was in our submission folder. We downloaded the tokenizer by running \"!python scripts/tokenizer_create.py\"\n","\n","If instead you want to create the full dataset run:\n"," - ! python main_dataset_create.py args/args_dataset_full\n","\n","If you want to change the dataset settings edit the args/args_dataset_demo.yaml file.\n","\n","*This can be skipped, and you can go straight to the the next step*"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":422916,"status":"ok","timestamp":1680534748732,"user":{"displayName":"Fred and Tertia","userId":"04166755790669271432"},"user_tz":-60},"id":"WaHjgwWNwqR_","outputId":"5a138224-1b53-4ea4-ecd6-1c12b755fcf5"},"outputs":[],"source":["! python main_dataset_create.py args/args_dataset_demo    # creates very small dataset, about 5mins \n","# !python main_dataset_create.py args/args_dataset_full   # takes much longer to run - 30mins"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"mLsQnmvuwz5e"},"source":["###2. Finetune model on Toxicity classifier with very small dataset\n","This is using a very small dataset, either the one just created or one already in the submitted folder\n","Takes about 5 mins\n","\n","If instead you want to train on the full dataset (4+hrs), first the full dataset will need to be created, then trained on that dataset:\n"," - !python main_dataset_create.py args/args_dataset_full\n"," - !python main_train.py args/args_train_full\n","\n","If you want to change the training settings edit the args_train_demo.yaml file\n","\n","*This can be skipped, and you can go straight to the the next step*"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":21308,"status":"ok","timestamp":1680534770032,"user":{"displayName":"Fred and Tertia","userId":"04166755790669271432"},"user_tz":-60},"id":"Pn6R_SgUxIJ7","outputId":"53c84674-2c0b-4383-a47f-1ce2839d31ff"},"outputs":[],"source":["! python main_train.py args/args_train_demo   # (5mins)\n","# !python main_train args/args_train_full     # (4hrs) Requires creating main dataset first - running !python main_dataset_create.py args/args_dataset_full"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"g1Xe5ml-xkLU"},"source":["###3. Evaluate our fully fine-tuned models"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"mDPl4nD7GlsO"},"source":["The main_evaluate.py is performing evaluation on any set of hyperparameters and can take an array of different models and paths and conditional tokens. These parameters should be defined in .yaml files. The full evaluations that were performed for the tables in the paper were prompted with 400 samples from the validation set. The demo ones only prompt with 20 to make it faster. The tables are saved in the folder 'eveluate_outputs' as defined in the .yaml files. The four ones below do the following:\n"," - args_evaluate_main1 --> compares best versions of the 4 models using the set of hyperparameters described in the paper, conditioning on both <|nontoxic|> and <|pos|> tokens on their own and together.\n"," - args_evaluate_main2 --> compares best versions of the 4 models using a different set of hyperparameters (temp=0.99), conditioning on both <|nontoxic|> and <|pos|> tokens on their own and together.\n"," - args_evaluate_extra --> Compares 6 models fine-tuned on different number of sentences condiitoned on both <|nontoxic|> and <|pos|>. These are refered to as M-TS1 to M-TS6 in the paper. Uses same hyperparameters as experiment_main_1\n","\n","**NB: The tables created from the demo might have values of misalignment that do not match the paper. The reason is that the evaluation with only 20 samples has very high variance. For 400 samples the results were relatively stable.**"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":338029,"status":"ok","timestamp":1680535108051,"user":{"displayName":"Fred and Tertia","userId":"04166755790669271432"},"user_tz":-60},"id":"3EJbW2E8GjSz","outputId":"feba9f42-20d0-4491-ca41-74f6f583be40"},"outputs":[],"source":["!python main_evaluate.py args/args_evaluate_main1_demo\n","# !python main_evaluate.py args/args_evaluate_main1\n","\n","# !python main_evaluate.py args/args_evaluate_main2_demo\n","# !python main_evaluate.py args/args_evaluate_main2\n","\n","# !python main_evaluate.py args/args_evaluate_extra_demo\n","# !python main_evaluate.py args/args_evaluate_extra"]},{"cell_type":"code","execution_count":null,"metadata":{"cellView":"form","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":295021,"status":"ok","timestamp":1680535403066,"user":{"displayName":"Fred and Tertia","userId":"04166755790669271432"},"user_tz":-60},"id":"aWUS7WWz5UXD","outputId":"b2b3e77b-8981-4a13-e221-a106f59d7ab7"},"outputs":[],"source":["#@title Play Time --> Write your own sentences to prompt our fine-tuned LM\n","#@markdown Please make sure the sentences are well defined and do not include punctuation\n","\n","Prompt1 = \"Once upon a time\" #@param {type:\"string\"}\n","Prompt2 = \"The most exciting thing is \" #@param {type:\"string\"}\n","Prompt3 = \"The most shitty thing is\" #@param {type:\"string\"}\n","Prompt4 = \"These motherfuckers who\" #@param {type:\"string\"}\n","Prompt5 = \"Whales are a species that\" #@param {type:\"string\"}\n","\n","from scripts.load_and_run_experiment_get_samples import load_and_run\n","\n","input_prompts = [\"Hey, how are you?\",\n","                 \"Fuck you man!\",\n","                 \"Have a good day, sir!\",\n","                 'Do birds fly?',\n","                 'Once upon a time']\n","\n","input_prompts_mine = [Prompt1,\n","                      Prompt2,\n","                      Prompt3,\n","                      Prompt4,\n","                      Prompt5,\n","]\n","                 \n","file_name = 'args/args_get_samples_user_input'\n","random_seed = 200\n","load_and_run(file_name,random_seed, input_prompts_mine)"]}],"metadata":{"accelerator":"GPU","colab":{"provenance":[]},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}