A GPT code in R language pretrained with Shakespeare text. You can use the trained model to generate Shakespeare-like texts or you can train from scratch with other hyperparameters (vocabulary size, context size, embedding dimension, etc) and other texts. In this repository, it is provided a training with BPE and without BPE (based only with characters like in Karpathy's video. The Shakespeare text was downloaded here.
Notice that the generate text is not coherent because the model and the corpus are both small. Increasing the context size, embedding dimension, number of layers and heads, and the size of vocabulary in a subword level, it is expected the generated text will maintain coherence along the words and sentences.
I would like to thanks Álvaro Kothe for helping me to organize parts of the code in an earlier implementation of GPT for R.
- tokenizers.bpe
- torch
- luz. Make sure you have installed its version with AMP support. If you have not, set
AMP = FALSEin the config.R file.
If you have a GPU, you can try training this baby GPT model from scratch. You need to define a batch size that fits into your GPU memory (default is 32). Your results might be different for other batch sizes. Before training the model, make sure the number of workers is in order with your machine. Default is 6 workers. In order to train with BPE (Byte-Pair Encoding) or without BPE, open the file main.R and set BPE = TRUE or BPE = FALSE. If you want to train set Train = TRUE. If you want to predict tokens set Run = TRUE. After setting all hyperparameters in the main.R file, run the following in the main folder:
source('main.R')
The file Train.R trains the model. If BPE = TRUE, then it will use a vocabulary tokenized by BPE (from the package tokenizers.bpe). The file youtokentome.bpe contains the tokenized vocabulary (of size 68 tokens) for the Shakespeare text. If BPE = TRUE, then it will use a vocabulary with 65 single characters plus a PAD character. This latter procedure is closer to what Karpathy does in his video and the generated text contains proper line breaks which makes the reading a little bit more pleasant.
The Automatic Mixing Precision (AMP)is set as by default in the config.R file to use the Automatic Mixing Precision (AMP). This helps increasing the capacity of your training. If you do not have installed the latest version of the luz package, set AMP = FALSE in the config.R file.
An example of 700 tokens generated by setting BPE = FALSE, Train = FALSE and Run = TRUE is given below:
My lord!
I take my boy a poison that I have been to death.
KING HENRY VI: There will I stand the wars at him: Therefore it is not the court?
BUCKINGHAM: This is the world and to be so.
HENRY BOLINGBROKE: As my liege, to the world as the world That thou shalt straight and truth in their That the worst season that which he does him As they are as they are all their children, Which was a present to the walls of thing.
BUCKINGHAM: Why stands him all the world that they would be That which they shall be the subject of this county, The second of his court?
BUCKINGHAM: Well, well thee to say that the way That which the world of the season Which they say to the crown of this welcome That we shall be s
An example of 700 tokens generated by setting BPE = TRUE, Train = FALSE and Run = TRUE is given below:
My lord! cots and call my brown father, and be gone, And that hath been to be past of his pawder, To stay the traitor of the court. CAMILLO: I do not burn the consul. ISABELLA: I can be before your subjects I had been in your grave A present for the crown of your conscience. ISABELLA: Ay, good Catesby your life, I have no more of her birth. ISABELLA: This be your lordship in your courtesy, I warrant you. The gods is this bark of your company, And I cannot but be as before your committed with her. LUCENTIO: As I charge your grace, sir. LUCIO: I have been for a king. ISABELLA: And thou hast bonded of this foolish That I may be continue to be another. If thou hast been for that I did not before your cousin And now me at an it in your charity, That you can b
MIT