Add RoPE positional encoding

[gordicaleksa]

Implemented RoPE - rotary position embedding from the RoFormer paper.

Note:

1. I do not conditionally remove the allocation of our learnable position embedding buffer (wpe) as that would require touching many parts of the codebase that rely on the particular order inside the parameter buffer (e.g. wpe has index 1).
2. I do turn off fwd/bwd computation / grad norm computation / update for the wpe buffer.
   The explicit tradeoff is: suffer a minimal memory bloat (maxT * C) but the PR has minimal impact on the readability of the codebase.

Tests:
I ran an A/B experiment: trained a 124M GPT-2 on 10B tokens (FineWeb subset) with:
a) learnable positional embeddings (default, -er == 0)
b) RoPE (-er == 1)
c) no positional embedding at all
all other settings being the same same.

Results:

Conclusions:

• The validation loss is significantly better with RoPE
• RoPE implementation slightly decresed the performance (consistent with what EleutherAI folks observed). I observed a drop ~1_632_000 -> ~1_603_000 tok/s (~1.7% perf hit).

[karpathy]

I don't like this because use_rope has nothing to do with this encoding function, which we want to be nice and modular and self-contained. Maybe something like use_positional or something like that.

[gordicaleksa]

agree, can refactor a few of those bits quickly

[OwenSanzas]

I tried this branch on HPRC, it did not reach the lowest loss you had.

#!/bin/bash
#SBATCH --job-name=gpt2_train
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --time=26:00:00 #Request 26 hours (2 extra hours)
#SBATCH --mem=128GB #Request 128GB per node
#SBATCH --partition=gpu #Request the GPU partition/queue
#SBATCH --gres=gpu:a100:1 #Request one A100 GPU to use

#SBATCH --output=gpt2_train.%j.log #Redirect stdout/err to file

Run the training script

./train_gpt2cu -i "dev/data/fineweb10B/fineweb_train_.bin" -j "dev/data/fineweb10B/fineweb_val_.bin" -o log124M -e "d12" -b 32 -t 1024 -d 524288 -r 0 -z 1 -c 0.1 -l 0.0006 -q 0.0 -u 700 -n 5000 -v 250 -s 20000 -h 1