1140310118 / tdlm Public

Notifications You must be signed in to change notification settings
Fork 5
Star 40

实现了Transformer中的几种位置编码方案

40 stars 5 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 15 Commits
bash		bash
img		img
model		model
utils		utils
README.md		README.md
tdlm_pretrain.py		tdlm_pretrain.py

Repository files navigation

tdlm

实现了Transformer中的几种位置编码方案

位置编码的介绍与实验结果见知乎-位置编码上的一次实验。

运行

设置好data目录和输出目录后运行

bash/pretrain.sh -c 0 -l 20 -p 'layerwise_rope' -e transformer

requirements

python3
pytorch-1.8.1
pytorch_lightning-1.4.9
transformers-4.6.1
einops-0.3.2
performer_pytorch-1.1.0

About

实现了Transformer中的几种位置编码方案

Report repository

Releases

No releases published

Packages

No packages published

Languages