Skip to content

This repo is to implement skyer_llm and build a llm from scratch.

License

Notifications You must be signed in to change notification settings

tunglinwood/build_LLM_from_skyer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Skyer: 从0开始手搓、训练、上传到huggingface使用的的0.5B大语言模型

基本参数

参数
Layer number 20
Input dimension 2048
Hidden dimension 1536
Query head numbers 24
Key/Value head numbers 12
Maximum length 1024
Vocabulary number 30000

预训练数据集

名称 数据大小
SkyPile-150B 620 GB
书生-万卷-文本数据集1.0 1094 GB
WuDaoCorpora 200 GB
BAAI-CCI3 1000 GB

About

This repo is to implement skyer_llm and build a llm from scratch.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published