Psunshin / build_LLM_from_skyer Public

forked from tunglinwood/build_LLM_from_skyer

Notifications You must be signed in to change notification settings
Fork 0
Star 0

This repo is to implement skyer_llm and build a llm from scratch.

0 stars 4 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
preprocess		preprocess
pretrain		pretrain
tokenizer		tokenizer
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
pretrain.sh		pretrain.sh
tokenizer.model		tokenizer.model
tokenizer.vocab		tokenizer.vocab

Repository files navigation

Skyer: 从0开始手搓、训练、上传到huggingface使用的的0.5B大语言模型

基本参数

参数	值
Layer number	20
Input dimension	2048
Hidden dimension	1536
Query head numbers	24
Key/Value head numbers	12
Maximum length	1024
Vocabulary number	30000

预训练数据集

名称	数据大小
SkyPile-150B	620 GB
书生-万卷-文本数据集1.0	1094 GB
WuDaoCorpora	200 GB
BAAI-CCI3	1000 GB

About

This repo is to implement skyer_llm and build a llm from scratch.

Report repository

Releases

No releases published

Packages

No packages published

Languages

Jupyter Notebook 96.3%
Python 2.4%
Shell 1.3%