趋动云 B1.small 等同于 3090 的 1/4 之一卡的硬件测试环境。运行性能测试对比 python benchmark.py，lite_llama 的运行速度最高是 transformers 的 4x 倍。
batch_size = 16 的提示词：
prompts: List[str] = [
"I believe the meaning of life is to find happiness in the simple things. but how to achieve the meaning of life?",
"VGG is a very important cnn backbone, please introduce vgg architecture and give implement code ",
"Can you introduce the History of the American Civil War. ",
"who is the first president of the United States and what's his life story?",
"How to learn c++, give me some code example.",
"How to learn python, give me some code examples.",
"How to learn llm, please introduce transformer architecture ",
"How to learn cnn, please introduce resnet architecture and give code ",
"How to learn cuda programming, give me some code example.",
"How to learn rust, give me some code examples.",
"How to learn java, give me some code example.",
"How to learn linux c, give me some code examples.",
"A Complete Introduction to the History of the American Civil War",
"Python is a good programming language, how tolearn it?",
"Please introduce llama model architecture and give implement cuda code."
"Please introduce Qwen2.5 model structure and give cuda implement code."
]max_gen_len = 1900 时，benchmark 性能测试运行结果:
lite_llama inference time: 67.8760 s
Transformers inference time: 131.8708 s
lite_llama throughput: 411.04 tokens/s
Transformers throughput: 104.70 tokens/s
lite_llama per token latency: 2.432831 ms/token
Transformers per token latency: 9.551007 ms/token/gemini/code/lite_llama/my_weight/Llama-3.2-1B-Instruct
趋动云 B1.big 等同于 3090 卡的硬件测试环境。运行性能测试对比 python benchmark.py，lite_llama 的运行速度最高是 transformers 的 4x 倍。
batch_size = 8 的提示词：
prompts: List[str] = [
"I believe the meaning of life is to find happiness in the simple things. This is a very subjective and personal perspective, and it may vary from person to person. However, I believe that the simple things can bring a sense of joy and fulfillment to our lives.",
"VGG is a very important cnn backbone, please introduce vgg architecture and give implement code ",
"A Complete Introduction to the History of the American Civil War",
"Roosevelt was the first president of the United States, he has a lot of information on the early history of the United States. He was born in 1883,",
"How to learn c++, give me some code example.",
"How to learn python, give me some code examples.",
"How to learn llm, please introduce transformer architecture ",
"How to learn cnn, please introduce resnet architecture and give code ",
]max_gen_len = 1900 时，benchmark 性能测试运行结果:
lite_llama inference time: 32.0826 s
Transformers inference time: 51.2225 s
lite_llama throughput: 458.97 tokens/s
Transformers throughput: 134.37 tokens/s
lite_llama per token latency: 2.178783 ms/token
Transformers per token latency: 7.441883 ms/tokenbatch_size = 12 的提示词：
prompts: List[str] = [
"I believe the meaning of life is to find happiness in the simple things. but how to achieve the meaning of life?",
"VGG is a very important cnn backbone, please introduce vgg architecture and give implement code ",
"Can you introduce the History of the American Civil War. ",
"who is the first president of the United States and what's his life story?",
"How to learn c++, give me some code example.",
"How to learn python, give me some code examples.",
"How to learn llm, please introduce transformer architecture ",
"How to learn cnn, please introduce resnet architecture and give code ",
"How to learn cuda programming, give me some code example.",
"How to learn rust, give me some code examples.",
"How to learn java, give me some code example.",
"How to learn linux c, give me some code examples.",
]max_gen_len = 1900 时，benchmark 性能测试运行结果:
lite_llama inference time: 31.3463 s
Transformers inference time: 69.1433 s
lite_llama throughput: 730.45 tokens/s
Transformers throughput: 183.95 tokens/s
lite_llama per token latency: 1.369015 ms/token
Transformers per token latency: 5.436221 ms/tokenbatch_size = 2 时的提示词
prompts: List[str] = [
"How to learn cnn, please introduce resnet architecture and give code ",
"How to learn cuda programming, give me some code example.",
]max_gen_len = 2000 时, benchmark 性能测试运行结果:
lite_llama inference time: 34.9293 s
Transformers inference time: 31.6787 s
lite_llama throughput: 98.71 tokens/s
Transformers throughput: 69.83 tokens/s
lite_llama per token latency: 10.130305 ms/token
Transformers per token latency: 14.321302 ms/tokenbatch_size = 4 时的提示词
prompts: List[str] = [
"How to learn cnn, please introduce resnet architecture and give code.",
"How to learn cuda programming, give me some code example.",
"How to learn rust, give me some code examples.",
"How to learn java, give me some code example.",
]max_gen_len = 256 时, benchmark 性能测试运行结果:
lite_llama inference time: 5.5739 s
Transformers inference time: 7.6803 s
lite_llama throughput: 182.28 tokens/s
Transformers throughput: 133.33 tokens/s
lite_llama per token latency: 5.486118 ms/token
Transformers per token latency: 7.500309 ms/tokenbatch_size = 12 时的提示词
prompts: List[str] = [
"I believe the meaning of life is to find happiness in the simple things. but how to achieve the meaning of life?",
"VGG is a very important cnn backbone, please introduce vgg architecture and give implement code ",
"Can you introduce the History of the American Civil War. ",
"who is the first president of the United States and what's his life story?",
"How to learn c++, give me some code example.",
"How to learn python, give me some code examples.",
"How to learn llm, please introduce transformer architecture ",
"How to learn cnn, please introduce resnet architecture and give code ",
"How to learn cuda programming, give me some code example.",
"How to learn rust, give me some code examples.",
"How to learn java, give me some code example.",
"How to learn linux c, give me some code examples.",
]max_gen_len = 1900 时，benchmark 性能测试运行结果:
lite_llama inference time: 26.8804 s
Transformers inference time: 63.2376 s
lite_llama throughput: 581.20 tokens/s
Transformers throughput: 172.19 tokens/s
lite_llama per token latency: 1.720564 ms/token
Transformers per token latency: 5.807474 ms/tokenbatch_size = 16 时的提示词
prompts: List[str] = [
"I believe the meaning of life is to find happiness in the simple things. but how to achieve the meaning of life?",
"VGG is a very important cnn backbone, please introduce vgg architecture and give implement code ",
"Can you introduce the History of the American Civil War. ",
"who is the first president of the United States and what's his life story?",
"How to learn c++, give me some code example.",
"How to learn python, give me some code examples.",
"How to learn llm, please introduce transformer architecture ",
"How to learn cnn, please introduce resnet architecture and give code ",
"How to learn cuda programming, give me some code example.",
"How to learn rust, give me some code examples.",
"How to learn java, give me some code example.",
"How to learn linux c, give me some code examples.",
"A Complete Introduction to the History of the American Civil War",
"Python is a good programming language, how tolearn it?",
"Please introduce llama model architecture and give implement cuda code."
"Please introduce Qwen2.5 model structure and give cuda implement code."
]max_gen_len = 512 时，benchmark 性能测试运行结果:
lite_llama inference time: 11.3434 s
Transformers inference time: 14.9981 s
lite_llama throughput: 724.38 tokens/s
Transformers throughput: 504.73 tokens/s
lite_llama per token latency: 1.380484 ms/token
Transformers per token latency: 1.981256 ms/tokenmax_gen_len = 1900 时，benchmark 性能测试运行结果:
lite_llama inference time: 38.4323 s
Transformers inference time: 70.3268 s
lite_llama inference output tokens number: 28276
Transformers inference output tokens number: 15164
lite_llama throughput: 735.73 tokens/s
Transformers throughput: 215.62 tokens/s
lite_llama per token latency: 1.359186 ms/token
Transformers per token latency: 4.637745 ms/token