Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor llama3 demo to the new generator API #16753

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open

Refactor llama3 demo to the new generator API #16753

wants to merge 39 commits into from
+1,254 −1,247

Conversation

mtairum
Copy link
Contributor

@mtairum mtairum commented Jan 15, 2025 •
edited
Loading

Ticket

#16752

What's changed

  • New Llama3 demo now uses the generator API
  • Improved prefill performance. E.g. Llama3-70B now at 182ms prefill time
  • Improved profiling in the demo
  • Removed old text demo and updated CI accordingly
  • Cleaned up the prompt input files and added missing ones.
  • New benchmark profiling for superset: now includes TTFT, and full decode perf for 4096 iteration (for plotting).
  • Add llama3 demo custom input support: you can now override any settings for easier testing.
  • Updated PERF.md with the latest numbers.

Checklist

@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from 4f4c250 to decc3fa Compare January 17, 2025 17:27
@mtairum mtairum marked this pull request as ready for review January 17, 2025 17:37
@mtairum mtairum self-assigned this Jan 17, 2025
@mtairum mtairum added the llama3 label Jan 17, 2025
@mtairum mtairum requested a review from skhorasganiTT January 17, 2025 17:37
skhorasganiTT
skhorasganiTT reviewed Jan 17, 2025
View reviewed changes
Copy link
Contributor

@skhorasganiTT skhorasganiTT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all the files except simple_text_demo.py for now since that one is still being cleaned up

models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/llama_model.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from 943dff3 to 5dfa882 Compare January 20, 2025 12:45
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from 3847b2b to 4161293 Compare January 21, 2025 14:30
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch 2 times, most recently from e89d7e5 to 82e3d03 Compare January 24, 2025 15:32
@mtairum mtairum requested a review from a team as a code owner January 24, 2025 16:30
@tt-rkim
Copy link
Collaborator

tt-rkim commented Jan 25, 2025

Did you run nightly ttnn single card, t3k demos, tg demos?

@mtairum
Copy link
Contributor Author

mtairum commented Jan 27, 2025

@tt-rkim In the process of fixing the CI issues. Will only merge after they pass 👍

@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch 2 times, most recently from ec155d5 to 2dc2e9b Compare January 28, 2025 15:06
@mtairum mtairum mentioned this pull request Jan 29, 2025
Open
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch 2 times, most recently from 679a4e2 to 27beb0c Compare January 29, 2025 17:11
@mtairum
Copy link
Contributor Author

mtairum commented Jan 29, 2025

This PR is now feature complete, with a bunch of new benchmark functionality and QoL updates to the demo to better measure what the customer team throws at us.

Waiting for CI to pass then it's good to go.

yieldthought
yieldthought approved these changes Jan 29, 2025
View reviewed changes
Copy link
Contributor

@yieldthought yieldthought left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean

models/demos/llama3/scripts/op_perf_results.py Outdated Show resolved Hide resolved
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch 2 times, most recently from 8424b8f to ac0318e Compare January 30, 2025 17:43
@mtairum
Copy link
Contributor Author

mtairum commented Jan 30, 2025 •
edited
Loading

Relaunching all tests after addressing @cglagovichTT comments.

The branch is feature complete. No more changes unless to fix the CI pipelines.

Checklist

mtairum added 28 commits January 31, 2025 09:49
@mtairum
#0: Moved llama3 input prompts to a separate folder. Added missing lo…
83b6404 
…ng prompts and tested all of them
@mtairum
#0: Fix tensor padding and moved to inside llama model
d1a6653
@mtairum
#0: Add TTFT info to lt
f2194c4
@mtairum
#0: Removed unused argmax_on_device
a5859ba
@mtairum
#0: Moved the dram mem config to the end
c027942
@mtairum
#0: Updated attention wo dense matmul program config to increase perf…
3a87554 
… and reduce time to first token
@mtairum
#: Reduced the printed outputs to increase readability when using 1 user
fce423c
@mtairum
#0: Removed demo.py. Updated references to new simple_text_demo.py
a819b8b
@mtairum
#0: Added missing requirements to CI demo tests. Improve prefill time…
d45815a 
… by reducing the size of rot_mats being passed to prefill
@mtairum
#0: Fix average tok/s/u to account for compile iteration
a33c7f1
@mtairum
#0: Add req to single chip demo script
a551fb2
@mtairum
#0: Add missing req to tg CI
597c3b8
@mtairum
#0: Updated ref in llama accuracy
249e79d
@mtairum
#0: Fix CI
0805d5a
@mtairum
#0: Fix llama test model
93cdda7
@mtairum
#0: Fix profiler benchmark names to actually work
a5a9efd
@mtairum
0: Added new stop_at_eos option in demo, to ignore stopping generatio…
3da2793 
…n until it reaches the max. Also added support for custom input parameters
@mtairum
#0: Added new llama3 demo CI options. Now they always run 4k iteratio…
1b05a63 
…n for 1 user or 32 users and we save that data to superset
@mtairum
#0: Moved llama pip requirements to the requirements dev. Fix llama C…
2d6116c 
…I tests
@mtairum
#0: Fix llama demo ci-32 to fit 70B on T3K
a359f83
@mtairum
#0: Add new metric to the llama benchmark
e5338d0
@mtairum
#0: Update Llama Readme
26c5ade
@mtairum
#0: Update llama3 perf.md
e227f35
@mtairum
#0: Re add llama3 op_perf_results script
9a3a872
@mtairum
#0: Removed LLama 11B demo from N150, since it can't fit a batch of 32
5db7119
@mtairum
#0: Minor adjustment to PCC check on llama vision test
8b9b7cd
@mtairum
#0: Avoid generating cos/sin matrices twice
6f9ab1e
@mtairum
#0: Fix CI N150/N300
a9940a0
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from 8dc64b3 to a9940a0 Compare January 31, 2025 09:50
@mtairum
Copy link
Contributor Author

mtairum commented Jan 31, 2025

All CI are passing.
But due to the higher priority of Qwen2.5 and Deepseek distilled models, the merge of this branch will wait for that to hit main to avoid delays.

Godspeed @yieldthought 🫡

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cglagovichTT cglagovichTT cglagovichTT left review comments

@skhorasganiTT skhorasganiTT skhorasganiTT approved these changes

@tt-rkim tt-rkim tt-rkim approved these changes

@yieldthought yieldthought yieldthought approved these changes

@uaydonat uaydonat Awaiting requested review from uaydonat uaydonat is a code owner

Assignees

@mtairum mtairum

Labels
llama3
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mtairum @tt-rkim @yieldthought @cglagovichTT @skhorasganiTT