Skip to content

(T3K) T3000 model perf tests #1442

(T3K) T3000 model perf tests

(T3K) T3000 model perf tests #1442

Manually triggered February 28, 2025 09:16
Status Failure
Total duration 49m 9s
Artifacts 9

t3000-model-perf-tests.yaml

on: workflow_dispatch
build-artifact-profiler  /  ...  /  check-docker-images
7s
build-artifact-profiler / build-docker-image / check-docker-images
build-artifact  /  ...  /  check-docker-images
7s
build-artifact / build-docker-image / check-docker-images
build-artifact-profiler  /  ...  /  🐳️ Build image
0s
build-artifact-profiler / build-docker-image / 🐳️ Build image
build-artifact  /  ...  /  🐳️ Build image
0s
build-artifact / build-docker-image / 🐳️ Build image
build-artifact-profiler  /  🛠️ Build Release ubuntu 20.04
3m 50s
build-artifact-profiler / 🛠️ Build Release ubuntu 20.04
build-artifact  /  🛠️ Build Release ubuntu 20.04
4m 4s
build-artifact / 🛠️ Build Release ubuntu 20.04
Matrix: t3000-model-perf-tests / t3000-model-perf-tests
Fit to window
Zoom out
Zoom in

Annotations

2 errors, 21 warnings, and 16 notices
t3000-model-perf-tests / t3k LLM mixtral model perf tests
Process completed with exit code 1.
t3000-model-perf-tests / t3k LLM mixtral model perf tests
Process completed with exit code 1.
hugepages-service-not-found-startup
Hugepages service not found. Using old rc.local method
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k CNN resnet50 model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k CNN resnet50 model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k CNN resnet50 model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k CNN resnet50 model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k CNN resnet50 model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k CNN resnet50 model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k CNN resnet50 model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k CNN resnet50 model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon7b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
t3000-model-perf-tests / t3k LLM falcon7b model perf tests: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
disk-usage-after-startup
Disk usage is 53 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Artifacts

Produced during runtime
Name Size
TTMetal_build_any
186 MB
TTMetal_build_any_profiler
191 MB
packages-ubuntu-20.04-amd64-Release-x86_64-linux-clang-17-libcpp
89.4 MB
packages-ubuntu-20.04-amd64-Release-x86_64-linux-clang-17-libcpp_profiler
89.5 MB
perf-report-csv--wormhole_b0--bare-metal
2.34 KB
perf-report-csv-CNN-wormhole_b0-resnet50-bare-metal
536 Bytes
perf-report-csv-LLM-wormhole_b0-falcon40b-bare-metal
662 Bytes
perf-report-csv-LLM-wormhole_b0-falcon7b-bare-metal
864 Bytes
perf-report-csv-LLM-wormhole_b0-llama2-70b-bare-metal
421 Bytes