Skip to content

(T3K) T3000 model perf tests #1325

(T3K) T3000 model perf tests

(T3K) T3000 model perf tests #1325

Triggered via schedule January 14, 2025 12:08
Status Failure
Total duration 59m 53s
Artifacts 7
build-artifact-profiler  /  ...  /  build-docker-image
22s
build-artifact-profiler / build-docker-image / build-docker-image
build-artifact  /  ...  /  build-docker-image
22s
build-artifact / build-docker-image / build-docker-image
Matrix: build-artifact-profiler / build-artifact
Matrix: build-artifact / build-artifact
Matrix: t3000-model-perf-tests / t3000-model-perf-tests
Fit to window
Zoom out
Zoom in

Annotations

4 errors, 7 warnings, and 23 notices
t3000-model-perf-tests / t3k LLM falcon40b model perf tests
Process completed with exit code 1.
t3000-model-perf-tests / t3k LLM falcon40b model perf tests
Process completed with exit code 1.
t3000-model-perf-tests / t3k LLM mixtral model perf tests
Process completed with exit code 1.
t3000-model-perf-tests / t3k LLM falcon7b model perf tests
Process completed with exit code 1.
hugepages-service-not-found-startup
Hugepages service not found. Using old rc.local method
t3000-model-perf-tests / t3k LLM falcon40b model perf tests
Your workflow is using a version of actions/cache that is scheduled for deprecation, actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2. Please update your workflow to use either v3 or v4 of actions/cache to avoid interruptions. Learn more: https://github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions/#actions-cache-v1-v2-and-actions-toolkit-cache-package-closing-down
t3000-model-perf-tests / t3k CNN resnet50 model perf tests
Your workflow is using a version of actions/cache that is scheduled for deprecation, actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2. Please update your workflow to use either v3 or v4 of actions/cache to avoid interruptions. Learn more: https://github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions/#actions-cache-v1-v2-and-actions-toolkit-cache-package-closing-down
t3000-model-perf-tests / t3k CCL perf tests
Your workflow is using a version of actions/cache that is scheduled for deprecation, actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2. Please update your workflow to use either v3 or v4 of actions/cache to avoid interruptions. Learn more: https://github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions/#actions-cache-v1-v2-and-actions-toolkit-cache-package-closing-down
t3000-model-perf-tests / t3k LLM llama2-70B model perf tests
Your workflow is using a version of actions/cache that is scheduled for deprecation, actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2. Please update your workflow to use either v3 or v4 of actions/cache to avoid interruptions. Learn more: https://github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions/#actions-cache-v1-v2-and-actions-toolkit-cache-package-closing-down
t3000-model-perf-tests / t3k LLM mixtral model perf tests
Your workflow is using a version of actions/cache that is scheduled for deprecation, actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2. Please update your workflow to use either v3 or v4 of actions/cache to avoid interruptions. Learn more: https://github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions/#actions-cache-v1-v2-and-actions-toolkit-cache-package-closing-down
t3000-model-perf-tests / t3k LLM falcon7b model perf tests
Your workflow is using a version of actions/cache that is scheduled for deprecation, actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2. Please update your workflow to use either v3 or v4 of actions/cache to avoid interruptions. Learn more: https://github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions/#actions-cache-v1-v2-and-actions-toolkit-cache-package-closing-down
printing-out-smi-info-cleanup
Touching and printing out SMI info
disk-usage-after-startup
Disk usage is 25 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
printing-out-smi-info-cleanup
Touching and printing out SMI info
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
printing-out-smi-info-cleanup
Touching and printing out SMI info
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
printing-out-smi-info-cleanup
Touching and printing out SMI info
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
printing-out-smi-info-cleanup
Touching and printing out SMI info
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
printing-out-smi-info-cleanup
Touching and printing out SMI info
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
printing-out-smi-info-cleanup
Touching and printing out SMI info

Artifacts

Produced during runtime
Name Size
TTMetal_build_wormhole_b0
331 MB
TTMetal_build_wormhole_b0_profiler
338 MB
perf-report-csv--wormhole_b0--bare-metal
2.22 KB
perf-report-csv-CNN-wormhole_b0-resnet50-bare-metal
520 Bytes
perf-report-csv-LLM-wormhole_b0-falcon7b-bare-metal
850 Bytes
perf-report-csv-LLM-wormhole_b0-llama2-70b-bare-metal
410 Bytes
perf-report-csv-LLM-wormhole_b0-mixtral-bare-metal
611 Bytes