v1.7.3-pre

[ggerganov]

Overview

Massive performance improvements for the Metal backend, especially for beams > 1. Especially for quantized models.
Setting as "pre-release" since there have been major changes to the build system (now using CMake) and I wan't to gather some feedback about how well the project builds now on various platforms. Please leave comments in the discussion to help fix any remaining issues before the official release.

CPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M2 Ultra	Metal	tiny	1	1	7.90	1.26	0.35	0.01	ed733e8
M2 Ultra	Metal	tiny-q5_0	1	1	8.44	1.23	0.36	0.01	ed733e8
M2 Ultra	Metal	tiny-q5_1	1	1	8.26	1.27	0.37	0.01	ed733e8
M2 Ultra	Metal	tiny-q8_0	1	1	8.03	1.21	0.35	0.01	ed733e8
M2 Ultra	Metal	base	1	1	13.77	1.80	0.42	0.02	ed733e8
M2 Ultra	Metal	base-q5_0	1	1	15.02	1.72	0.42	0.02	ed733e8
M2 Ultra	Metal	base-q5_1	1	1	14.93	1.74	0.42	0.02	ed733e8
M2 Ultra	Metal	base-q8_0	1	1	14.26	1.68	0.41	0.02	ed733e8
M2 Ultra	Metal	small	1	1	39.76	3.54	0.85	0.05	ed733e8
M2 Ultra	Metal	small-q5_0	1	1	45.07	3.47	0.87	0.05	ed733e8
M2 Ultra	Metal	small-q5_1	1	1	44.82	3.49	0.87	0.05	ed733e8
M2 Ultra	Metal	small-q8_0	1	1	41.79	3.30	0.84	0.05	ed733e8
M2 Ultra	Metal	medium	1	1	106.73	7.28	1.78	0.11	ed733e8
M2 Ultra	Metal	medium-q5_0	1	1	124.43	6.63	1.83	0.12	ed733e8
M2 Ultra	Metal	medium-q5_1	1	1	124.19	6.70	1.84	0.12	ed733e8
M2 Ultra	Metal	medium-q8_0	1	1	113.88	6.52	1.75	0.11	ed733e8
M2 Ultra	Metal	medium-dis	1	1	94.97	0.97	0.22	0.01	ed733e8
M2 Ultra	Metal	large-v2	1	1	193.33	10.53	2.65	0.20	ed733e8
M2 Ultra	Metal	large-v2-q5_0	1	1	229.22	9.52	2.72	0.23	ed733e8
M2 Ultra	Metal	large-v2-q5_1	1	1	229.40	9.62	2.73	0.23	ed733e8
M2 Ultra	Metal	large-v2-q8_0	1	1	207.30	9.36	2.59	0.21	ed733e8
M2 Ultra	Metal	large-v2-dis	1	1	171.43	1.09	0.25	0.02	ed733e8
M2 Ultra	Metal	large-v3-turbo	1	1	173.45	1.73	0.41	0.03	ed733e8
M2 Ultra	Metal	large-v3-turbo-q5_0	1	1	205.52	1.52	0.42	0.04	ed733e8
M2 Ultra	Metal	large-v3-turbo-q8_0	1	1	185.90	1.48	0.40	0.03	ed733e8

What's Changed

• sync : ggml by @ggerganov in sync : ggml #2573
• ruby : Follow source tree change by @KitaitiMakoto in ruby : Follow source tree change #2580
• Add q8_0 models to download-ggml-model.sh by @mrienstra in Add q8_0 models to download-ggml-model.sh #2589
• ruby : Add low-level methods to transcribe by @KitaitiMakoto in ruby : Add low-level methods to transcribe #2585
• sync : ggml by @ggerganov in sync : ggml #2608

Full Changelog: v1.7.2...v1.7.3-pre

---

This discussion was created from the release v1.7.3-pre.

[zapsleep]

Xcode 16.1 and 16.2RC:
Swift Package Manager gives a warning couldn't find pc file for whisper when updating the package to ed733e8.
Then when trying to Build, it fails with:
'whisper.h' file not found with <angled> include; use "quotes" instead
in whisper/Sources/whisper/whisper.h
and every file with import whisper doesn't see imported functions and types.

[ggerganov]

The SPM package now requires to build the whisper.cpp binaries separately using CMake. The Github workflow demonstrates one way to do that:

whisper.cpp/.github/workflows/build.yml

Lines 522 to 566 in 6aa1d7b

	ios-xcode-build:
	runs-on: macos-latest
	strategy:
	matrix:
	build: [Release]
	steps:
	- name: Checkout code
	uses: actions/checkout@v4
	- name: Configure
	run: |
	cp models/for-tests-ggml-base.en.bin models/ggml-base.en.bin
	mkdir models/ggml-base.en-encoder.mlmodelc
	- name: Build
	id: cmake_build
	run: |
	sysctl -a
	mkdir build
	cd build
	cmake -G Xcode .. \
	-DGGML_METAL_USE_BF16=ON \
	-DGGML_METAL_EMBED_LIBRARY=ON \
	-DWHISPER_BUILD_EXAMPLES=OFF \
	-DWHISPER_BUILD_TESTS=OFF \
	-DWHISPER_BUILD_SERVER=OFF \
	-DCMAKE_SYSTEM_NAME=iOS \
	-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
	-DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
	cmake --build . --config Release -j $(sysctl -n hw.logicalcpu) -- CODE_SIGNING_ALLOWED=NO
	sudo cmake --install . --config Release
	- name: xcodebuild for swift package
	id: xcodebuild
	run: |
	xcodebuild -scheme whisper-Package -destination 'generic/platform=iOS'
	#- name: Build objc example
	# run: xcodebuild -project examples/whisper.objc/whisper.objc.xcodeproj -scheme whisper.objc -configuration ${{ matrix.build }} -sdk iphoneos build
	- name: Build swiftui example
	run: xcodebuild -project examples/whisper.swiftui/whisper.swiftui.xcodeproj -scheme WhisperCppDemo -configuration ${{ matrix.build }} -sdk iphoneos CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= -destination 'generic/platform=iOS' build

This is one of the main changes in the build process for this release. It would be great if we get some help with providing build instructions that are easy to follow and work for various deployment targets. For more info, also check ggerganov/llama.cpp#10747 (comment)

[itsthisjustin]

I'm having all kinds of issues getting this working in my project but even after following the build commands here I get this error in the example project:

Building for 'iOS', but linking in dylib (/usr/local/lib/libggml.dylib) built for 'macOS'

[zapsleep]

I was having the same issue at first, cmake generated 'whisper.cpp.xcodeproj' configured for macOS for some reason. Maybe I wasn't careful with cmake flags.
It seems, after the xcodeproj is generated, calling cmake -G Xcode .. doesn't regenerate it even with new parameters. Maybe I'm wrong, but in my case after it was generated with macOS, it doesn't change to iOS with cmake -G Xcode <...> -D CMAKE_SYSTEM_NAME=iOS <...>.
What helped was removing 'whisper.cpp.xcodeproj' generated by cmake (I remove the whole /build dir completely) and generate again. But this time ensure you pass -D CMAKE_SYSTEM_NAME=iOS. For me it works every time: remove and then generate with 'iOS' or 'Darwin' (for macOS).
Then build and install again.

[itsthisjustin]

Any advice on how to do this for a project that has both iOS and macOS builds in it? I'd have to have to do this every single time I want to build for one or the other

[zapsleep]

Not much, unfortunately ):
As I understand it, you just build dylibs once and then just link to them. No need to regenerate on every build.
But now the problems are:

1. Dynamic libraries need to be shipped with the app. Installing them into /usr/local/lib wouldn't do. And iOS apps (not sure about macOS) have specific requirements about dylibs locations inside the bundle. I had my share of trouble with @rpath evaluation for that reason. (today's XCode doesn't look in /usr/local/lib anymore as I understand)
   I got the impression that Apple insists on packing dylibs as Frameworks and including them into bundles like that. It removes the trouble with runtime search paths completely. You can add 'Copy Files' phase to your XCode target, choose 'Framework' destination and add all the dylibs there.
2. We now have dylibs for iOS, iOS simulator and macOS (maybe even Catalyst). I guess they should be linked conditionally depending on target platform. Another way would be making fat libraries.

It becomes cumbersome. But this all is just a state of my understanding. I may be wrong and overcomplicating, and eventually someone with better knowledge of linking dylibs in XCode comes in and show an obvious easy way (: