Tracking issue for performance improvements PRs

[bonzini]

Describe the bug
Meson should be faster. :)

1.7.0 has some microoptimizations and there are several open PRs

• use an unordered set for NinjaBuildElement (1.7.0)
• cache more file operations (1.7.0)
• cache version_compare results in compilers (1.7.0)
• speed up regex usage (1.7.0)

For 1.8:

• memoize result of canonicalize_filename (Speedup extract_objects #14109)
• speed up flatten_object_list/_determine_ext_objs (Speedup extract_objects #14109)
• speed up arglist (Speedup arglist #13879)
• speed up object creation (Speedup object creation #14091)
• cache target.get_id() at time of first use (Remove get_id() hotspot #14108)
• do not use pathlib in validate_within_subproject (Remove hot uses of pathlib #14110)
• speed up object_filename_from_source (Optimize object filename from source #14200)
• speed up is_windows et al. (micro-optimizations for is_windows() et al. #14206)
• speed up determine_rpath_dirs (Optimize determine_rpath_dirs #14207)

Other potential improvements:

• check slowdown from ABCMeta.__instancecheck__

To Reproduce
I'm using QEMU as the reproducer (mkdir build && ../configure --enable-rust) just because it produces a very large build.ninja, but using other similarly large projects (e.g. mesa, libvirt or systemd) should be feasible. This patch needs to be applied (or something like that) to use an external meson version:

diff --git a/configure b/configure
index 18336376bff..842c31ced6f 100755
--- a/configure
+++ b/configure
@@ -999,8 +999,10 @@ mkvenv="$python ${source_path}/python/scripts/mkvenv.py"
 
 # Finish preparing the virtual environment using vendored .whl files
 
-$mkvenv ensuregroup --dir "${source_path}/python/wheels" \
-     ${source_path}/pythondeps.toml meson || exit 1
+$python -m pip install $HOME/work/upstream/meson
+ls -l pyvenv/bin
+#$mkvenv ensuregroup --dir "${source_path}/python/wheels" \
+#     ${source_path}/pythondeps.toml meson || exit 1
 
 # At this point, we expect Meson to be installed and available.
 # We expect mkvenv or pip to have created pyvenv/bin/meson for us.
@@ -1973,7 +1975,8 @@ if test "$skip_meson" = no; then
   test "$plugins" = yes && meson_option_add "-Dplugins=true"
   test "$tcg" != enabled && meson_option_add "-Dtcg=$tcg"
   run_meson() {
-    NINJA=$ninja $meson setup "$@" "$PWD" "$source_path"
+    #NINJA=$ninja time $python -m cProfile -o stats7.out -m mesonbuild.mesonmain setup "$@" "$PWD" "$source_path"
+    NINJA=$ninja time $meson setup "$@" "$PWD" "$source_path"
   }
   eval run_meson $meson_options
   if test "$?" -ne 0 ; then

system parameters
I am testing on Fedora 41 (Python 3.13)

[bonzini]

Progress for QEMU:

1.6.0: 35.09 wall, 30.29 user, 4.64 sys
1.7.0: 31.32 wall, 26.74 user, 4.45 sys
c616f1ed: 30.92 wall, 26.23 user, 4.53 sys
pending: 26.85 wall, 22.33 user, 4.38 sys