rec: allow use of meson to create dist file #15076

omoerbeek · 2025-01-22T14:18:36Z

Short description

Draft as I need to double check the differences between the the autotool generated tarball and meson generated tarball.

meson dist -C yourbuilddir --formats bztar gets you the tarball.

Meson dist uses a partial checkout of the tree for subprojects (as rec is), so symlinks into the parent are dangling. There is a dist script to fix those.

To bootstrap, a few symlinks are replaced in git by actual files, as the project clause needs them before the dist script is run.

Next, after the dist dir is created, we replace the remaining symlinks into the rest of the tree by the actual files using some tar magic in the dist script.

I checked the produced tarball can be used to build both using meson and autotools.

version.sh script from meson docs: https://mesonbuild.com/Creating-releases.html#cement-a-version-obtained-from-vcs

Checklist

I have:

read the CONTRIBUTING.md document
compiled this code
tested this code
included documentation (including possible behaviour changes)
documented the code
added or modified regression test(s)
added or modified unit test(s)

coveralls · 2025-01-22T14:45:38Z

Pull Request Test Coverage Report for Build 13134799115

Details

0 of 0 changed or added relevant lines in 0 files are covered.
113 unchanged lines in 16 files lost coverage.
Overall coverage decreased (-8.2%) to 64.7%

Files with Coverage Reduction	New Missed Lines	%
pdns/packethandler.cc	1	72.32%
pdns/misc.cc	1	62.71%
pdns/backends/gsql/gsqlbackend.hh	1	97.71%
pdns/sstuff.hh	2	56.83%
pdns/stubresolver.cc	3	77.58%
pdns/iputils.cc	3	56.07%
pdns/dnsdistdist/dnsdist-lbpolicies.cc	4	70.51%
pdns/dnsdistdist/dnsdist.cc	4	68.73%
pdns/recursordist/test-syncres_cc1.cc	5	89.68%
pdns/rcpgenerator.cc	5	90.46%

Totals
Change from base Build 13133792161:	-8.2%
Covered Lines:	128215
Relevant Lines:	167067

💛 - Coveralls

omoerbeek · 2025-01-22T15:10:44Z

Dir compare shows that the meson tarball has some extra files.
autotools show a few generated files extra that are currenlty missing in the meson tarball.
I marked them to stand out

The question is if we want to have these files in the meson tarball as well. They would require ragel and python as abuild dep. I'd say include in tarball for ragel and nope for python generated files.

Only in meson: .gitignore
Only in meson: Caddyfile
Only in meson: README.md
Only in meson: autom4te.cache
Only in meson/build-aux: gen-version
*** Only in autotools: dnslabeltext.cc (handled)
Only in meson: docs
*** Only in autotools: effective_tld_names.dat
Only in meson: examples
Only in meson/ext/arc4random: .gitignore
Only in meson/ext/arc4random: meson.build
Only in meson/ext/json11: .gitignore
Only in meson/ext/json11: meson.build
Only in meson/ext/luawrapper: meson.build
Only in meson/ext/probds: .gitignore
Only in meson/ext/probds: meson.build
Only in meson/ext/protozero: CHANGELOG.md
Only in meson/ext/protozero: LICENSE.from_folly
Only in meson/ext/protozero: LICENSE.md
Only in meson/ext/protozero: meson.build
Only in meson/ext/yahttp: .gitignore
Only in meson/ext/yahttp: meson.build
Only in meson/ext/yahttp/yahttp: meson.build
Only in meson/m4: pdns_enable_p11kit.m4
Only in meson/m4: pdns_with_gnutls.m4
Only in meson: make-ext-symlinks.py
Only in meson: meson
Only in meson: meson-dist-script.sh
Only in meson: meson.build
Only in meson: meson_options.txt
Only in meson: recursor-lsan.supp
Only in meson: recursor-tsan.supp
Only in meson/settings: .gitignore
*** Only in autotools/settings: cxxsettings-generated.cc (not needed)
Only in meson/settings: meson.build
Only in meson/settings/rust: .gitignore
Only in meson/settings/rust: build_settings
Only in meson/settings/rust: meson.build
*** Only in autotools/settings/rust/src: lib.rs (not needed)
Only in meson: version.sh

eli-schwartz · 2025-01-23T05:56:33Z

Dir compare shows that the meson tarball has some extra files.

gitignore files can be excluded pretty easily: #15078

pdns/recursordist/meson-dist-script.sh

omoerbeek · 2025-01-24T08:56:24Z

Current state is that this shows there are a few files missing form the autotools tarball (files not essential for building, but still...). For the rest extra files in meson tarball are 1) files needed for meson to work 2) files that wil be skipped by #15078
or files added by this PR

Only in meson: .gitattributes
Only in meson: .gitignore
Only in meson/build-aux: gen-version
Only in meson/ext/arc4random: .gitignore
Only in meson/ext/arc4random: meson.build
Only in meson/ext/json11: .gitignore
Only in meson/ext/json11: meson.build
Only in meson/ext/luawrapper: meson.build
Only in meson/ext/probds: .gitignore
Only in meson/ext/probds: meson.build
Only in meson/ext/protozero: CHANGELOG.md
Only in meson/ext/protozero: LICENSE.from_folly
Only in meson/ext/protozero: LICENSE.md
Only in meson/ext/protozero: meson.build
Only in meson/ext/yahttp: .gitignore
Only in autotools/ext/yahttp: README.md
Only in meson/ext/yahttp: meson.build
Only in meson/ext/yahttp/yahttp: meson.build
Only in meson/m4: pdns_enable_p11kit.m4
Only in meson/m4: pdns_with_gnutls.m4
Only in meson: meson
Only in meson: meson-dist-script.sh
Only in meson: meson.build
Only in meson: meson_options.txt
Only in autotools: pubsuffix.cc (generated using shell script)
Only in autotools: rec-metrics-gen.h (generated using python script))
Only in autotools: rec-oids-gen.h (idem)
Only in autotools: rec-prometheus-gen.h (idem)
Only in autotools: rec-snmp-gen.h (idem)
Only in meson/settings: .gitignore
Only in autotools/settings: cxxsettings-generated.cc (idem)
Only in meson/settings: meson.build
Only in meson/settings/rust: .gitignore
Only in meson/settings/rust: build_settings
Only in meson/settings/rust: meson.build
Only in autotools/settings/rust/src: lib.rs (idem)
Only in meson: version.sh

…get VERSION)

…pendency

Co-authored-by: Peter van Dijk <peter.van.dijk@powerdns.com>

omoerbeek · 2025-01-27T10:06:32Z

And after a rebase to pick up #15078:

Only in meson/build-aux: gen-version
Only in meson/ext/arc4random: meson.build
Only in meson/ext/json11: meson.build
Only in meson/ext/luawrapper: meson.build
Only in meson/ext/probds: meson.build
Only in meson/ext/protozero: CHANGELOG.md
Only in meson/ext/protozero: LICENSE.from_folly
Only in meson/ext/protozero: LICENSE.md
Only in meson/ext/protozero: meson.build
Only in meson/ext/yahttp: meson.build
Only in meson/ext/yahttp/yahttp: meson.build
Only in meson/m4: pdns_enable_p11kit.m4
Only in meson/m4: pdns_with_gnutls.m4
Only in meson: meson
Only in meson: meson-dist-script.sh
Only in meson: meson.build
Only in meson: meson_options.txt
Only in autotools: pubsuffix.cc
Only in autotools: rec-metrics-gen.h
Only in autotools: rec-oids-gen.h
Only in autotools: rec-prometheus-gen.h
Only in autotools: rec-snmp-gen.h
Only in autotools/settings: cxxsettings-generated.cc
Only in meson/settings: meson.build
Only in meson/settings/rust: build_settings
Only in meson/settings/rust: meson.build
Only in autotools/settings/rust/src: lib.rs
Only in meson: version.sh

All mentioned Only in autotools files are generated by calling a python or shell script.

rgacogne · 2025-01-27T10:11:15Z

It looks quite good, nice work!

eli-schwartz · 2025-01-27T13:30:54Z

pdns/recursordist/.gitattributes

+recursor-lsan.supp export-ignore
+recursor-tsan.supp export-ignore


Is it worth keeping these in and setting it up so that meson sanitizer builds automatically use them?

At some point yes, but for now my goal is to produce a valid distfile. Extra features can be done later.

Habbie · 2025-02-04T10:58:38Z

gen-version now being a copy instead of a symlink is intended?

omoerbeek · 2025-02-04T11:07:19Z

gen-version now being a copy instead of a symlink is intended?

yes, as mentioned in the PR header: "To bootstrap, a few symlinks are replaced in git by actual files, as the project clause needs them before the dist script is run."

The root cause is the partial checkout, any symlink into the parent is dangling. The dist script fixes that, but for the project clause that's too late.

Habbie · 2025-02-04T11:08:44Z

Alright. I wonder if we should add a CI check that makes sure the multiple copies of the script are still identical.

Other than that, I'll approve :)

omoerbeek · 2025-02-04T11:14:47Z

I could do that in meson when building. That way any violator will get a notice asap and doesn't need to wait for CI.

omoerbeek · 2025-02-04T11:25:53Z

We already have several version in-tree: in ./build-aux, ./builder-support and ./builder

$ ls -l $(find . -name gen-version)
-rwxr-xr-x  1 otto  staff  1387 Jun 20  2022 ./build-aux/gen-version
-rwxr-xr-x  1 otto  staff  2819 Sep  5  2023 ./builder-support/gen-version
-rwxr-xr-x  1 otto  staff  2499 Dec 20  2022 ./builder/gen-version
lrwxr-xr-x  1 otto  staff    36 Jun 20  2022 ./pdns/dnsdistdist/build-aux/gen-version -> ../../../builder-support/gen-version
lrwxr-xr-x  1 otto  staff    36 Jun 20  2022 ./pdns/dnsdistdist/builder-support/gen-version -> ../../../builder-support/gen-version
lrwxr-xr-x  1 otto  staff    36 Jun 20  2022 ./pdns/recursordist/build-aux/gen-version -> ../../../builder-support/gen-version
-rwxr-xr-x  1 otto  staff  2819 Feb  4 12:17 ./pdns/recursordist/builder-support/gen-version

Habbie · 2025-02-04T11:26:47Z

We already have several version in-tree: in ./build-aux, ./builder-support and ./builder

oh no

anyway

omoerbeek · 2025-02-04T11:39:09Z

Check added. But we need to review the gen-version script situation.

Habbie · 2025-02-04T11:48:39Z

pdns/recursordist/meson-dist-script.sh

 # Get the dereffed symbolic links (the actual files being pointed to) from the source dir
 # Extract them over the existing symbolic links
-tar -C "$MESON_SOURCE_ROOT" -hcf - $symlinks | tar -xf - -C "$MESON_PROJECT_DIST_ROOT"
+tar -C "$MESON_SOURCE_ROOT" -Hcf - $symlinks | tar -xf - -C "$MESON_PROJECT_DIST_ROOT"


is the uppercase H intentional?

Nope, stray edit, thanks

omoerbeek changed the title ~~Rec meson dist~~ Rec use meson to create dist file Jan 22, 2025

omoerbeek marked this pull request as draft January 22, 2025 14:19

omoerbeek added the rec label Jan 22, 2025

omoerbeek force-pushed the rec-meson-dist branch from 07127ce to b1d114d Compare January 22, 2025 15:50

Habbie requested changes Jan 23, 2025

View reviewed changes

pdns/recursordist/meson-dist-script.sh Outdated Show resolved Hide resolved

omoerbeek force-pushed the rec-meson-dist branch from bd23dda to 6cb9deb Compare January 24, 2025 08:54

omoerbeek marked this pull request as ready for review January 24, 2025 08:57

omoerbeek requested a review from Habbie January 24, 2025 08:57

omoerbeek mentioned this pull request Jan 26, 2025

Force various files to be skipped when generating git archives #15078

Merged

omoerbeek and others added 10 commits January 27, 2025 09:13

Fix gen-version path

a6807a3

Add dist script

8756214

Replace a few symlinks with actual files

efdcbb5

replace symlinks with actual files

ec409ef

Fix version number for Meson build

323133f

Run autoconf -vi to create configure script (so that autotool builds …

1e256fe

…get VERSION)

Include dnslabeltext.cc in trabal, otherwise ragel becomes a build de…

3b87737

…pendency

Typo in comment

805596a

Co-authored-by: Peter van Dijk <peter.van.dijk@powerdns.com>

Update requirements, fetch effective_tld_names.dat

28e0d7a

Skip a few files and dirs in the meson dist tarball

6f2c896

omoerbeek force-pushed the rec-meson-dist branch from 6cb9deb to 6f2c896 Compare January 27, 2025 09:27

eli-schwartz reviewed Jan 27, 2025

View reviewed changes

omoerbeek changed the title ~~Rec use meson to create dist file~~ rec: allow use of meson to create dist file Feb 4, 2025

Habbie approved these changes Feb 4, 2025

View reviewed changes

Habbie reviewed Feb 4, 2025

View reviewed changes

Add a check to see if gen-version still is up-to-date

4b02618

omoerbeek force-pushed the rec-meson-dist branch from 464ae07 to 4b02618 Compare February 4, 2025 11:50

omoerbeek merged commit 7c4e395 into PowerDNS:master Feb 4, 2025
83 checks passed

omoerbeek deleted the rec-meson-dist branch February 4, 2025 12:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rec: allow use of meson to create dist file #15076

rec: allow use of meson to create dist file #15076

omoerbeek commented Jan 22, 2025 •

edited

Loading

coveralls commented Jan 22, 2025 •

edited

Loading

omoerbeek commented Jan 22, 2025 •

edited

Loading

eli-schwartz commented Jan 23, 2025

omoerbeek commented Jan 24, 2025 •

edited

Loading

omoerbeek commented Jan 27, 2025 •

edited

Loading

rgacogne commented Jan 27, 2025

eli-schwartz Jan 27, 2025

omoerbeek Jan 27, 2025

Habbie commented Feb 4, 2025

omoerbeek commented Feb 4, 2025

Habbie commented Feb 4, 2025

omoerbeek commented Feb 4, 2025

omoerbeek commented Feb 4, 2025

Habbie commented Feb 4, 2025

omoerbeek commented Feb 4, 2025

Habbie Feb 4, 2025

omoerbeek Feb 4, 2025

		recursor-lsan.supp export-ignore
		recursor-tsan.supp export-ignore

rec: allow use of meson to create dist file #15076

rec: allow use of meson to create dist file #15076

Conversation

omoerbeek commented Jan 22, 2025 • edited Loading

Short description

Checklist

coveralls commented Jan 22, 2025 • edited Loading

Pull Request Test Coverage Report for Build 13134799115

Details

💛 - Coveralls

omoerbeek commented Jan 22, 2025 • edited Loading

eli-schwartz commented Jan 23, 2025

omoerbeek commented Jan 24, 2025 • edited Loading

omoerbeek commented Jan 27, 2025 • edited Loading

rgacogne commented Jan 27, 2025

eli-schwartz Jan 27, 2025

Choose a reason for hiding this comment

omoerbeek Jan 27, 2025

Choose a reason for hiding this comment

Habbie commented Feb 4, 2025

omoerbeek commented Feb 4, 2025

Habbie commented Feb 4, 2025

omoerbeek commented Feb 4, 2025

omoerbeek commented Feb 4, 2025

Habbie commented Feb 4, 2025

omoerbeek commented Feb 4, 2025

Habbie Feb 4, 2025

Choose a reason for hiding this comment

omoerbeek Feb 4, 2025

Choose a reason for hiding this comment

omoerbeek commented Jan 22, 2025 •

edited

Loading

coveralls commented Jan 22, 2025 •

edited

Loading

omoerbeek commented Jan 22, 2025 •

edited

Loading

omoerbeek commented Jan 24, 2025 •

edited

Loading

omoerbeek commented Jan 27, 2025 •

edited

Loading