Use tags in docs building to split up requirements #13168

matthewhughes934 · 2025-01-15T19:06:14Z

The goal is to allow a build of just the man pages with a minimal set of dependencies, like:

sphinx-build \
    --tag man \
    -c docs/html \
    -d docs/build/doctrees/man \
    -b man \
    docs/man \
    docs/build/man

This is for the sake of people distributing pip who want to build the man pages (but not the HTML ones) along side the application.

This is an alternative to the approach proposed in[1] and similarly addresses[2]

Link: #12900 [1]
Link: #12881 [2]

matthewhughes934 · 2025-01-15T19:26:25Z

🤔 I need to tell readthedocs about --tag html for it to correctly build

EDIT: quick hack e06b1cf, I don't see any out of the box support for them (I found one issue: readthedocs/readthedocs.org#4603)

matthewhughes934 · 2025-01-15T20:02:21Z

docs/html/conf.py

 ]

+# 'tags' is a special object handled by sphinx
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-tags
+if "html" in tags or "READTHEDOCS" in os.environ:  # type: ignore[name-defined] # noqa: F821


The linters get a bit upset with using tags like this, and complain about each nice, but maybe more usage makes things more clear, e.g.:

_tags: Tags = tags # type: ignore[name-defined] # noqa: F821 if "READTHEDOCS" in os.environ: _tags.add("html") if "html" in _tags:

I think I've also seen (and commented on) some tag use by @AA-Turner that was accessing them through the Sphinx application object in setup(). Adam — do you remember of such an example? I can't seem to find a project that uses said approach…

Found it! I was thinking about those projects linked in sphinx-doc/sphinx#12490. Looks like the tags are accessible via app.tags.tags in the extension context...

The object is injected into conf.py. Please only use tags or app.tags, we have deprecated app.tags.tags!

Yeah, I know it's injected. I was just thinking about the postponed evaluation case. So I'd go for calling app.setup_extension() based on the state of app.tags or the currently selected builder…

ichard26 · 2025-01-15T21:31:51Z

@eli-schwartz would gating the HTML build behind a Sphinx tag --tag html work for you? Presumably, you can modify the sphinx-build invocation to pass custom flags, but I don't want to assume!

eli-schwartz · 2025-01-15T22:03:19Z

This is a neat trick that I didn't know sphinx could do!

Presumably, you can modify the sphinx-build invocation to pass custom flags, but I don't want to assume!

We can, yes. It's already necessary in order to pass -c to override the conf.py directory.

That being said, it looks like this actually defaults to omitting the html logic by default (it doesn't matter whether you use --tag man) and only doing something different when building html. Was that the intention? You could alternatively do if "man" not in tags to require people to manually opt in to this by passing --tag man. I'm not sure how much it matters since noxfile.py will always do the correct invocation and anyone manually running sphinx-build is presumably in "you must check how noxfile.py did it and ensure your usage is aligned" mode.

webknjaz · 2025-01-16T03:08:12Z

docs/html/conf.py

 ]

+# 'tags' is a special object handled by sphinx
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-tags
+if "html" in tags or "READTHEDOCS" in os.environ:  # type: ignore[name-defined] # noqa: F821


FWIW, I think the idiomatic check is tags.has('html').

Also note that it's not just html but htmldir too.

FWIW, I think the idiomatic check is tags.has('html').

I was just following the docs on that one https://www.sphinx-doc.org/en/master/usage/configuration.html#project-tags (look to be a recent update <1 year ago sphinx-doc/sphinx#12490)

Ah, makes sense.

webknjaz · 2025-01-16T03:14:31Z

docs/html/conf.py

+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-tags
+if "html" in tags or "READTHEDOCS" in os.environ:  # type: ignore[name-defined] # noqa: F821
+    # html specific deps
+    extensions.extend(


It occurred to me that maybe if the extension loading was to be moved to the setup() entry point, the builder name would already be known by then and so this might not even have to use tags but would rely on the requested builders.

Another approach I used in the past, in a little different context, was scanning for the builder args in sys.argv: sphinx-contrib/spelling#55 (comment) / cherrypy/cheroot@63097fb#diff-85933aa74a2d66c3e4dcdf7a9ad8397f5a7971080d34ef1108296a7c6b69e7e3R12-R15.

webknjaz · 2025-01-16T03:18:42Z

🤔 I need to tell readthedocs about --tag html for it to correctly build

That's one of the reasons I've migrated my RTD configs to just call the project's main workflow tool: https://github.com/ansible/awx-plugins/blob/0d569b5/.readthedocs.yaml#L27-L29. In my case it's tox, but in pip's case it'd be nox. This brings RTD and local+CI builds even closer in what they produce.

noxfile.py

webknjaz · 2025-01-16T03:26:43Z

docs/html/conf.py

-    "sphinx.ext.extlinks",
-    "sphinx.ext.intersphinx",
-    # our extensions
+    # extensions common to 'man' and 'html' builds


I'd also recommend keeping the order of extensions: first-party -> third-party -> local. Just like with regular imports.

AA-Turner · 2025-01-16T15:07:28Z

docs/html/conf.py

-    # first-party extensions
-    "sphinx.ext.autodoc",
-    "sphinx.ext.todo",
-    "sphinx.ext.extlinks",
-    "sphinx.ext.intersphinx",


These are all built-in so there's no value moving to conditional (and it makes things harder to reason about)

I think there might be value in skipping some code paths, though.

AA-Turner · 2025-01-16T15:09:49Z

docs/html/conf.py

+            "sphinx_copybutton",
+            "sphinx_inline_tabs",


These two seem to be the only genuine 'HTML-only' extensions. Perhaps the condition might be better expressed as 'man' not in tags?

You could also look at the Python docs, where we use a pattern to check if an extension is importable, and if so add it to extensions. Eg

import importlib.util if importlib.util.find_spec(...): extensions.append(...)

Perhaps the condition might be better expressed as 'man' not in tags?

👍 this was also suggested above (#13168 (comment)) and I agree it makes this clearer (and simplifies the code!) so trying that: 033c31c

AA-Turner · 2025-01-16T15:12:39Z

docs/html/conf.py

 ]

+# 'tags' is a special object handled by sphinx
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-tags
+if "html" in tags or "READTHEDOCS" in os.environ:  # type: ignore[name-defined] # noqa: F821


The object is injected into conf.py. Please only use tags or app.tags, we have deprecated app.tags.tags!

webknjaz · 2025-01-17T15:44:09Z

docs/html/conf.py

+if "man" not in tags:  # type: ignore[name-defined] # noqa: F821
+    # extensions not needed for building man pages
+    extensions.extend(
+        (
+            # first-party extensions
+            "sphinx.ext.autodoc",
+            "sphinx.ext.todo",
+            "sphinx.ext.extlinks",
+            "sphinx.ext.intersphinx",
+            # third-party extensions
+            "myst_parser",
+            "sphinx_copybutton",
+            "sphinx_inline_tabs",
+            "sphinxcontrib.towncrier",
+        ),
+    )


Would this be cleaner?

Suggested change

if "man" not in tags: # type: ignore[name-defined] # noqa: F821

# extensions not needed for building man pages

extensions.extend(

(

# first-party extensions

"sphinx.ext.autodoc",

"sphinx.ext.todo",

"sphinx.ext.extlinks",

"sphinx.ext.intersphinx",

# third-party extensions

"myst_parser",

"sphinx_copybutton",

"sphinx_inline_tabs",

"sphinxcontrib.towncrier",

),

)

IS_MAN_BUILD = "man" in tags # type: ignore[name-defined] # noqa: F821

extensions += [] if IS_MAN_BUILD else [

# first-party extensions

"sphinx.ext.autodoc",

"sphinx.ext.todo",

"sphinx.ext.extlinks",

"sphinx.ext.intersphinx",

# third-party extensions

"myst_parser",

"sphinx_copybutton",

"sphinx_inline_tabs",

"sphinxcontrib.towncrier",

]

ichard26

Looks good! I'll wait for other people to comment, however.

ichard26 · 2025-01-27T00:00:36Z

Oh yeah, we should probably add a docs news entry to let our redistributors know about this change.

The goal is to allow a build of just the `man` pages with a minimal set of dependencies, like: sphinx-build \ --tag man \ -c docs/html \ -d docs/build/doctrees/man \ -b man \ docs/man \ docs/build/man This is for the sake of people distributing `pip` who want to build the man pages (but not the HTML ones) along side the application. This is an alternative to the approach proposed in[1] and similarly addresses[2] Link: pypa#12900 [1] Link: pypa#12881 [2]

matthewhughes934 · 2025-01-29T20:00:36Z

Oh yeah, we should probably add a docs news entry to let our redistributors know about this change.

Done, and squashed my fixup commits in with 782c3a2

There have been some good suggestions on this PR about large restructurings of the conf.py that could be useful, but given I'm no sphinx expert I decided to lean on the side of keeping this PR small/simple. Maybe it's wroth a separate issue to discuss some broader improves that could simplify the whole setup.

ichard26 · 2025-02-16T19:25:15Z

@pradyunsg are you interested in/have time for reviewing this documentation PR? I'd like to merge it, but I also know this is your domain of experience, so I don't want to step on your toes without your permission.

webknjaz · 2025-02-17T01:22:41Z

@ichard26 when I last mentioned this PR to Pradyun (3-4 weeks ago), he wanted to review it and wasn't sure if keeping manpages in the project was even reasonable. I also wanted to try sending another alternative PR, but never found time :)

eli-schwartz · 2025-02-17T01:34:24Z

I don't see the downside in providing manpages... I have been looking forward for quite some time to the ability to read about what pip options are available in an offline manner using my system documentation viewer. It is a significantly better (more pleasant and readable, faster) experience than pip XXX --help since a documentation viewer has the opportunity to handle layout much better including semantic tagging, mentioning environment variables, using paragraph-based layout, and providing text searching within the pager context (try to search for a specific keyword you are sure is relevant to your goal, in the output of --help, you will quickly realize it's just not a pleasant experience).

webknjaz · 2025-02-20T21:47:17Z

@eli-schwartz the thing is that they aren't currently integrated into CLI. Only some downstreams would use them up until now. What you're suggesting is what git and pipx do IIRC. But that would be a feature request (with the corresponding maintenance cost).

eli-schwartz · 2025-02-20T22:24:21Z

No, I'm not asking for pip help install to open a manpage viewer. There is nearly no software ever that does this. Git is the most extreme of outliers.

Vast quantities of software do provide some mechanism to install manpages so that man pip-install works. For example, CPython itself does so!

It is significantly more beautiful to read than python --help quickly followed by python --help-all, and provides tons more information than the CLI does too. It's even syntax highlighted (manpages have syntactically meaningful elements to differentiate flag names or inline code quotes within a paragraph of prose description). It's installed by default by CPython's installer (make install).

I have it locally set up for pip, but this PR would make it significantly easier to promote it to other Gentoo users (by modifying the packaging of pip to start running sphinx) as it doesn't require adding novel additional software to Gentoo (e.g. sphinxcontrib.towncrier).

There is no expectation that pip should install manpages on its own. Wheels do not possess this capability, that is one of the major advantages of a linux distribution -- the ability to integrate at all levels of the operating system. I am simply trying to make it a bit easier to do so.

I'm not the only one. Debian ships manpages: https://salsa.debian.org/python-team/packages/python-pip/-/commit/88d10f619bc2398b36f44768e90a560a4089912f

Commit 88d10f61 authored 11 months ago by Stefano Rivera

Build manpages from pip's Sphinx documentation. This builds a fleet of manpages, instead of one, but it's at least up to date. (Closes: #1061184)

Including a local patch to "Disable sphinxcontrib-towncrier" by commenting it out.

matthewhughes934 marked this pull request as ready for review January 15, 2025 19:06

matthewhughes934 mentioned this pull request Jan 15, 2025

Add separate conf.py for the different docs types #12900

Open

matthewhughes934 commented Jan 15, 2025

View reviewed changes

webknjaz reviewed Jan 16, 2025

View reviewed changes

noxfile.py Show resolved Hide resolved

webknjaz reviewed Jan 16, 2025

View reviewed changes

AA-Turner reviewed Jan 16, 2025

View reviewed changes

matthewhughes934 force-pushed the experiment-tags-for-split-builds branch from 82da058 to 033c31c Compare January 16, 2025 19:23

webknjaz reviewed Jan 17, 2025

View reviewed changes

ichard26 requested a review from pradyunsg January 18, 2025 20:06

ichard26 approved these changes Jan 26, 2025

View reviewed changes

matthewhughes934 force-pushed the experiment-tags-for-split-builds branch from 10ef204 to 782c3a2 Compare January 29, 2025 19:57

psf-chronographer bot added the bot:chronographer:provided label Jan 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use tags in docs building to split up requirements #13168

Use tags in docs building to split up requirements #13168

matthewhughes934 commented Jan 15, 2025

matthewhughes934 commented Jan 15, 2025 •

edited

Loading

matthewhughes934 Jan 15, 2025

webknjaz Jan 16, 2025

webknjaz Jan 16, 2025

AA-Turner Jan 16, 2025

webknjaz Jan 16, 2025

ichard26 commented Jan 15, 2025

eli-schwartz commented Jan 15, 2025

webknjaz Jan 16, 2025

webknjaz Jan 16, 2025

matthewhughes934 Jan 16, 2025

webknjaz Jan 17, 2025

webknjaz Jan 16, 2025

webknjaz commented Jan 16, 2025

webknjaz Jan 16, 2025

AA-Turner Jan 16, 2025

webknjaz Jan 16, 2025

AA-Turner Jan 16, 2025

matthewhughes934 Jan 16, 2025

AA-Turner Jan 16, 2025

webknjaz Jan 17, 2025 •

edited

Loading

ichard26 left a comment

ichard26 commented Jan 27, 2025

matthewhughes934 commented Jan 29, 2025

ichard26 commented Feb 16, 2025

webknjaz commented Feb 17, 2025

eli-schwartz commented Feb 17, 2025

webknjaz commented Feb 20, 2025

eli-schwartz commented Feb 20, 2025

Use tags in docs building to split up requirements #13168

Are you sure you want to change the base?

Use tags in docs building to split up requirements #13168

Conversation

matthewhughes934 commented Jan 15, 2025

matthewhughes934 commented Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ichard26 commented Jan 15, 2025

eli-schwartz commented Jan 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

ichard26 left a comment

Choose a reason for hiding this comment

ichard26 commented Jan 27, 2025

matthewhughes934 commented Jan 29, 2025

ichard26 commented Feb 16, 2025

webknjaz commented Feb 17, 2025

eli-schwartz commented Feb 17, 2025

webknjaz commented Feb 20, 2025

eli-schwartz commented Feb 20, 2025

matthewhughes934 commented Jan 15, 2025 •

edited

Loading

webknjaz Jan 17, 2025 •

edited

Loading