Generating Python API docs

[StefanHabel]

This PR adds support for generating Python API documentation in HTML format using Sphinx from the MaterialX Python modules that are built in the lib/ directory.

A new CMake build option named MATERIALX_BUILD_PYTHON_DOCS allows developers to turn generating Python API documentation on.

When MATERIALX_BUILD_PYTHON_DOCS is set to ON, MATERIALX_BUILD_PYTHON is set to ON as well, ensuring we have Python modules for which to build the Python API docs.

The core functionality of generating Python API documentation lives in a new directory named documents/PythonAPI/. It is controlled with a new CMakeLists.txt file in that directory, which defines a new target named MaterialXDocsPython, similar to the existing target MaterialXDocs that generates API documentation for the MaterialX C++ API.

Revised docstrings for modules were recently added in the following separate PR:

• Improvements to PyMaterialX docstrings #2039

Revised docstrings for classes are added in the following separate PRs:

• Docstrings for PyMaterialXCore classes. #2051
• Docstrings for PyMaterialXFormat classes. #2061
• Docstrings for PyMaterialXGenShader classes. #2063
• Docstrings for PyMaterialXGen* classes. #2064
• Docstrings for PyMaterialXRender classes. #2065
• Docstrings for PyMaterialXRender* classes. #2066

Revised docstrings for methods and functions are to be added in subsequent PRs separately.

Documentation in markdown format from the existing DeveloperGuide is integrated into the new Python API documentation by way of symlinking the four main .md files into the documents/PythonAPI/sources/ directory from which Sphinx generates the resulting HTML documentation.

To build the docs from scratch on macOS, I've used the following build script, naming it build.sh in the MaterialX checkout directory:

#!/usr/bin/env tcsh

echo build.sh: Updating Git submodules...
git submodule update --init --recursive

# Create and activate a virtual environment for installing Python dependencies
python3 -m venv /tmp/venv
source /tmp/venv/bin/activate.csh

echo build.sh: Installing dependencies...
python3 -m ensurepip
python3 -m pip install myst_parser  # https://pypi.org/project/myst-parser/

echo build.sh: Making build directory and changing into it...
mkdir build
cd build

echo build.sh: Configuring...
cmake .. \
    --fresh \
    -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk \
    -DMATERIALX_BUILD_PYTHON=ON \
    -DMATERIALX_BUILD_VIEWER=ON \
    -DMATERIALX_BUILD_GRAPH_EDITOR=ON \
    -DMATERIALX_BUILD_DOCS=ON \
    -DMATERIALX_BUILD_PYTHON_DOCS=ON \
    -DMATERIALX_BUILD_TESTS=ON \
&& \
echo build.sh: Building... \
&& \
cmake --build . -j 8 \
&& \
echo build.sh: Building target MaterialXDocs... \
&& \
cmake --build . --target MaterialXDocs \
&& \
echo build.sh: Building target MaterialXDocsPython... \
&& \
cmake --build . --target MaterialXDocsPython \
&& \
afplay /System/Library/Sounds/Blow.aiff

# Deactivate the virtual environment
deactivate

The build output currently ends with the following messages:

The parsed MaterialX Python API consists of:
    * 11 modules
    * 48 functions
    * 141 classes
    * 67 attributes
    * 1178 methods
    * 6 exception types

WARNING: 48 functions look like they do not have docstrings yet.
WARNING: 1020 methods look like they do not have docstrings yet.
WARNING: 32 functions look like their parameters have not all been named using `py::arg()`.
WARNING: 500 methods look like their parameters have not all been named using `py::arg()`.
build succeeded, 4 warnings.

The HTML pages are in ..
[100%] Built target MaterialXDocsPython

---

Screenshot showing the new Python API documentation main page:

Screenshot showing the modified Viewer.md contents from #2037 in the context of the Python API documentation:

Screenshot showing the (currently) incomplete PyMaterialXCore docs (to be amended in subsequent PRs):

Split from #1567.

Update #342.

[StefanHabel]

Demo of generated MaterialX Python API Documentation now available here: https://stefanhabel.github.io

Source repo for the live demo available here: https://github.com/StefanHabel/StefanHabel.github.io

[StefanHabel]

I've rebased the branch of this PR on top of main, and have added another commit to revert the changes in PyMaterialX.h: d76eb5a

This follows a discussion on #2039 in which the proposed new PYMATERIALX_DOCSTRING was first used.

The changes in this PR now stand as +1,174 −0.

[StefanHabel]

I've added a couple of commits to replace the previous Alphabetical Index on a module's documentation page, which did not provide links to module functions, with alphabetical listings of Classes, Exception Types, and Module Functions, each linking to either a separate documentation page or to a section on the same page:

Before	After

The only warnings that are now being displayed by Sphinx at the end of building the Python API documentation are those that we add ourselves to highlight missing documentation:

WARNING: 48 functions look like they do not have docstrings yet.
WARNING: 1019 methods look like they do not have docstrings yet.
WARNING: 32 functions look like their parameters have not all been named using `py::arg()`.
WARNING: 499 methods look like their parameters have not all been named using `py::arg()`.

-build succeeded, 149 warnings.
+build succeeded, 4 warnings.

I've updated the generated HTML pages in the live demo here: https://stefanhabel.github.io/

Example of a class with attribute names and other properties:
https://stefanhabel.github.io/generated/PyMaterialXCore.Backdrop.html#PyMaterialXCore.Backdrop

The summary of the MaterialX Python API is now stated as follows:

The parsed MaterialX Python API consists of:
    * 11 modules
    * 48 functions
    * 139 classes
    * 59 attributes
    * 1175 methods
    * 6 exception types

We can add the missing documentation in subsequent separate PRs dedicated to individual modules and classes.

[jstone-lucasfilm]

@StefanHabel I still need to test this locally using Sphinx, but overall this proposal is looking really good to me!

[jstone-lucasfilm]

@StefanHabel Today I had a chance to try out this latest work with a local installation of Sphinx, but I encountered the following error when building the new MaterialXDocsPython project:

Could not import extension myst_parser (exception: No module named 'myst_parser')

Should we call out the need to install myst_parser in addition to sphinx, or will most developers assume both are installed together?

[StefanHabel]

@StefanHabel Today I had a chance to try out this latest work with a local installation of Sphinx, but I encountered the following error when building the new MaterialXDocsPython project:

Could not import extension myst_parser (exception: No module named 'myst_parser')

Should we call out the need to install myst_parser in addition to sphinx, or will most developers assume both are installed together?

Thanks for giving this a go!

Yeah, probably good to highlight as a dependency in the same way as highlighting that Sphinx is required.

I've taken a stab at this in cb47402.

[StefanHabel]

I've added another commit (f649cf1) to make the build fail early if myst-parser is not available, e.g.:

[...]
-- Found Sphinx: /tmp/venv/bin/sphinx-build
WARNING: Package(s) not found: myst-parser
CMake Error at documents/PythonAPI/CMakeLists.txt:11 (message):
  The `myst-parser` Python package does not appear to be available for
  `/tmp/venv/bin/python3.12`.

  Consider installing it via `pip install myst-parser`.

  PyPI: https://pypi.org/project/myst-parser/


-- Configuring incomplete, errors occurred!

[jstone-lucasfilm]

@StefanHabel Now that we're wrapping up work for the upcoming v1.39.2 release, I wanted to check in to see what remaining work is needed to merge this proposal. Here is the Python API page that I get with the latest code in the PR, making no modifications to my display settings in Chrome:

I see a few sections where the color scheme doesn't read clearly in light mode, and wanted to see whether this could potentially be resolved by leveraging the same color scheme as our C++ API documentation.

What are your thoughts?

[StefanHabel]

@StefanHabel Now that we're wrapping up work for the upcoming v1.39.2 release, I wanted to check in to see what remaining work is needed to merge this proposal. Here is the Python API page that I get with the latest code in the PR, making no modifications to my display settings in Chrome:

I see a few sections where the color scheme doesn't read clearly in light mode, and wanted to see whether this could potentially be resolved by leveraging the same color scheme as our C++ API documentation.

What are your thoughts?

Thanks for flagging this @jstone-lucasfilm!

I can reproduce the issue by switching to the Light appearance in macOS System Settings.

This likely needs a tweak or two to our custom.css in the branch of this PR, which is applied on top of the standard alabaster.css and the doxygen-awesome.css that the C++ API Documentation uses.

I'll have a look at applying tweaks tomorrow (currently in Germany for the holidays)

[StefanHabel]

I've applied some tweaks in 9ffd9e4

Would you mind trying again @jstone-lucasfilm?

Screen recording of macOS switching from the Dark appearance to the Light appearance:

Screen.Recording.2024-12-17.at.17.52.38.mov

[jstone-lucasfilm]

Thanks for the updates, @StefanHabel, and I'll give this a try!

[jstone-lucasfilm]

The text colors look good in latest, thanks @StefanHabel!

Given the complexity of this PR, I'm thinking that we should aim to merge it near the beginning of development on MaterialX v1.39.3, followed by the dependent PRs for the various MaterialX modules.

[StefanHabel]

OK cool, sounds good, thanks @jstone-lucasfilm!

I'll stand by to update the linked PRs when the time comes.