This folder contains the code for the Dagster documentation site at https://docs.dagster.io.
To install the dependencies required to run the docs, run the following in the /docs
directory:
make docs_dev_install
Then, run the developer server in the same /docs
directory:
make dev
Open http://localhost:3001 with your browser to see the result.
All content is in the /content
folder.
Try it out
Open http://localhost:3001/concepts/solids-pipelines/solids in your browser, and the file /content/concepts/solid-pipelines/solids.mdx
in your editor. Try editing the file and see your changes be reflected live in the browser.
- Editing API Docs: we use Sphinx to generate API Docs.
- Writing MDX: we use MDX to write our main content. This section will explain how to include code snippets and use various custom components.
- Navigation: this explains how to update the sidebar.
The API documentation is authored separately in the dagster/docs/sphinx
folder using Sphinx, a Python document generator. We use Sphinx because it has built-in functionality to document Python methods and classes by parsing the docstrings.
If you update the API documentation in the dagster/docs/sphinx
folder, you need to rebuild the output from Sphinx in order to see your changes reflected in the documentation site.
To rebuild the docs site, run in the docs dir:
tox -vv -e py38-sphinx
Sphinx requires all the modules for which it is building API docs to be installed, but doesn't require their dependencies to be installed, as long as the Dagster modules can be imported. We create the docs environment for Sphinx by first installing from docs-dagster-requirements.txt
with no dependencies, then installing any required dependencies from docs-build-requirements.txt
. Most dependencies imported by dagster can go in the list of mocked imports in autodoc_mock_imports
in docs/sphinx/conf.py
, since they're not actually needed to build the docs, but the relatively small number of dependencies that are actually needed to import the dagster packages and build the docs can be found in docs-build-requirements.txt
.
If you don't build the API documentation and include the changes in your diff, you will see a build error reminding you to do so.
If the test that checks whether the sphinx build passes is failing due to an import-related error, it likely indicates that a dependency needs to be added to autodoc_mock_imports
in conf.py
if it isn't actually needed to import the Dagster library (most common), or added to docs-build-requirements.txt
if it actually needs to be included just for the library to be able to be imported (less common).
We use MDX, which is a content format that lets us use JSX in our markdown documents. This lets us import components and embed them in our documentation.
We want the code snippets in our documentation to be high quality. We maintain quality by authoring snippets in the /examples/docs_snippets
folder and testing them to make sure they work and we don't break them in the future.
Example:
To include snippets from the code base, you can provide additional properties to the markdown code fence block:
Using markers:
```python file=/concepts/solids_pipelines/solid_definition.py startafter=start_solid_definition_marker_0 endbefore=end_solid_definition_marker_0
```
Render:
Run make snapshot
to actually get the snippets to render. This will replace the body of the code block with the code you referenced.
Important: to change the value of a literal include, you must change the referenced code, not the code inside the code block. Run make snapshot
once you update the underlying code to see the changes in the doc site. This behavior is different from previous versions of the site.
Properties:
-
file
: The path to file relative to the/examples/docs_snippets/docs_snippets
folder. You can use a relative path from this folder to access snippets from other parts of the codebase, but this is highly discouraged. Instead, you should copy the bit of code you need intodoc_snippets
and test it appropriately. -
startafter
andendbefore
: Use this property to specify a code snippet in between two markers. You will need to include the markers in the source file as comments. -
lines
: (This is highly discouraged) Use this property to specify a range of lines to include in the code snippet. You can also pass multiple ranges separated by commas.For example:
```python file=/concepts/solids_pipelines/solid_definition.py lines=1-10,12 ```
-
trim=true
: Sometimes, our Python formatterblack
gets in the way and adds some spacing before the end of your snippet and theendbefore
marker comment. Use trim to get rid of any extra newlines.
Here are the available components we built to use in MDX:
- PyObject
- Link
- Check
- Cross
- LinkGrid
- LinkGridItem
- Warning
- Experimental
- CodeReferenceLink
See more details in the /components/MDXComponents
file.
PyObject Component
The most important component is the <PyObject />
component. It is used to link to references in the API documentation from MDX files.
Example usage:
Here is an example of using PyObject: <PyObject module="dagster" object="SolidDefinition" />
By default, we just display the object name. To override, use the `displayText` prop: <PyObject module="dagster" object="solid" displayText="@solid"/ >
If you are adding a new page or want to update the navigation in the sidebar, update the docs/content/_navigation.json
file.
Docs screenshots are generated manually. Previously, this meant that you would run Dagit and hit Shift-Command-4. Now, we are moving towards using the docs/screenshot_capture/capture-screenshot.py
script, which adds some automation. To use it, you add a "screenshot spec" to docs/screenshot_capture/screenshots.yaml
and then run the script to generate an image from that spec.
A screenshot spec includes:
- A
path
to an image file that the screenshot should be stored in. - A
defs_file
that the script will rundagit -f
ordagit -w
on to load a set of Dagster definitions. - A
url
for the page to take the screenshot from. - An optional set of manual
steps
, which the person generating the screenshot is expected to carry out before the screenshot is taken. - A
vetted
boolean, which indicates whether the current screenshot in the repo was generated using the spec.
pip install selenium
Install Selenium's chrome driver:
brew install chromedriver
OR download chromedriver from here) and manually add to /usr/local/bin- You may need to run
xattr -d com.apple.quarantine /usr/local/bin/chromedriver
(gross)
Having written a screenshot spec, you can (re-)generate the screenshot by running the capture-screenshot script and providing the path
value that you want to regenerate the screenshot for. E.g.
python docs/screenshot_capture/capture-screenshot.py concepts/dagit/runs-tab.png
It goes without saying that opportunities to automate this further abound.