Dynamically load additional commands and options from other modules on demand

[thmsklngr]

First Check

• I added a very descriptive title here.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the Typer documentation, with the integrated search.
• I already searched in Google "How to X in Typer" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to Typer but to Click.

Commit to Help

• I commit to help with one of those options 👆

Example Code

#main.py
import typer
import os
import importlib

app = typer.Typer()
state = {"verbose": False}


@app.command('list')
def _list():
    mods = []
    for file in os.listdir():
        if file.startswith('module') and file.endswith('.py'):
            mods.append(file.split('.')[0])
    print(f'Available modules: {mods}')


@app.command()
def run(module: str, *args, **kwargs):
    if os.path.exists(f'{module}.py'):
        mod = importlib.import_module(module)
        if hasattr(mod, 'app'):
            mod.app(*args, **kwargs)
    else:
        raise ValueError(f'{module} does not exist!')

if __name__ == "__main__":
    app()

# module1.py
import typer
from typing import Optional

app = typer.Typer()

@app.command()
def mod1_cmd1(arg1_1: str, arg1_2: Optional[str]):
    print(f"{arg1_1}")
    if arg1_2:
        print(f"{arg1_2}")

@app.command()
def mod1_cmd2(arg1_3: Optional[str], arg1_4: Optional[str]):
    print(f"{arg1_3}")
    if arg1_4:
        print(f"{arg1_4}")

Description

I once implemented a solution with Click, which loaded additional commands and their options from other files, i.e. modules. I cannot recall it right now, the code is somewhere on my worker laptop. But speaking generally I have the idea to have main commands in a main file and then run module specific commands.

The sample code may not work right now, but it should demonstrate maybe the main idea. Whenever I run python main.py run module1 I should be able to add additional options and arguments like mod1_cmd1. Additionally it should display its own help, like python main.py run module1 --help and there I should see mod1_cmd1 and mod1_cmd2.

Has anyone implemented a similar solution with Typer? As I noted above, I have a solution implemented with Click a while ago, so I thought it may be possible with Typer, too, since its core is Click as well.

Regards, Thomas

Operating System

Linux, macOS

Operating System Details

No response

Typer Version

0.7.0

Python Version

3.6.15

Additional Context

No response

[thmsklngr]

Here's an example on how I've done it with Click:

class PluginCommands(click.MultiCommand):
    """Dynamically read and execute subcommands from specified directory."""

    def list_commands(self, ctx):
        """Read subcommand groups from `plugins_dir`."""
        rv = []

        etl_dir = config.get_value('defaults:directories/etls')

        for name in Path(etl_dir).glob('*/*'):
            if name.is_file() and name.match('commands.py'):
                plugin = name.parent.parts[-1]
                rv.append(plugin)

        rv.sort()
        return rv

    def get_command(self, ctx, name):
        """Import and execute subcommand groups from `plugins_dir`."""
        try:
            mod = importlib.import_module(f"edis.etl.{name}.commands")
            # Since `debug` cannot be passed to this function,
            # only `print()` can be used. In case of troubleshooting
            # uncomment the next line to see if a module is
            # imported properly
            # print(mod)
        except ImportError as e:
            raise ImportError(e)
        else:
            if hasattr(mod, name):
                return getattr(mod, name)
            else:
                return None


@click.command(cls=PluginCommands)
@click.option(
...

This reads all files named commands.py and adds its commands, arguments and options. A command.py may look like this:

@click.group()
@click.pass_context
def accounting(ctx, sql_debug):
    ...

@accounting.command('ilv')
@click.pass_context
def ilv(ctx):
    """Handling of products."""
...

But I guess this can be achieved with "Nested Subcommands" in a quite similar way, but not as dynamic as this solution might be. In this case above I do not have to explicitly import any subcommand group, it's auto discovered.

Regards, Thomas

[Roy-Kid]

Yeah, I also need this. Do you have solution now?

[wolph]

It would be very useful if app.add_typer could lazy/dynamically load modules. Especially when you have larger projects with large dependencies (think of numpy/scipy/pandas) the import duration can quickly add up.

One possible option might be add support for a string argument to app.add_typer(...). Perhaps something like this:

import typer


app = typer.Typer()
app.add_typer('users.app')

As opposed to the current:

import typer

import users

app = typer.Typer()
app.add_typer(users.app)