[dg] definitions_component

[schrockn]

Summary & Motivation

When using Pythonic Components, it's a fairly common pattern to want to create a basket of definitions in the component hierarchy.

For a lot of operations (e.g. attaching a resource) it feels heavy to create an entire component, and creating a Definitions leads to a potential import-time construction.

Here I propose a decorator, definitions_component, that enables the creation a Definitions in decorated function, but it creates a component loader on the users behalf.

@definitions_component
def my_definitions_component(context):
    @asset
    def an_asset() -> None: ...
    
    return Definitions([an_asset], resources={"a_resource", AResource())

The decorator implementation is quite minimal, and transforms the function into a component loader

def definitions_component(
    fn: DefinitionsLoadFn,
) -> Callable[[ComponentLoadContext], DecoratorDefinitionsComponent]:
    @component
    def load(context: ComponentLoadContext) -> DecoratorDefinitionsComponent:
        return DecoratorDefinitionsComponent(fn)
    return load

I think this will be a common pattern to make Pythonic components feel lighter.

How I Tested These Changes

BK

Changelog

NOCHANGELOG

[schrockn]

• [dg] definitions_component #28872 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[OwenKephart]

I think this will be a common pattern to make Pythonic components feel lighter.

Interested in following this thread. I think the question in the general case is "what is the function that the decorator is decorating doing?". Making sure that there's a consistent thread there feels important to me. e.g. right now @dbt_assets decorates the core execution function of the produced op, but a hypothetical @dbt_project decorator would likely be decorating something along the lines of a function to produce a DbtProject object, which would then get fed in to the DbtProjectComponent arguments.

If we expect people to do this sort of decorator stuff often, it feels like the decorator should be a property of the Component class itself to allow for subclassing.

But even then, it seems like this sort of pattern (if propagated to other component types) would really just introduce a third interface for constructing component types, which feels wrong. i.e. we'd have:

1. yaml schema
2. python class arguments
3. whatever we expect you to return from the decorated function that will eventually be used to construct (2)

Anyway, I'm totally ok with this pattern for a pure Definitions object because of how fundamental that feels (and the simplicity of the component). But if we want / expect this to propagate to other component types we might want to restructure this to be something like @DefinitionsComponent.decorator

[OwenKephart]

Talked through this, on board with the pattern but decided on the following logic loading a directory as a component (to allow for these decorators to be placed in files other than component.py)

Look at all files in the current directory (not including subdirs).

• if there is a component.yaml -> yaml component
• if there is a component.py file, search that for a component loader -> python component
• if there is a definitions.py file, search that for a Definitions loader -> definitions component

otherwise, load objects from all python files in that directory (again, not including subdirs)

• if there is a component loader in there -> python component
• if there is a Definitions object in there -> definitions component
• if multiple of either of the above, then we can error

otherwise:

• load each python file in that directory as a bare python defs component
• recurse into subdirs