Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MultiPartitionsDefinition and BackfillPolicy.single_run() not giving correct partition set in backfill job #28861

Open
mtofano opened this issue Mar 28, 2025 · 0 comments
Labels
type: bug Something isn't working

Comments

@mtofano
Copy link

mtofano commented Mar 28, 2025

What's the issue?

Hi there,

I have the following partitions definition:

version_partitions_definition = StaticPartitionsDefinition(["v04"])
gsm_partitions_definition = MultiPartitionsDefinition(
    {"version": version_partitions_definition, "trade_date": weekday_partitions_definition}
)

I have an asset defined as follows:

@asset(
    partitions_def=gsm_partitions_definition,
    backfill_policy=BackfillPolicy.single_run(),
    group_name="security_info",
    kinds=["parquet"],
)
def security_info_versioned_staging(context: AssetExecutionContext) -> None:
    ...

My job is defined as follows:

security_info_versioned_job = define_asset_job(
    name="security_info_versioned_job",
    selection=[security_info_versioned_staging, security_info_versioned_staging_hive, security_info_versioned],
    hooks=build_last_job_result_hooks(
        dagster_instance="dagster_qe", code_location="dagster_pigs"
    ),
)

I manually launched a backfill on 61 partitions in total:

Image

When I print out the partition information available via the context I see this:

>>> print(context.partition_keys)
['2025-01-02|v04']
>>> print(context.partition_time_window)
TimeWindow(start=datetime.datetime(2025, 1, 2, 0, 0, tzinfo=datetime.timezone.utc), end=datetime.datetime(2025, 1, 3, 0, 0, tzinfo=datetime.timezone.utc))
>>> print(context.partition_key_range)
PartitionKeyRange(start='2025-01-02|v04', end='2025-01-02|v04')

I would expect this to give me the complete range of dates. I can see in the tags of the run that it is properly set there, but does not materialize into the context instance correctly:

Image

Is this a bug, or am I setting something in correctly?

Dagster version

dagster, version 1.9.8

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

@mtofano mtofano added the type: bug Something isn't working label Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant