You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using the allow_partials true option in an INCREMENTAL_BY_TIME_RANGE model, SQLMesh is refreshing the most recent completed interval along with the current incomplete (partial) interval.
I have not fully vetted if this happens generally, but I observed this happen with a scenario similar to this model:
The intent is to have the current month interval refresh daily.
As of writing this, 2/18/2025 using SQLMesh v0.156.0, here is the sequence of events I observed:
Initially load the model- For each month before 1/1/2025, SQLMesh runs a query using the first and last date of the month, as expected e.g.
1/1/2024 - 1/31/2024, ... , 11/1/2024 - 11/30/2024, 12/1/2024 - 12/31/2024.
During initial load still- For Jan and Feb 2025, the last complete interval and current interval, SQLMesh runs one query using 1/1/2025 - 2/xx/2025 (xx being dependent on what day in Feb initial load is done).
Running sqlmesh plan --restate-model -s 2025-01-01 -e 2025-02-xx results in same effect. One query runs to populate both intervals.
Running sqlmesh plan --restate-model -s 2025-01-01 -e 2025-01-31 results in correct effect of populating just the Jan 2025 interval.
Running sqlmesh run (the day after initial load) results in SQLMesh again running one query using 1/1/2025 - 2/xx+1/2025 to populate both intervals despite the fact that the Jan 2025 interval is already loaded
What I would expect to occur is that the initial load would populate Jan 2025 separately from Feb 2025 (the current partial interval) using 1/1/2025 - 1/31/2025 for Jan, and 2/1/2025 - 2/xx/2025 for Feb.
For subsequent sqlmesh run I would expect that only Feb 2025 would be refreshed using 2/1/2025 - 2/xx/2025.
Actually, on that last point, it would be preferable if the end date was always based on the boundaries of the intervals. E.g. in this case where interval_unit month, the current partial interval Feb 2025 would always use 2/1/2025 - 2/28/2025 each day it is refreshed.
The text was updated successfully, but these errors were encountered:
When using the
allow_partials true
option in an INCREMENTAL_BY_TIME_RANGE model, SQLMesh is refreshing the most recent completed interval along with the current incomplete (partial) interval.I have not fully vetted if this happens generally, but I observed this happen with a scenario similar to this model:
The intent is to have the current month interval refresh daily.
As of writing this, 2/18/2025 using SQLMesh v0.156.0, here is the sequence of events I observed:
1/1/2024 - 1/31/2024, ... , 11/1/2024 - 11/30/2024, 12/1/2024 - 12/31/2024.
sqlmesh plan --restate-model -s 2025-01-01 -e 2025-02-xx
results in same effect. One query runs to populate both intervals.sqlmesh plan --restate-model -s 2025-01-01 -e 2025-01-31
results in correct effect of populating just the Jan 2025 interval.sqlmesh run
(the day after initial load) results in SQLMesh again running one query using 1/1/2025 - 2/xx+1/2025 to populate both intervals despite the fact that the Jan 2025 interval is already loadedWhat I would expect to occur is that the initial load would populate Jan 2025 separately from Feb 2025 (the current partial interval) using 1/1/2025 - 1/31/2025 for Jan, and 2/1/2025 - 2/xx/2025 for Feb.
For subsequent
sqlmesh run
I would expect that only Feb 2025 would be refreshed using 2/1/2025 - 2/xx/2025.The text was updated successfully, but these errors were encountered: