-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[native] fix(iceberg): Date partition value parse issue #24583
base: master
Are you sure you want to change the base?
[native] fix(iceberg): Date partition value parse issue #24583
Conversation
@aditi-pandit @majetideepak @Yuhta Can you please review this? |
890fc53
to
bc135f3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nmahadevuni for these changes. Have 2 questions for you.
@Test | ||
public void testDateQueries() | ||
{ | ||
assertQuery("SELECT * FROM ice_table_partitioned WHERE ds >= date ('1994-01-01')", "VALUES (1, date('2022-04-09')), (2, date('2022-03-18'))"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current test returns all rows from the underlying table without actually demonstrating the filtering expected. Add rows with values < date('1994-01-01) to be skippped by the filter.
@Test | ||
public void testDateQueries() | ||
{ | ||
assertQuery("SELECT * FROM ice_table_partitioned WHERE ds >= date ('1994-01-01')", "VALUES (1, date('2022-04-09')), (2, date('2022-03-18'))"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All the queries (both insertion and reading) use the date function in predicates. The date function actually transforms the string date value to integer days since epoch. The constant folding will evaluate this function before creating the partition filter. So I feel this issue happens irrespective of Iceberg.
Can you try queries without date function in the filter predicate ? What happens ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's how the date literals can be specified. I have removed the parenthesis to avoid the confusion. If we just specify the date string, then we get the varchar to date conversion error.
bc135f3
to
6d88a9e
Compare
Thanks @aditi-pandit , I have addressed the comments. |
Description
Fixes #24371
Motivation and Context
Filter on date column fails in Iceberg since the value returned by Iceberg API is in daysSinceEpoch but we assume its in ISO format "YYYY-MM-DD" and try to convert it to daysSinceEpoch. Velox side changes are merged in facebookincubator/velox#12126
Impact
None.
Test Plan
Added tests in TestPrestoNativeIcebergGeneralQueries.java
Release Notes