-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge branch-24.10 into main [skip ci] #11578
Commits on Jul 24, 2024
-
Keep JNI and private dependency version as 24.08.0-SNAPSHOT until the branch-24.10 nightly CI is done. Track dependency by: #11240 Signed-off-by: Tim Liu <timl@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 1b064c6 - Browse repository at this point
Copy the full SHA 1b064c6View commit details
Commits on Jul 25, 2024
-
Merge pull request #11252 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for eadc6db - Browse repository at this point
Copy the full SHA eadc6dbView commit details -
Merge pull request #11253 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for bce9d51 - Browse repository at this point
Copy the full SHA bce9d51View commit details -
Merge pull request #11257 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 5d1b82b - Browse repository at this point
Copy the full SHA 5d1b82bView commit details
Commits on Jul 26, 2024
-
Merge pull request #11261 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 469bab4 - Browse repository at this point
Copy the full SHA 469bab4View commit details -
Merge pull request #11262 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for c5059a7 - Browse repository at this point
Copy the full SHA c5059a7View commit details
Commits on Jul 30, 2024
-
Merge pull request #11271 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 09b2dd5 - Browse repository at this point
Copy the full SHA 09b2dd5View commit details -
Merge pull request #11272 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 97f6d4d - Browse repository at this point
Copy the full SHA 97f6d4dView commit details -
Update the rapids JNI and private dependency version to 24.10.0-SNAPS…
Configuration menu - View commit details
-
Copy full SHA for 2e0e22b - Browse repository at this point
Copy the full SHA 2e0e22bView commit details -
Merge pull request #11274 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 99c2d12 - Browse repository at this point
Copy the full SHA 99c2d12View commit details -
Merge pull request #11275 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 079cf38 - Browse repository at this point
Copy the full SHA 079cf38View commit details -
Explicitly disable ANSI mode for ast_test.py [databricks] (#11258)
* disable ansi for ast_test * Signing off Signed-off-by: Raza Jafri <raza.jafri@gmail.com> --------- Signed-off-by: Raza Jafri <raza.jafri@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 6779154 - Browse repository at this point
Copy the full SHA 6779154View commit details
Commits on Jul 31, 2024
-
Merge pull request #11279 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for e53f234 - Browse repository at this point
Copy the full SHA e53f234View commit details -
Asynchronously copy table data to the host during shuffle (#11280)
* Copy table columns back to the host asynchronously Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Avoid synchronizing until after the device buffers have been freed * Use withResource --------- Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for dfcff71 - Browse repository at this point
Copy the full SHA dfcff71View commit details
Commits on Aug 2, 2024
-
remove spark31x tools supported files (#11285)
Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 6c93a1b - Browse repository at this point
Copy the full SHA 6c93a1bView commit details -
Move easy unshimmed classes to sql-plugin-api (#11288)
Contributes to #11208 Signed-off-by: Gera Shegalov <gera@apache.org>
Configuration menu - View commit details
-
Copy full SHA for cfac0b9 - Browse repository at this point
Copy the full SHA cfac0b9View commit details -
Use distinct count to estimate join magnification factor (#11284)
Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 2af9c0e - Browse repository at this point
Copy the full SHA 2af9c0eView commit details
Commits on Aug 5, 2024
-
Merge pull request #11298 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 96d32f0 - Browse repository at this point
Copy the full SHA 96d32f0View commit details -
Remove redundant classes from the dist jar and unshimmed list (#11295)
Fixes #11294 - Additionally removes classes from unshimmed-common-from-spark320.txt, which should have been done in #11288. - rapids4spark build properties can be removed from that list because there is a copy of it in sql-plugin-api jar - Sort remaining entries Signed-off-by: Gera Shegalov <gera@apache.org>
Configuration menu - View commit details
-
Copy full SHA for e7f218a - Browse repository at this point
Copy the full SHA e7f218aView commit details -
Use the new chunked API fro multi-get_json_object (#11289)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 93cdae1 - Browse repository at this point
Copy the full SHA 93cdae1View commit details
Commits on Aug 6, 2024
-
Enable get_json_object by default and remove legacy version (#11299)
* Enable get_json_object by default and remove legacy version Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Updated docs for 4.0.0 release --------- Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for a0cbd1d - Browse repository at this point
Copy the full SHA a0cbd1dView commit details
Commits on Aug 7, 2024
-
Skip deploying non-critical intermediate artifacts (#11301)
Signed-off-by: Yinqing Hao <haoyinqing@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 18980f7 - Browse repository at this point
Copy the full SHA 18980f7View commit details -
Fix display issue of lore.md (#11302)
* Fix display issue of lore.md Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com> * Add limitations * Fix comments --------- Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 8ef2e41 - Browse repository at this point
Copy the full SHA 8ef2e41View commit details
Commits on Aug 8, 2024
-
Add jihoonson as an authorized user for blossom-ci (#11312)
Signed-off-by: Jihoon Son <ghoonson@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for a54c9b3 - Browse repository at this point
Copy the full SHA a54c9b3View commit details
Commits on Aug 9, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8827bd8 - Browse repository at this point
Copy the full SHA 8827bd8View commit details -
Configuration menu - View commit details
-
Copy full SHA for a4bafa7 - Browse repository at this point
Copy the full SHA a4bafa7View commit details -
Merge pull request #11313 from NvTimLiu/fix-auto-merge-conflict-11310
Fix auto merge conflict 10845 11310 [skip ci]
Configuration menu - View commit details
-
Copy full SHA for de39a94 - Browse repository at this point
Copy the full SHA de39a94View commit details -
Safely close multiple resources in RapidsBufferCatalog (#11307)
* Safely close multiple resources in RapidsBufferCatalog Signed-off-by: Jihoon Son <ghoonson@gmail.com> * remove duplicate null filtering * add nullafying back --------- Signed-off-by: Jihoon Son <ghoonson@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 05152f7 - Browse repository at this point
Copy the full SHA 05152f7View commit details
Commits on Aug 10, 2024
-
Merge pull request #11315 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 1101427 - Browse repository at this point
Copy the full SHA 1101427View commit details
Commits on Aug 13, 2024
-
Update passing JSON tests after list support added in CUDF (#11319)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 36b9e2c - Browse repository at this point
Copy the full SHA 36b9e2cView commit details
Commits on Aug 14, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 48d8b3d - Browse repository at this point
Copy the full SHA 48d8b3dView commit details -
Merge pull request #11325 from NvTimLiu/fix-auto-merge-conflict-11317
Fix auto merge conflict 11317 [skip ci]
Configuration menu - View commit details
-
Copy full SHA for a85d101 - Browse repository at this point
Copy the full SHA a85d101View commit details
Commits on Aug 15, 2024
-
Append ustcfy to blossom-ci whitelist [skip ci] (#11324)
Signed-off-by: ustcfy <yafeng@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for a876df0 - Browse repository at this point
Copy the full SHA a876df0View commit details -
Make hive column matches not case-sensitive (#11327)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 25be396 - Browse repository at this point
Copy the full SHA 25be396View commit details -
Fix the mismatching default configs in integration tests (#11283)
* Add a new interface to retrieve all configs with their defaults; Add a new stage for integration test to populate default configs Signed-off-by: Jihoon Son <ghoonson@gmail.com> * address comments * missing version update in 2.13 pom * fix match arms * take the json file path as an input * add the new config file in the assembly * missing 2.13 change * use maven build directory var * revert unintended change * remove unnecessary clean * Add a new interface to retrieve all configs with their defaults; Add a new stage for integration test to populate default configs Signed-off-by: Jihoon Son <ghoonson@gmail.com> * address comments * missing version update in 2.13 pom * fix match arms * take the json file path as an input * add the new config file in the assembly * missing 2.13 change * use maven build directory var * revert unintended change * remove unnecessary clean * Add things in RapidsConf * missing change for 2.13 * fix directory path for scala 2.13 * exclude jackson from spark-hive * missing change for 2.13 * exclude old jackson stuff from iceberg * copyrights * antrun * fix config file path * move most dump changes to rapids conf - fork generation step with maven.compile.classpath - change to a phase before package Signed-off-by: Gera Shegalov <gera@apache.org> * clean up after merge * scala 2.13 * more strict arg check * unpack ambiguous string arguments * allow legacy negative scale for decimals for some tests * should fork for RapidsConf * remove System.exit() from RapidsConf.main() * missing change for scala 2.13 * Fix more tests to set configs * add back explicit configs --------- Signed-off-by: Jihoon Son <ghoonson@gmail.com> Signed-off-by: Gera Shegalov <gera@apache.org> Co-authored-by: Gera Shegalov <gera@apache.org>
Configuration menu - View commit details
-
Copy full SHA for bc8c577 - Browse repository at this point
Copy the full SHA bc8c577View commit details
Commits on Aug 16, 2024
-
Merge pull request #11336 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 6283029 - Browse repository at this point
Copy the full SHA 6283029View commit details -
Merge pull request #11338 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 2d90ec8 - Browse repository at this point
Copy the full SHA 2d90ec8View commit details -
Audit script - Check commits from sql-hive directory [skip ci] (#11340)
* Audit script - Check commits from sql-hive directory Signed-off-by: Niranjan Artal <nartal@nvidia.com> * update audit path Signed-off-by: Niranjan Artal <nartal@nvidia.com> --------- Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for c0c93b3 - Browse repository at this point
Copy the full SHA c0c93b3View commit details -
replace inputFiles with location.rootPaths.toString (#11323)
Signed-off-by: Zach Puller <zpuller@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 5302651 - Browse repository at this point
Copy the full SHA 5302651View commit details
Commits on Aug 17, 2024
-
Configuration menu - View commit details
-
Copy full SHA for a61b834 - Browse repository at this point
Copy the full SHA a61b834View commit details
Commits on Aug 18, 2024
-
Merge remote-tracking branch 'upstream/branch-24.08' into fix-auto-me…
…rge-conflict-11354
Configuration menu - View commit details
-
Copy full SHA for c412433 - Browse repository at this point
Copy the full SHA c412433View commit details -
Merge pull request #11357 from pxLi/fix-auto-merge-conflict-11354
Fix auto merge conflict 11354 [skip ci]
Configuration menu - View commit details
-
Copy full SHA for 2133b7d - Browse repository at this point
Copy the full SHA 2133b7dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6fb3205 - Browse repository at this point
Copy the full SHA 6fb3205View commit details
Commits on Aug 19, 2024
-
Swap build side for outer joins when natural build side is explosive …
…[databricks] (#11328) * Swap build side for outer joins when natural build side is explosive Signed-off-by: Jason Lowe <jlowe@nvidia.com> * scalastyle fix * Add clarification and sanity checking for sub join type * Use GpuExpression for bound join keys to improve type safety * test fix --------- Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 15aa45c - Browse repository at this point
Copy the full SHA 15aa45cView commit details -
Add string escaping JSON tests to the test_json_matrix (#10604)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 92d9bbb - Browse repository at this point
Copy the full SHA 92d9bbbView commit details -
conform dependency list in 341db to previous versions style [databric…
…ks] (#11321) * conform dependency list in 341db to previous versions style Signed-off-by: Zach Puller <zpuller@nvidia.com> * make shim deps artifact parent of all delta lake spark shim builds Signed-off-by: Zach Puller <zpuller@nvidia.com> * remove rapids-4-spark-db-bom explicit dependency from delta databricks shims Signed-off-by: Zach Puller <zpuller@nvidia.com> --------- Signed-off-by: Zach Puller <zpuller@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for b23e3fa - Browse repository at this point
Copy the full SHA b23e3faView commit details -
Add tests for repeated columns (#11362)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 2495b72 - Browse repository at this point
Copy the full SHA 2495b72View commit details
Commits on Aug 20, 2024
-
Fix failing test compile for Spark 4.0.0 (#11363)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 6924404 - Browse repository at this point
Copy the full SHA 6924404View commit details -
Set numRows for the ColumnBatch created in GpuBringBackToHost (#11365)
* Set numRows for the ColumnBatch created in GpuBringBackToHost Signed-off-by: Jihoon Son <ghoonson@gmail.com> * Directly get row count from the input ColumnarBatch * update copyright * fix copyright for the right file --------- Signed-off-by: Jihoon Son <ghoonson@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 9fcd8c7 - Browse repository at this point
Copy the full SHA 9fcd8c7View commit details
Commits on Aug 21, 2024
-
Change reference to
MapUtils
intoJSONUtils
(#11329)Signed-off-by: Nghia Truong <nghiat@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 3023625 - Browse repository at this point
Copy the full SHA 3023625View commit details
Commits on Aug 22, 2024
-
Move SparkRapidsBuildInfoEvent to its own file (#11368)
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for eb7b927 - Browse repository at this point
Copy the full SHA eb7b927View commit details -
Fix nightly snapshots being downloaded in premerge build (#11383)
Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 0f5254b - Browse repository at this point
Copy the full SHA 0f5254bView commit details -
Create a PrioritySemaphore to back the GpuSemaphore (#11376)
* priority semaphore implementation and tests Signed-off-by: Zach Puller <zpuller@nvidia.com> --------- Signed-off-by: Zach Puller <zpuller@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 35d2163 - Browse repository at this point
Copy the full SHA 35d2163View commit details -
Fix spark400 build in datagen and tests (#11375)
Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for d53de06 - Browse repository at this point
Copy the full SHA d53de06View commit details
Commits on Aug 23, 2024
-
JSON tests for corrected date, timestamp, and mixed types (#11388)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 6148257 - Browse repository at this point
Copy the full SHA 6148257View commit details -
Add support for Spark 3.5.2 [databricks] (#11334)
* Added 352 support * 352 building * Remove 352 from SNAPSHOT * Generated 2.13 pom * Removed 352 from snapshot in pom.xml * Signing off Signed-off-by: Raza Jafri <raza.jafri@gmail.com> * Added 352 to 213 snapshots and copyrights updated * Updated copyrights * Updated copyrights * Updated copyrights for SparkShimsSuite * addressed review comments * Fixed bad merge --------- Signed-off-by: Raza Jafri <raza.jafri@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for ac49393 - Browse repository at this point
Copy the full SHA ac49393View commit details
Commits on Aug 26, 2024
-
Revert work-around for empty split-string (#11393)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 35c593c - Browse repository at this point
Copy the full SHA 35c593cView commit details
Commits on Aug 27, 2024
-
Drop cudf-py python 3.9 support (#11396)
Change the default cudf-py version to python3.10, because rapidsai has dropped python3.9 support Signed-off-by: Tim Liu <timl@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for eb5735c - Browse repository at this point
Copy the full SHA eb5735cView commit details -
Add distinct join support for right outer joins (#11291)
* Add distinct join support for RightOuter joins Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Fix build --------- Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 3e04c1f - Browse repository at this point
Copy the full SHA 3e04c1fView commit details -
prevent duplicate queueing in the prio semaphore (#11389)
* prevent duplicate queueing in the prio semaphore Signed-off-by: Zach Puller <zpuller@nvidia.com> --------- Signed-off-by: Zach Puller <zpuller@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 549daf6 - Browse repository at this point
Copy the full SHA 549daf6View commit details -
stop using copyWithBooleanColumnAsValidity (#11399)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 46057a4 - Browse repository at this point
Copy the full SHA 46057a4View commit details
Commits on Aug 28, 2024
-
Merge pull request #11406 from NVIDIA/branch-24.08
[auto-merge] branch-24.08 to branch-24.10 [skip ci] [bot]
Configuration menu - View commit details
-
Copy full SHA for 61f0cca - Browse repository at this point
Copy the full SHA 61f0ccaView commit details
Commits on Aug 29, 2024
-
Support MinBy and MaxBy for non-float ordering (#11371)
* Support minBy on GPU Signed-off-by: Firestarman <firestarmanllc@gmail.com> * Support minBy on GPU Signed-off-by: Firestarman <firestarmanllc@gmail.com> * max_by wip Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * wip Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * test wip Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * wip test Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * reverse order and value Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * use min instead of min_by Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * verify again Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * regenerate shim docs Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * refine reduction and limit float type Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * 400 doc Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * update tests Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * verify and address comments Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * fix IT on spark320 and combine tests Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * remove 311 docs Signed-off-by: Haoyang Li <haoyangl@nvidia.com> * comment address Signed-off-by: Haoyang Li <haoyangl@nvidia.com> --------- Signed-off-by: Firestarman <firestarmanllc@gmail.com> Signed-off-by: Haoyang Li <haoyangl@nvidia.com> Co-authored-by: Firestarman <firestarmanllc@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for dbd92d2 - Browse repository at this point
Copy the full SHA dbd92d2View commit details
Commits on Aug 30, 2024
-
Fix a Pandas UDF slowness issue (#11395)
Close #10770 In CombiningIterator, the call to hasNext of pythonOutputIter may trigger a read without setting the target rows number, and the default rows number is Int.MaxValue, then the GpuArrowReader will try to read in a quite big batch when the partition data is big enough, leading to too much data copying by DirectByteBufferOutputStream at the writer side. Then slowness comes up. This PR changes the default read rows number to arrowMaxRecordsPerBatch to align with the Arrow batching behavior in Spark, and set the target read rows number in the hasNext function too. --------- Signed-off-by: Firestarman <firestarmanllc@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for db1d580 - Browse repository at this point
Copy the full SHA db1d580View commit details -
Fix asymmetric join crash when stream side is empty (#11411)
Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for ee2049a - Browse repository at this point
Copy the full SHA ee2049aView commit details
Commits on Sep 3, 2024
-
Stop using
copyWithBooleanColumnAsValidity
[databricks] (#11418)* stop using copyWithBooleanColumnAsValidity Signed-off-by: Chong Gao <res_life@163.com> * Refactor: rename variable --------- Signed-off-by: Chong Gao <res_life@163.com> Co-authored-by: Chong Gao <res_life@163.com>
Configuration menu - View commit details
-
Copy full SHA for 6d4c4ef - Browse repository at this point
Copy the full SHA 6d4c4efView commit details
Commits on Sep 4, 2024
-
Add in array_join support (#11420)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for d2abcd9 - Browse repository at this point
Copy the full SHA d2abcd9View commit details
Commits on Sep 5, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 53bf03a - Browse repository at this point
Copy the full SHA 53bf03aView commit details -
Move '. version-def.sh' ahead of 'cd scala2.13' in the spark-nightly-build.sh, to work with the "Dynamic Shim Detection" PR11308, Get the base shim version in deploy.sh using fuzzy matching instead of relying on version-def.sh Signed-off-by: timl <timl@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 5d7dc39 - Browse repository at this point
Copy the full SHA 5d7dc39View commit details
Commits on Sep 6, 2024
-
Dynamic Shim Detection for
build
Process [databricks] (#11308)* Removed irrelevant profiles for Scala 2.13 * Added an ant target to replace profiles * Modified buildall script to use the new way * Added function to consume the antrun plugin for version-def * Added all.buildvers * Added changes to version-def and shimplify * Clean up * Refactored python script to an external file * Changed the location of release.properties, added it to a property and added logging * Use exec tag to execute bash Signed-off-by: Raza Jafri <raza.jafri@gmail.com> * clean up * few more changes * Added a way to remove shims from releases * included_buildvers changes * undo version-def echos * removed unnecessary comment * removed method for mapping json * Added an xpath alternative to speed up the build * Added message to inform user about the missing package for speeding up the creation of the release.properties * Removed bash script and used jython to get releases * undo .gitignore change * removed the unused param overwrite_properties * cleanup * removed the call to create properties file * addressed review comments * removed unnecessary property * regenerated scala2.13 * Removed antrun execution from pom and refactored the python script so we can call it independently from CLI. * Addressed adding comments to Python script * added quotes to project so it's not mistaken for the word project but the variable project * moved the comment down a line * Call python script to get buildvers * Addressed review comments * Removed some of the references of snapshots and noSnapshots * addressed review comments and other minor changes * replaced expression with buildvers * Added dist profiles * Changed the phase so it runs after initialize * Added databricks profile * Regenerated 2.13 pom * Addressed review comments * missed comma in databricks --------- Signed-off-by: Raza Jafri <raza.jafri@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 4c2203e - Browse repository at this point
Copy the full SHA 4c2203eView commit details -
remove the redundant archive link (#11421)
Signed-off-by: liyuan <yuali@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 77615e6 - Browse repository at this point
Copy the full SHA 77615e6View commit details -
Add companion metrics for all nsTiming metrics without semaphore (#11331
Configuration menu - View commit details
-
Copy full SHA for 7de3fc4 - Browse repository at this point
Copy the full SHA 7de3fc4View commit details
Commits on Sep 9, 2024
-
xfail array and map cast to string tests (#11451)
* xfail array and map cast to string tests Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Ignore extract mortgage data test * copyrights --------- Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 9fc0667 - Browse repository at this point
Copy the full SHA 9fc0667View commit details
Commits on Sep 10, 2024
-
Log DBR BuildInfo [databricks] (#11455)
Fixes #8587 - Match the dbrVersion from the binaries - Log the build info exposed in current_version - Fix the WITH_BLOOP build ``` 2024-09-10 16:21:42,019 [Thread-6] WARN com.nvidia.spark.rapids.DatabricksShimServiceProvider - Databricks Runtime Build Info match: SUCCESS DBR_VERSION: 11.3.x-snapshot-gpu-ml-scala2.12 spark.BuildInfo.gitHash: d65fb2374451fd10bf416297dc22549bcbaf2702 databricks.BuildInfo.gitHash: b7fd9d058866e0f98f78304bf90c690198e2b208 databricks.BuildInfo.gitTimestamp: 20240820204043 ``` Signed-off-by: Gera Shegalov <gera@apache.org>
Configuration menu - View commit details
-
Copy full SHA for a92bfbf - Browse repository at this point
Copy the full SHA a92bfbfView commit details -
Fixed some of the failing parquet_tests [databricks] (#11429)
* Fixed some of the failing parquet_tests * Signing off Signed-off-by: Raza Jafri <raza.jafri@gmail.com> * addressed review comments * removed unused import --------- Signed-off-by: Raza Jafri <raza.jafri@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 502f5a3 - Browse repository at this point
Copy the full SHA 502f5a3View commit details
Commits on Sep 12, 2024
-
Skip test_hash_groupby_approx_percentile byte and double tests tempor…
…arily (#11469) Signed-off-by: Alessandro Bellina <abellina@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 5beeba8 - Browse repository at this point
Copy the full SHA 5beeba8View commit details
Commits on Sep 14, 2024
-
Replace scala.util.Try with a try statement in the DBR buildinfo [dat…
…abricks] (#11466) * Switch to a regular try Signed-off-by: Gera Shegalov <gera@apache.org> * drop Maven tarball Signed-off-by: Gera Shegalov <gera@apache.org> * unused import Signed-off-by: Gera Shegalov <gera@apache.org> * repro Signed-off-by: Gera Shegalov <gera@apache.org> --------- Signed-off-by: Gera Shegalov <gera@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 938d4df - Browse repository at this point
Copy the full SHA 938d4dfView commit details
Commits on Sep 16, 2024
-
Revert "Skip test_hash_groupby_approx_percentile byte and double test…
Configuration menu - View commit details
-
Copy full SHA for 65f0095 - Browse repository at this point
Copy the full SHA 65f0095View commit details -
Enable tests after string_split was fixed (#11474)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 00cd422 - Browse repository at this point
Copy the full SHA 00cd422View commit details
Commits on Sep 17, 2024
-
Use improved CUDF JSON validation (#11464)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 2589976 - Browse repository at this point
Copy the full SHA 2589976View commit details
Commits on Sep 18, 2024
-
Fix a json test for non utc time zone (#11482)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for f4119c1 - Browse repository at this point
Copy the full SHA f4119c1View commit details
Commits on Sep 19, 2024
-
Use reusable auto-merge workflow (#11483)
Signed-off-by: Peixin Li <pxLi@nyu.edu>
Configuration menu - View commit details
-
Copy full SHA for 7c13383 - Browse repository at this point
Copy the full SHA 7c13383View commit details
Commits on Sep 20, 2024
-
Enable tests for all JSON white space normalization (#11456)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 4b26190 - Browse repository at this point
Copy the full SHA 4b26190View commit details
Commits on Sep 23, 2024
-
Support yyyyMMdd in GetTimestamp operator for LEGACY mode [databricks] (
#11449) * Support yyyyMMdd in GetTimestamp operator for LEGACY mode Signed-off-by: Chong Gao <res_life@163.com> Co-authored-by: Chong Gao <res_life@163.com>
Configuration menu - View commit details
-
Copy full SHA for 0f5d510 - Browse repository at this point
Copy the full SHA 0f5d510View commit details -
Support non-UTC timezone for casting from date type to timestamp type…
… [databricks] (#11462) * Support non-UTC timezone for casting from date type to timestamp type Signed-off-by: Chong Gao <res_life@163.com> Co-authored-by: Chong Gao <res_life@163.com>
Configuration menu - View commit details
-
Copy full SHA for ebcc146 - Browse repository at this point
Copy the full SHA ebcc146View commit details
Commits on Sep 24, 2024
-
Install cuDF-py against python 3.10 on Databricks (#11477)
* Install cuDF-py against python 3.10 on Databricks Fix on Databricks runtime for : #11394 Enable the udf_cudf_test test case for Databricks-13.3 Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/ Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests, because on DB-13.3 Conda is not installed by default. Signed-off-by: timl <timl@nvidia.com> * Check if 'conda' exists to make the if/else expression more readable Signed-off-by: timl <timl@nvidia.com> --------- Signed-off-by: timl <timl@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 510ee83 - Browse repository at this point
Copy the full SHA 510ee83View commit details -
Enable parquet suites from Spark UT (#11366)
* add parquet column index ut test Signed-off-by: fejiang <fejiang@nvidia.comm> * change Signed-off-by: fejiang <fejiang@nvidia.comm> * added parquet suite Signed-off-by: fejiang <fejiang@nvidia.com> * pom changed Signed-off-by: fejiang <fejiang@nvidia.com> * DeltaEncoding Suite Signed-off-by: fejiang <fejiang@nvidia.com> * enable more suites Signed-off-by: fejiang <fejiang@nvidia.com> * remove ignored case Signed-off-by: fejiang <fejiang@nvidia.com> * format Signed-off-by: fejiang <fejiang@nvidia.com> * added ignored cases Signed-off-by: fejiang <fejiang@nvidia.com> * change to parquet hadoop version Signed-off-by: fejiang <fejiang@nvidia.comm> * remove parquet.version Signed-off-by: fejiang <fejiang@nvidia.comm> * adding scope and classifier Signed-off-by: fejiang <fejiang@nvidia.comm> * pom remove unused Signed-off-by: fejiang <fejiang@nvidia.com> * pom chang3 2.13 Signed-off-by: fejiang <fejiang@nvidia.com> * add schema suite Signed-off-by: fejiang <fejiang@nvidia.comm> * remove dataframe Signed-off-by: fejiang <fejiang@nvidia.comm> * RapidsParquetThriftCompatibilitySuite Signed-off-by: fejiang <fejiang@nvidia.com> * ThriftCompaSuite added Signed-off-by: fejiang <fejiang@nvidia.com> * more suites but the RowIndexSuite one Signed-off-by: fejiang <fejiang@nvidia.com> * formatting issues Signed-off-by: fejiang <fejiang@nvidia.com> * exlude SPARK-36803: Signed-off-by: fejiang <fejiang@nvidia.comm> * setting change Signed-off-by: fejiang <fejiang@nvidia.comm> * setting change Signed-off-by: fejiang <fejiang@nvidia.comm> * adjust order Signed-off-by: fejiang <fejiang@nvidia.comm> * adjust settings Signed-off-by: fejiang <fejiang@nvidia.comm> * adjust settings Signed-off-by: fejiang <fejiang@nvidia.comm> * RapidsParquetThriftCompatibilitySuite settings * known issue added Signed-off-by: fejiang <fejiang@nvidia.com> * format new line Signed-off-by: fejiang <fejiang@nvidia.com> * known issue added Signed-off-by: fejiang <fejiang@nvidia.com> * RapidsParquetDeltaByteArrayEncodingSuite Signed-off-by: fejiang <fejiang@nvidia.comm> * RapidsParquetAvroCompatibilitySuite Signed-off-by: fejiang <fejiang@nvidia.comm> * ParquetFiledIdSchemaSuite and Avro suite added * pom Avro suite modified * ParquetFileFormatSuite added * RapidsParquetRebaseDatetimeSuite and QuerySuite added * RapidsParquetSchemaPruningSuite added * setting adjust Signed-off-by: fejiang <fejiang@nvidia.com> * setting adjust Signed-off-by: fejiang <fejiang@nvidia.com> * UT adjuct exclude added Signed-off-by: fejiang <fejiang@nvidia.com> * RapidsParquetThriftCompatibilitySuite adjust setting Signed-off-by: fejiang <fejiang@nvidia.com> * comment Create parquet table with compression Signed-off-by: fejiang <fejiang@nvidia.com> * SPARK_HOME NOT FOUND issue solved. Signed-off-by: fejiang <fejiang@nvidia.com> * enabling more suite Signed-off-by: fejiang <fejiang@nvidia.com> * remove exclude from RapidsParquetFieldIdIOSuite Signed-off-by: fejiang <fejiang@nvidia.com> * formate and remove parquet files Signed-off-by: fejiang <fejiang@nvidia.com> * comment setting Signed-off-by: fejiang <fejiang@nvidia.com> * pom modified and remove unnecess case Signed-off-by: fejiang <fejiang@nvidia.com> --------- Signed-off-by: fejiang <fejiang@nvidia.comm> Signed-off-by: fejiang <fejiang@nvidia.com> Co-authored-by: fejiang <fejiang@nvidia.comm>
Configuration menu - View commit details
-
Copy full SHA for a34f33e - Browse repository at this point
Copy the full SHA a34f33eView commit details -
Optimzing Expand+Aggregate in sqls with many count distinct (#10798)
* optimzing Expand+Aggregate in sqlw with many count distinct Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> * simplify Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> * add comment Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> * address comments Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> --------- Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 5725e2e - Browse repository at this point
Copy the full SHA 5725e2eView commit details -
Use UnaryLike instead of UnaryExpression (#11490)
Signed-off-by: Alessandro Bellina <abellina@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 6a9731f - Browse repository at this point
Copy the full SHA 6a9731fView commit details
Commits on Sep 26, 2024
-
Download artifacts via wget (#11503)
To fix: #11502 Download jars using wget instead of 'mvn dependency:get' to fix 'missing intermediate jars' failures, as we stopped deploying these intermediate jars since version 24.10 Signed-off-by: timl <timl@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 10eaa22 - Browse repository at this point
Copy the full SHA 10eaa22View commit details -
Replace libmamba-solver with mamba command [skip ci] (#11507)
* quick workaround to make image build work Signed-off-by: Peixin Li <pxLi@nyu.edu> * use mamba directly --------- Signed-off-by: Peixin Li <pxLi@nyu.edu>
Configuration menu - View commit details
-
Copy full SHA for b113c46 - Browse repository at this point
Copy the full SHA b113c46View commit details -
GPU device watermark metrics (#11457)
* add max memory watermark metric Signed-off-by: Zach Puller <zpuller@nvidia.com> --------- Signed-off-by: Zach Puller <zpuller@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for c047707 - Browse repository at this point
Copy the full SHA c047707View commit details
Commits on Sep 27, 2024
-
Fix FileAlreadyExistsException in LORE dump process (#11484)
* Updated parameters to enable file overwriting when dumping. Signed-off-by: ustcfy <yafeng@nvidia.com> * Validate LORE dump root path before execution Signed-off-by: ustcfy <yafeng@nvidia.com> * Add loreOutputRootPathChecked map for tracking lore output root path checks. Signed-off-by: ustcfy <yafeng@nvidia.com> * Delay path and filesystem initialization until actually needed. Signed-off-by: ustcfy <yafeng@nvidia.com> * Add test and update dev/lore.md doc. Signed-off-by: ustcfy <yafeng@nvidia.com> * Format code to ensure line length does not exceed 100 characters Signed-off-by: ustcfy <fengyan_@mail.ustc.edu.cn> * Format code to ensure line length does not exceed 100 characters Signed-off-by: ustcfy <fengyan_@mail.ustc.edu.cn> * Improved resource management by using withResource. Signed-off-by: ustcfy <fengyan_@mail.ustc.edu.cn> * Update docs/dev/lore.md Co-authored-by: Renjie Liu <liurenjie2008@gmail.com> * Improved resource management by using withResource. Signed-off-by: ustcfy <fengyan_@mail.ustc.edu.cn> * Removed for FileSystem instance. Signed-off-by: ustcfy <fengyan_@mail.ustc.edu.cn> * Update docs/dev/lore.md Co-authored-by: Gera Shegalov <gshegalov@nvidia.com> --------- Signed-off-by: ustcfy <yafeng@nvidia.com> Signed-off-by: ustcfy <fengyan_@mail.ustc.edu.cn> Co-authored-by: Renjie Liu <liurenjie2008@gmail.com> Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 41351c0 - Browse repository at this point
Copy the full SHA 41351c0View commit details -
Deploy all submodules for default sparkver (#11516)
Signed-off-by: Peixin Li <pxLi@nyu.edu>
Configuration menu - View commit details
-
Copy full SHA for 9625e58 - Browse repository at this point
Copy the full SHA 9625e58View commit details -
Update from_json to use new cudf features (#11497)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for fbd4db9 - Browse repository at this point
Copy the full SHA fbd4db9View commit details -
Fixed buildall script (#11515)
Signed-off-by: Raza Jafri <raza.jafri@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for f87ccb9 - Browse repository at this point
Copy the full SHA f87ccb9View commit details
Commits on Sep 30, 2024
-
Only test years after 1900 for LEGACY mode (#11545)
Signed-off-by: Chong Gao <res_life@163.com> Co-authored-by: Chong Gao <res_life@163.com>
Configuration menu - View commit details
-
Copy full SHA for c74e2dd - Browse repository at this point
Copy the full SHA c74e2ddView commit details
Commits on Oct 1, 2024
-
Fix negative rs. shuffle write time (#11548)
* Fix negative rs. shuffle write time Signed-off-by: Alessandro Bellina <abellina@nvidia.com> * Stop double counting openTimeNs in shuffleWriteTimeMetric --------- Signed-off-by: Alessandro Bellina <abellina@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 5fae883 - Browse repository at this point
Copy the full SHA 5fae883View commit details -
Update test now that code is fixed (#11496)
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 8207f7b - Browse repository at this point
Copy the full SHA 8207f7bView commit details
Commits on Oct 8, 2024
-
Fix test case unix_timestamp(col, 'yyyyMMdd') failed for Africa/Casab…
…lanca timezone and LEGACY mode (#11567) Signed-off-by: Chong Gao <res_life@163.com>
Configuration menu - View commit details
-
Copy full SHA for b9b7629 - Browse repository at this point
Copy the full SHA b9b7629View commit details
Commits on Oct 9, 2024
-
Configuration menu - View commit details
-
Copy full SHA for b715ef2 - Browse repository at this point
Copy the full SHA b715ef2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 670d8ec - Browse repository at this point
Copy the full SHA 670d8ecView commit details -
Signed-off-by: nvauto <70000568+nvauto@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 85db666 - Browse repository at this point
Copy the full SHA 85db666View commit details
Commits on Oct 11, 2024
-
backport fixes of #11573 to branch 24.10 (#11588)
* avoid long tail tasks due to PrioritySemaphore (#11574) * use task id as tie breaker Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> * save threadlocal lookup Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> --------- Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> * addressing jason's comment Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> --------- Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
Configuration menu - View commit details
-
Copy full SHA for ec9d008 - Browse repository at this point
Copy the full SHA ec9d008View commit details
Commits on Oct 14, 2024
-
update doc for 2410 release (#11582)
Signed-off-by: liyuan <yuali@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for 8254f63 - Browse repository at this point
Copy the full SHA 8254f63View commit details -
Update rapids JNI and private dependency to 24.10.0 (#11576)
\nWait for the pre-merge CI job to SUCCEED Signed-off-by: nvauto <70000568+nvauto@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 3c9bb8b - Browse repository at this point
Copy the full SHA 3c9bb8bView commit details -
Update latest changelog [skip ci] (#11577)
* Update latest changelog [skip ci] Update change log with CLI: \n\n scripts/generate-changelog --token=<GIT_TOKEN> --releases=24.08,24.10 Signed-off-by: nvauto <70000568+nvauto@users.noreply.github.com> * Update changelog Signed-off-by: timl <timl@nvidia.com> * Update changelog Signed-off-by: timl <timl@nvidia.com> --------- Signed-off-by: nvauto <70000568+nvauto@users.noreply.github.com> Signed-off-by: timl <timl@nvidia.com> Co-authored-by: timl <timl@nvidia.com>
Configuration menu - View commit details
-
Copy full SHA for b535f2d - Browse repository at this point
Copy the full SHA b535f2dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4007f2c - Browse repository at this point
Copy the full SHA 4007f2cView commit details