Skip to content

Commit

Permalink
Remove redundant classes from the dist jar and unshimmed list (#11295)
Browse files Browse the repository at this point in the history
Fixes #11294

- Additionally removes classes from unshimmed-common-from-spark320.txt, 
which should have been done in #11288.
-  rapids4spark build properties can be removed from that list because there is a copy of it in sql-plugin-api jar
- Sort remaining entries
    
Signed-off-by: Gera Shegalov <gera@apache.org>
  • Loading branch information
gerashegalov authored Aug 5, 2024
1 parent 96d32f0 commit e7f218a
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 7 deletions.
2 changes: 1 addition & 1 deletion dist/scripts/binary-dedupe.sh
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ done < "$UNSHIMMED_LIST_TXT"
echo "$((++STEP))/ removing duplicates of unshimmed classes"

while read unshimmed_class; do
for pw in ./parallel-world/spark[34]* ; do
for pw in ./parallel-world/spark[34-]* ; do
unshimmed_path="$pw/$unshimmed_class"
[[ -f "$unshimmed_path" ]] && echo "$unshimmed_path" || true
done >> "$DELETE_DUPLICATES_TXT"
Expand Down
7 changes: 1 addition & 6 deletions dist/unshimmed-common-from-spark320.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,12 @@ META-INF/LICENSE
META-INF/NOTICE
com/nvidia/spark/GpuCachedBatchSerializer*
com/nvidia/spark/ParquetCachedBatchSerializer*
com/nvidia/spark/rapids/ColumnarRdd*
com/nvidia/spark/rapids/GpuColumnVectorUtils*
com/nvidia/spark/rapids/ExplainPlan.class
com/nvidia/spark/rapids/ExplainPlan$.class
com/nvidia/spark/rapids/ExplainPlanBase.class
com/nvidia/spark/rapids/PlanUtils*
com/nvidia/spark/rapids/SQLExecPlugin*
com/nvidia/spark/rapids/optimizer/SQLOptimizerPlugin*
com/nvidia/spark/rapids/ShimLoaderTemp*
com/nvidia/spark/rapids/SparkShims*
com/nvidia/spark/rapids/optimizer/SQLOptimizerPlugin*
org/apache/spark/sql/rapids/AdaptiveSparkPlanHelperShim*
org/apache/spark/sql/rapids/ExecutionPlanCaptureCallback*
rapids/*.py
rapids4spark-version-info.properties

0 comments on commit e7f218a

Please sign in to comment.