[GLUTEN-6876] Support Spark-352 #7138
Conversation
|
Run Gluten Clickhouse CI |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
zhouyuan
force-pushed
the
wip_spark352
branch
from
e844f5d
to
2497695
Compare
|
Run Gluten Clickhouse CI |
| @@ -107,7 +107,7 @@ class ColumnarShuffleManager(conf: SparkConf) extends ShuffleManager with Loggin | |||
| metrics, | |||
| shuffleExecutorComponents) | |||
| case other: BaseShuffleHandle[K @unchecked, V @unchecked, _] => | |||
| new SortShuffleWriter(other, mapId, context, shuffleExecutorComponents) | |||
| new SortShuffleWriter(other, mapId, context, _, shuffleExecutorComponents) | |||
There was a problem hiding this comment.
Does this mean that gluten will not guarantee spark minor version compatibility?
There was a problem hiding this comment.
Hi @wForget,
Not guaranteed for all minor versions indeed. Actually it's the same for Spark 32 & 3.3 & 3.4. It's only tested with the latest minor release.
The issue is mostly due to the shim layer design, and the CI/CD resources are also not enough for so many small versions.
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI |
| p: SparkPlan, | ||
| reason: String, | ||
| fallbackNodeToReason: mutable.HashMap[String, String]): Unit = { | ||
| p.getTagValue(QueryPlan.OP_ID_TAG).foreach { |
There was a problem hiding this comment.
https://issues.apache.org/jira/browse/SPARK-48610
QueryPlan.OP_ID_TAG doesn't exist in spark 3.5.2+ / 4.0.0
There was a problem hiding this comment.
Thanks, there's also a minor change on sortshufflewriter, will move these changes to shim layer
|
Run Gluten Clickhouse CI |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
zhouyuan
force-pushed
the
wip_spark352
branch
from
2f439e6
to
7ce863d
Compare
|
Run Gluten Clickhouse CI |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI |
What changes were proposed in this pull request?
Support Spark 352
fixes #6876
How was this patch tested?
Pass GHA