【VL】How to resolve problem：HdfsReadFile.h Line: 53 Read failure in HDFSReadFile::preadInternal.

[xunxunmimi5577]

各位好，我在使用velox作为后端时，遇到一个问题，请求各位的帮助，

Gluten 版本
gluten-velox-bundle-spark3.3_2.12-1.1.1.jar

Spark 版本
spark-3.3.2-hadoop-3.3

报错如下：

Reason: Read failure in HDFSReadFile::preadInternal.
Retriable: False
Expression: bytesRead >= 0
Context: Split [Hive: hdfs://hdfsHACluster/user/hive/warehouse/tpcds_bin_partitioned_parquet_2000.db/date_dim/000000_0 0 - 4194304] Task Gluten_Stage_1_TID_2149
Top-Level Context: Same as context.
Function: read
File: /root/src/oap-project/gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/hdfs/HdfsReadFile.h
Line: 53
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorEPKcEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZNK8facebook5velox12HdfsReadFile13preadInternalEmmPc
# 4  _ZNK8facebook5velox12HdfsReadFile5preadEmmPv
# 5  _ZN8facebook5velox4dwio6common19ReadFileInputStream4readEPvmmNS2_10MetricsLog11MetricsTypeE
# 6  _ZN8facebook5velox4dwio6common23SeekableFileInputStream4NextEPPKvPi
# 7  _ZN8facebook5velox7parquet10ReaderBase16loadFileMetaDataEv
# 8  _ZN8facebook5velox7parquet10ReaderBaseC1ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 9  _ZN8facebook5velox7parquet13ParquetReaderC2ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 10 _ZN8facebook5velox7parquet20ParquetReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 11 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 12 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 13 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 14 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 15 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 16 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 17 _ZN6gluten24WholeStageResultIterator4nextEv
# 18 Java_io_glutenproject_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 19 0x00007f9ca1018427

        at io.glutenproject.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
        at io.glutenproject.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:65)
        at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
        ... 42 more

datanode有如下日志，看起来是相关的，

2024-06-28 14:50:20,390 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Failed to read expected SASL data transfer protection handshake from client at /xxx.xx.xx.xxx:xxxxx. Perhaps the client is running an older version of Hadoop which does not support SASL data transfer protection
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.InvalidMagicNumberException: Received 1c519d instead of deadbeef from client.
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:375)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:309)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:136)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
        at java.lang.Thread.run(Thread.java:750)

是我的kerberos配置不正确吗？

[xunxunmimi5577]

@zhouyuan 劳烦您帮忙看下

[zhouyuan]

Hi @xunxunmimi5577

目前在gluten + velox 这里是支持基础的kerberos + HA
https://github.com/apache/incubator-gluten/blob/main/docs/get-started/Velox.md#configuration-about-hdfs-support
确认一下是否已经把相关配置信息写入hdfs-client.xml

如果kerberos这开启了高级的安全配置，比如 hadoop.rpc.protection=privacy 目前native libhdfs3还不支持。我们在推动使用JVM 版的libhdfs 这样可以之前vanilla spark这里所有hadoop配置，见下面这个patch:
#6172

thanks, -yuan

[xunxunmimi5577]

@zhouyuan 你好，想请教下，看datanode的日志似乎跟hdfs客户端的版本不匹配有关？目前我们用的是hadoop 3.2.4， 不知道这个 libhdfs3 用的是什么版本？