Question

我收到以下错误：

ERROR hive.log - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe (Write failed) while trying to connect Drill to Hive.

对于“Hive”Microsoft Azure HDInsight（Ranger Enabled）（远程Metastore（MS SQL Server））正在使用，而对于Drill，我正在使用与群集在同一VNet下的其他VM。我能够使用以下配置制作Drill Storage插件

{
"type": "hive",
"enabled": true,
"configProps": {
"hive.metastore.uris": "thrift://hn0-xyz.cloudapp.net:9083,thrift://hn1-    xyz.cloudapp.net:9083",
"hive.metastore.warehouse.dir": "/hive/warehouse",
"fs.default.name": "wasb://qwerty@demo.blob.core.windows.net",
"hive.metastore.sasl.enabled": "false"
}
}

堆栈错误跟踪：

17:57:19.515 [2779bbff-d7a9-058c-d133-b41795a0ee58:foreman] ERROR hive.log - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe (Write failed)
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed)
        at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) ~[drill-hive-exec-shaded-1.9.0.jar:1.9.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_all_databases(ThriftHiveMetastore.java:733) ~[hive-metastore-1.2.1.jar:1.2.1]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_databases(ThriftHiveMetastore.java:726) ~[hive-metastore-1.2.1.jar:1.2.1]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1031) ~[hive-metastore-1.2.1.jar:1.2.1]
        at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient.getDatabasesHelper(DrillHiveMetaStoreClient.java:205) [drill-storage-hive-core-1.9.0.jar:1.9.0]

核心-site.xml中：

<configuration>
<property>
        <name>fs.azure.account.keyprovider.kkhdistore.blob.core.windows.net</name>
      <value>org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider</value>
</property>
<property>
    <name>fs.azure.shellkeyprovider.script</name>
    <value>/usr/lib/python2.7/dist-   packages/hdinsight_common/decrypt.sh</value>
  </property>
  <property>
    <name>fs.azure.account.key.kkhdistore.blob.core.windows.net</name>
    <value>{COPY FROM CLUSTER core-site.xml}</value>
  </property>
  <property>
    <name>fs.AbstractFileSystem.wasb.impl</name>
    <value>org.apache.hadoop.fs.azure.Wasb</value>
  </property>
 </configuration>

Answer 1

根据官方文档Ports and URIs used by HDInsight的{{3}}部分，如下面的说明所示，我怀疑您使用的Hive是在Azure HDInsight群集上手动安装的，而不是Hive群集类型。

某些服务仅适用于特定群集类型。例如，HBase仅适用于HBase集群类型。

因此，对于像Drill这样的其他人来说，节点端口9083是非公共端口，即使它们位于同一个VNet下也是如此。该问题的解决方案是在文档Non-public ports之后创建规则以允许集群的NSG处的端口入站。希望它有所帮助。

钻到Hive连接错误（org.apache.thrift.transport.TTransportException java.net.SocketException：Broken pipe（写入失败））

1 个答案: