Question

尝试在single node上运行apache钻探，然后执行article从嵌入式钻探访问HDFS，但出现错误

➜  Apps /home/hph_etl/Apps/apache-drill-1.16.0/bin/sqlline -u "jdbc:drill:zk=local;schema=dfs"

...

apache drill (dfs)> select * from dfs.`tmp/`;
Error: RESOURCE ERROR: Failed to load schema for "dfs"!

java.net.ConnectException: Call From HW04.ucera.local/172.18.4.49 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused


[Error Id: 2fd541ee-2290-4cf8-979b-aca3c77859e2 ] (state=,code=0)
apache drill (dfs)> !q
Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl

dfs存储插件文件看起来像...

{
  "type": "file",
  "connection": "hdfs://localhost:8020/",
  "config": null,
  "workspaces": {
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
   ....
}

（请注意，我真的不知道如何确定hdfs连接应该在哪个端口上），错误消息的链接（http://wiki.apache.org/hadoop/ConnectionRefused）无处可寻。尝试使用另一个SO post的替代解决方案会引发错误：

➜  Apps /home/hph_etl/Apps/apache-drill-1.16.0/bin/sqlline -u "jdbc:drill:drillbit=localhost:31010;schema=dfs"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: CONNECTION : io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:31010 (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: CONNECTION : io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:31010
    at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:178)
    at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)

目前不确定要检查什么；有任何调试建议或修复方法吗？

Answer 1

最终，有效的方法是将连接hdfs IP设置为hadoop群集的名称节点的IP （通常是从另一个连接到HDFS的SO post获得），因此钻dfs存储插件配置如下：

{
  "type": "file",
  "connection": "hdfs://localhost:8020/",
  "config": null,
  "workspaces": {
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
   ....
}

我们可以做到

➜  bin /home/hph_etl/Apps/apache-drill-1.16.0/bin/sqlline -u "jdbc:drill:zk=local;schema=dfs"
Apache Drill 1.16.0
"Got Drill?"
apache drill (dfs)> select * from dfs.`tmp/`;
Error: PERMISSION ERROR: Not authorized to read table [tmp/] in schema [dfs.default]


[Error Id: 2e248da5-ba30-43f7-a983-1784d77cf81b ] (state=,code=0)
apache drill (dfs)>

（请注意，现在我需要解决权限错误，但至少现在可以尝试查询位置）。

如何从Drill以嵌入式模式访问HDFS？

1 个答案: