Question

尝试在Ubuntu 16.04中从sql server 2017创建外部表到Hadoop时会抛出以下错误

Msg 105019，Level 16，State 1，Line 1 由于内部错误，EXTERNAL TABLE访问失败：＆＃39;调用HdfsBridge_IsDirExist时出现Java异常。 Java异常消息：从DESKTOP-VE8KNAG / xxx.xxx.x.xxx调用到xxx.xxx.x.x：54310连接异常失败：java.net.ConnectException：连接被拒绝：没有进一步的信息;有关详细信息，请参阅：http://wiki.apache.org/hadoop/ConnectionRefused：错误[从DESKTOP-VE8KNAG / 1xxx.xxx.x.xxx调用到xxx.xxx.xx：54310在连接异常时失败：java.net.ConnectException：连接被拒绝：没有进一步信息;有关详细信息，请参阅：访问外部文件时发生http://wiki.apache.org/hadoop/ConnectionRefused]。＆＃39;

从core-site.xml
创建Tmp文件夹并向用户添加权限并将其添加到core-site.xml
所有节点都在Hadoop中运行 10625 DataNode 10869 SecondaryNameNode 17113 ResourceManager 17434 NodeManager 10490 NameNode 21566 Jps

sql查询

create EXTERNAL DATA SOURCE [HDP2] 
WITH (TYPE = HADOOP,
 LOCATION = N'hdfs://xxx.xxx.x.x:54310',
 CREDENTIAL = [HDPUser])
GO

create EXTERNAL TABLE [dbo].CLASS_DIM_EXP (
    [CLASS_ID] [varchar](8) NOT NULL,
    [CLASS_DESC] [varchar](100) NULL,
    [INSERT_DATE] [datetime2](7) NOT NULL,
    [LAST_UPDATE_DATETIME] [datetime2](7) NOT NULL)
WITH   (LOCATION='/user/pdw_user',
DATA_SOURCE = HDP2,
FILE_FORMAT = TSV,
REJECT_TYPE = VALUE,
REJECT_VALUE = 0);

核心-site.xml中

<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

这是否需要改变？

Answer 1

这里的问题是cores-site.xml文件包含hdfs：// localhost：54310需要将其替换为相应的IP地址hdfs：//xxx.xxx.x.x：54310。

从polybase连接到hadoop

1 个答案: