Sqoop导入-连接到资源管理器时作业卡住

时间:2018-07-16 13:10:15

标签: hadoop hive yarn sqoop

我正在尝试使用Sqoop将数据从sql server导入到Hive。当我执行import命令时,sql server连接成功,但是在连接到资源管理器时,作业卡住了。这是日志:

18/07/16 12:35:20 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-jexl-2.1.1.jar
18/07/16 12:35:20 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/avro.jar
18/07/16 12:35:20 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/fastutil-6.3.jar
18/07/16 12:35:20 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-codec-1.4.jar
18/07/16 12:35:21 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

这是sqoop导入命令:

sqoop import --connect "jdbc:sqlserver://ip;databaseName=TEST" --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --username user1 --password pass1 --hive-import --create-hive-table --hive-table "customer_data_march" --table "customer_data_march_parsed" --split-by Account_Branch_Converted -m 1 --verbose

当我检查资源管理器进程分配时,它显示以下内容:

[root@quickstart /]# jps
3527
1609 ThriftServer
850 JobHistoryServer
1135 ResourceManager
2611 HRegionServer
786 Bootstrap
506 NameNode
4225 Jps
3499
950 NodeManager
382 JournalNode
1983 RunJar
3469 Bootstrap
3124 Bootstrap
1745 RunJar
2459 Bootstrap
2493 HistoryServer
248 QuorumPeerMain
1446 HMaster
634 SecondaryNameNode
297 DataNode

以下是yarn-site.xml中的配置

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

  <property>
    <name>yarn.dispatcher.exit-on-error</name>
    <value>true</value>
  </property>

  <property>
    <description>List of directories to store localized files in.</description>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>
  </property>

  <property>
    <description>Where to store container logs.</description>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/var/log/hadoop-yarn/containers</value>
  </property>

  <property>
    <description>Where to aggregate logs to.</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/var/log/hadoop-yarn/apps</value>
  </property>

  <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
     <value>
        $HADOOP_CONF_DIR,
        $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
        $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
        $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
        $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
     </value>
  </property>
<!--
 <property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>

 <property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>

 <property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>
-->
</configuration>

离开我的工作一段时间后,出现以下错误:

18/07/16 14:02:19 INFO mapreduce.Job: Job job_1531745885301_0001 running in uber mode : false
18/07/16 14:02:20 INFO mapreduce.Job:  map 0% reduce 0%
18/07/16 14:02:20 INFO mapreduce.Job: Job job_1531745885301_0001 failed with state FAILED due to: Application application_1531745885301_0001 failed 2 times due to ApplicationMaster for attempt appattempt_1531745885301_0001_000002 timed out. Failing the application.
18/07/16 14:02:20 INFO mapreduce.Job: Counters: 0
18/07/16 14:02:20 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
18/07/16 14:02:20 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 1,241.2774 seconds (0 bytes/sec)
18/07/16 14:02:20 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
18/07/16 14:02:20 INFO mapreduce.ImportJobBase: Retrieved 0 records.
18/07/16 14:02:20 ERROR tool.ImportTool: Error during import: Import job failed!

我正在使用以下命令运行docker:

 docker run --hostname=quickstart.cloudera --privileged=true -t -p 8889:8888 -i 00a03c98e0d2 /u
sr/bin/docker-quickstart

我要去哪里错了?

0 个答案:

没有答案