Spring-XD Sqoop配置

时间:2016-10-28 16:03:42

标签: sqoop spring-xd

我们计划通过Spring-XD执行sqoop作业,并按照Spring-XD文档中的建议进行配置。

以下是servers.xml文件中的配置


spring:
  hadoop:
    fsUri: hdfs://nnservice
    resourceManagerHost: server-m01.mydomain.com
    resourceManagerPort: 8032
    resourceManagerSchedulerAddress: ${spring.hadoop.resourceManagerHost}:8030
    jobHistoryAddress: server-m02.mydomain.com:10020
    security:
      authMethod: kerberos
      userPrincipal: springxd@mydomain.COM
      userKeytab: /home/springxd/springxd.keytab
      namenodePrincipal: nn/_HOST@mydomain.COM
      rmManagerPrincipal: rm/_HOST@mydomain.COM
    config:
      mapreduce.framework.name: yarn
      dfs.nameservices: nnservice
      dfs.ha.namenodes.nnservice: nn1, nn2
      dfs.namenode.rpc-address.nnservice.nn1: server-m01.mydomain.com:8020
      dfs.namenode.rpc-address.nnservice.nn2: server-m02.mydomain.com:8020
      dfs.client.failover.proxy.provider.nnservice: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
      dfs.ha.automatic-failover.enabled: True
      mapreduce.application.framework.path: '/hdp/apps/2.4.2.0-258/mapreduce/mapreduce.tar.gz#mr-framework'
      mapreduce.application.classpath: '$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.4.2.0-258/hadoop/lib/hadoop-lzo-0.6.0.2.4.2.0-258.jar:/etc/hadoop/conf/secure'
      yarnApplicationClasspath: '$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*'

sqoop作业因以下错误而终止

2016-10-28T10:08:35-0400 1.3.1.RELEASE INFO main mapreduce.Job - Job job_1477527264038_0051 failed with state FAILED due to: Application application_1477527264038_0051 failed 2 times due to AM Container for appattempt_1477527264038_0051_000002 exited with  exitCode: 1

知道我们是否缺少任何配置?

0 个答案:

没有答案