Question

Spring XD on YARN: not able to stream kafka source to hdfs sink

I have a proper HDFS Resource Manager up and running. I'm able to bring up Admin server and containers successfully in YARN.

But I'm not able to stream Kafka(source) to HDFS(sink)

I configured the custom modules provided for Kafka(source) and hdfs(sink). But when I produce a kafka message for a topic, nothing is happening in the YARN cluster.

Setup details:

HDFS / YARN apache version 2.6.0

Spring XD on YARN --- spring-xd-1.2.0.RELEASE-yarn.zip

Answer 1

我刚尝试使用Ambari部署的XD集群，然后在YARN上使用XD，两者都运行良好。对于纱线，我使用了spring-xd-1.2.1.RELEASE-yarn.zip，并且刚刚使用了ambari部署的xd-shell。我刚刚从集群中获取设置并使用了ambari的postgred数据库，我用xdjob用户创建了springxd数据库。

我的config / servers.yml在bin/xd-yarn push之前看起来像这样。

xd:
  appmasterMemory: 512M
  adminServers: 1
  adminMemory: 512M
  adminJavaOpts: -XX:MaxPermSize=128m
  adminLocality: false
  containers: 3
  containerMemory: 512M
  containerJavaOpts: -XX:MaxPermSize=128m
  containerLocality: false

spring:
  yarn:
    applicationBaseDir: /xd/yarn/
---
xd:
  container:
    groups: yarn
---
spring:
  yarn:
    siteYarnAppClasspath: "/etc/hadoop/conf,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*"
    siteMapreduceAppClasspath: "/usr/hdp/current/hadoop-mapreduce-client/*,/usr/hdp/current/hadoop-mapreduce-client/lib/*"
    config:
      mapreduce.application.framework.path: '/hdp/apps/2.2.6.0-2800/mapreduce/mapreduce.tar.gz#mr-framework'
      mapreduce.application.classpath: '$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.2.6.0-2800/hadoop/lib/hadoop-lzo-0.6.0.2.2.4.2-2.jar:/etc/hadoop/conf/secure'
---
spring:
  hadoop:
    fsUri: hdfs://ambari-2.localdomain:8020
    resourceManagerHost: ambari-3.localdomain
    resourceManagerPort: 8050
    resourceManagerSchedulerAddress: ambari-3.localdomain:8030
    jobHistoryAddress: ambari-3.localdomain:10020
---
zk:
  namespace: xd
  client:
    connect: ambari-3.localdomain:2181
    sessionTimeout: 60000
    connectionTimeout: 30000
    initialRetryWait: 1000
    retryMaxAttempts: 3

---
xd:
  customModule:
    home: ${spring.hadoop.fsUri}/xd/yarn/custom-modules

---
xd:
  transport: kafka
  messagebus:
    kafka:
      brokers: ambari-2.localdomain:6667
      zkAddress: ambari-3.localdomain:2181

---
spring:
  datasource:
    url: jdbc:postgresql://ambari-1.localdomain/xdjob
    username: springxd
    password: springxd
    driverClassName: org.postgresql.Driver
    validationQuery: select 1
---
server:
  port: 0
---
spring:
  profiles: admin
management:
  port: 0

您可以使用time来源，但我使用http来源。找到正确的地址使用runtime containers和runtime modules来查看http源的运行位置。从xd-yarn shell，您可以使用admininfo查询zk以查看xd admin正在运行的位置。这是必需的，以便您可以xd-shell与yarn上运行的管理员联系。

xd:>admin config server http://ambari-2.localdomain:50254
Successfully targeted http://ambari-2.localdomain:50254

然后让我们使用sink / source创建kafka流。

xd:>stream create httpToKafkaStream --definition "http | kafka --topic=mytopic --brokerList=ambari-2.localdomain:6667" --deploy
xd:>stream create kafkaToHdfsStream --definition "kafka --zkconnect=ambari-3.localdomain:2181 --topic=mytopic --outputType=text/

xd:>http post --target http://ambari-5.localdomain:9000 --data "message1"
xd:>http post --target http://ambari-5.localdomain:9000 --data "message2"
xd:>http post --target http://ambari-5.localdomain:9000 --data "message3"
xd:>http post --target http://ambari-5.localdomain:9000 --data "message4"
xd:>http post --target http://ambari-5.localdomain:9000 --data "message5"

xd:>hadoop fs cat /xd/kafkaToHdfsStream/kafkaToHdfsStream-0.txt
message1
message2
message3
message4
message5

注意：您需要创建mytopic，如果它不存在，当前kafka源将无法启动。

Spring XD on YARN: not able to stream kafka source to hdfs sink

1 个答案: