Kafka - 在端口更改后SimpleConsumer无法正常工作

时间:2016-10-19 15:03:00

标签: apache-kafka

我有2.5版本的Hortonwork Hadoop集群堆栈,其中包括Kafka 0.10,其中我有两个运营商正在运行并正在监听PLAINTEXT://localhost:6667,但如果我将端口更改为PLAINTEXT://localhost:9092,由于公司端口限制,我无法启动之前能够使用的Spark作业(使用端口6667),收到以下错误:

16/10/19 16:30:23 INFO SimpleConsumer: Reconnect due to socket error: java.nio.channels.ClosedChannelException
Exception in thread "main" org.apache.spark.SparkException: java.nio.channels.ClosedChannelException
        at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$checkErrors$1.apply(KafkaCluster.scala:366)
        at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$checkErrors$1.apply(KafkaCluster.scala:366)
        at scala.util.Either.fold(Either.scala:97)
        at org.apache.spark.streaming.kafka.KafkaCluster$.checkErrors(KafkaCluster.scala:365)
        at org.apache.spark.streaming.kafka.DirectKafkaInputDStream$DirectKafkaInputDStreamCheckpointData.restore(DirectKafkaInputDStream.scala:197)
        at org.apache.spark.streaming.dstream.DStream.restoreCheckpointData(DStream.scala:515)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$restoreCheckpointData$2.apply(DStream.scala:516)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$restoreCheckpointData$2.apply(DStream.scala:516)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.streaming.dstream.DStream.restoreCheckpointData(DStream.scala:516)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$restoreCheckpointData$2.apply(DStream.scala:516)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$restoreCheckpointData$2.apply(DStream.scala:516)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.streaming.dstream.DStream.restoreCheckpointData(DStream.scala:516)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$restoreCheckpointData$2.apply(DStream.scala:516)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$restoreCheckpointData$2.apply(DStream.scala:516)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.streaming.dstream.DStream.restoreCheckpointData(DStream.scala:516)
        at org.apache.spark.streaming.DStreamGraph$$anonfun$restoreCheckpointData$2.apply(DStreamGraph.scala:151)
        at org.apache.spark.streaming.DStreamGraph$$anonfun$restoreCheckpointData$2.apply(DStreamGraph.scala:151)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.spark.streaming.DStreamGraph.restoreCheckpointData(DStreamGraph.scala:151)
        at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:158)
        at org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:877)
        at org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:877)
        at scala.Option.map(Option.scala:145)
        at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:877)
        at org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:775)
        at org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
        at com.ncr.dataplatform.Runner.main(Runner.java:48)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/10/19 16:30:24 INFO SparkContext: Invoking stop() from shutdown hook

在所有群集节点上,我禁用了iptablesselinux,人们可以从外部连接到Kafka新端口。 我可以从任何将运行spark作业的datanode telnet到两个端口(6667和9092)上的kafka broker,也可以telnet到我配置的端口上的Zookeeper。

知道为什么会这样吗?我无法从此错误消息中获取更多信息,而且我的想法也用完了。

0 个答案:

没有答案