Question

我按照https://spark.apache.org/docs/2.1.0/streaming-kafka-0-10-integration.html的说明在java中实现了一个火花流工作。当Kafka和Cassandra服务器是独立的时，它可以很好地工作。日志摘录

...
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

18/02/04 20:40:35 INFO AppInfoParser: Kafka version : 0.11.0.0
18/02/04 20:40:35 INFO AppInfoParser: Kafka commitId : cb8625948210849f
18/02/04 20:40:35 INFO AbstractCoordinator: Discovered coordinator spark-vm:9092 (id: 2147483647 rack: null) for group group_1.
18/02/04 20:40:35 INFO ConsumerCoordinator: Revoking previously assigned partitions [] for group group_1
18/02/04 20:40:35 INFO AbstractCoordinator: (Re-)joining group group_1
[2018-02-04 20:40:35,929] INFO [GroupCoordinator 0]: Preparing to rebalance group group_1 with old generation 6 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
[2018-02-04 20:40:35,946] INFO [GroupCoordinator 0]: Stabilized group group_1 generation 7 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
...

当我在mesos集群中部署时出现问题，因为作业连接到kafka但它没有发现任何协调器

metrics.sample.window.ms = 30000
auto.offset.reset = latest

18/02/04 19:27:41 INFO AppInfoParser: Kafka version : 0.10.0.1
18/02/04 19:27:41 INFO AppInfoParser: Kafka commitId : a7a17cdec9eaa6c5
18/02/04 19:27:42 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_RUNNING
18/02/04 19:27:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.65.111:50842) with ID 1
18/02/04 19:27:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.65.111:39485 with 366.3 MB RAM, BlockManagerId(1, 192.168.65.111, 39485, None)
18/02/04 19:27:47 INFO CassandraConnector: Disconnected from Cassandra cluster: cassandra
2018-02-04 19:32:19,161:7(0x7fb4fd02b700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 12ms

Kafka使用的ZK是master.mesos：2181 / dcos-service-kafka但是我不知道如何在Spark作业中配置它，我甚至不确定这是问题还是只是一个kafka版本问题

Answer 1

问题是连通性问题。 Spark作业无法使用DC / OS提供的VIP端点连接到Kafka，但日志并没有像这些技术中的新人所期望的那样抛出任何异常。

在DC / OS（Mesos）中部署时，Spark Kafka流式作业不会发现协调器

1 个答案: