Question

我已经定义了Spark消费者和Kafka生产者应用程序。但是，消息不会流入Spark。因此，我不确定到底是哪里存在问题-消费者还是生产者。我使用Kubernetes部署了Kafka，并为架构注册表定义了NodePort服务，而Kafka的链接我定义为引导服务器和架构注册表URL。该错误的原因可能是什么？

Kafka配置：

  val props = new Properties()
  props.put("bootstrap.servers", "192.168.99.100:32400")
  props.put("client.id", "avro_data")
  props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
  props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer")
  props.put("schema.registry.url", "http://192.168.99.100:32500")
val producer = new KafkaProducer[String, Object](props)

val data = new ProducerRecord[String, Object]("test_topic", getRandomAvroRecord())
producer.send(data)

火花配置：

def kafkaParams() = Map[String, Object](
      "bootstrap.servers" -> "192.168.99.100:32400",
      "schema.registry.url" -> "http://192.168.99.100:32500",
      "key.deserializer" -> classOf[StringDeserializer],
      "value.deserializer" -> classOf[KafkaAvroDeserializer],
      "group.id" -> "avro_data",
      "auto.offset.reset" -> "earliest",
      "enable.auto.commit" -> (false: java.lang.Boolean)
    )

   val lines = KafkaUtils.createDirectStream[String, Object](
      streamingContext,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe[String, Object](Array("test_topic"), kafkaParams())
    )

Kafka的节点端口服务定义（与架构注册表类似）：

kind: Service
apiVersion: v1
metadata:
  name: kafka-service
spec:
  selector:
    app: cp-kafka
    release: my-confluent-oss
  ports:
    - protocol: TCP
      targetPort: 9092
      port: 32400
      nodePort: 32400
  type: NodePort

更新：我可以在Spark应用程序日志中看到这样的消息：

INFO ConsumerCoordinator: [Consumer clientId=consumer-1, groupId=avro_data] Revoking previously assigned partitions []
INFO AbstractCoordinator: [Consumer clientId=consumer-1, groupId=avro_data] (Re-)joining group
WARN NetworkClient: [Consumer clientId=consumer-1, groupId=avro_data] Connection to node 2147483645 (/172.17.0.17:9092) could not be established. Broker may not be available.
INFO AbstractCoordinator: [Consumer clientId=consumer-1, groupId=avro_data] Group coordinator 172.17.0.17:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery
WARN NetworkClient: [Consumer clientId=consumer-1, groupId=avro_data] Connection to node 1 (/172.17.0.14:9092) could not be established. Broker may not be available.
WARN NetworkClient: [Consumer clientId=consumer-1, groupId=avro_data] Connection to node 2 (/172.17.0.17:9092) could not be established. Broker may not be available.

Kafka StatefulSet配置：

 Command:
      sh
      -exc
      export KAFKA_BROKER_ID=${HOSTNAME##*-} && \
      export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${POD_IP}:9092,EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID})) && \
      exec /etc/confluent/docker/run

    Environment:
      POD_IP:                                   (v1:status.podIP)
      HOST_IP:                                  (v1:status.hostIP)
      KAFKA_ZOOKEEPER_CONNECT:                 my-confluent-oss-cp-zookeeper-headless:2181
      KAFKA_ADVERTISED_LISTENERS:              EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID}))
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:    PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

Spark使用者无法阅读Kafka消息Scala

0 个答案: