Question

我是整个Scala场景的新手，但到目前为止一直很喜欢这个骑行！但是，我遇到了一个问题而且还没有能够掌握原因...... 我目前正在与Kafka合作，并试图从主题中读取数据并将其传递到其他地方。

问题是：内部for-comprehension中的println输出底部的线条，正如预期的那样，但是内部的所有其他prinln都被跳过，并且函数最终没有返回任何内容（可以＆甚至在测试用例中发出一个getClass！）......可能导致它的原因是什么？我真的没想到了......

相关代码：

def tryBatchRead(maxMessages: Int = 100, skipMessageOnError: Boolean = true): List[String] = {
  var numMessages = 0L

  var list = List[String]()

  val iter = if (maxMessages >= 0) stream.slice(0, maxMessages) else stream

  for (messageAndTopic <- iter) {
    for (m <- messageAndTopic) {
      println(m.offset.toString + " --- " + new String(m.message))
      list = list ++ List(new String(m.message))
      println("DEBUG " + list)
      numMessages += 1
    }
    println("test1")
  }

  println("test2")
  println("FINISH" + list)
  connector.shutdown()
  println("test3")
  list
}

输出：

6 --- {"user":{"id":"4d9e3582-2d35-4600-b070-e4d92e42c534","age":25,"sex":"M","location":"PT"}}
DEBUG List({"user":{"id":"4d9e3582-2d35-4600-b070-e4d92e42c534","age":25,"sex":"M","location":"PT"}})
7 --- test 2
DEBUG List({"user":{"id":"4d9e3582-2d35-4600-b070-e4d92e42c534","age":25,"sex":"M","location":"PT"}}, test 2)
8 --- {"StartSurvey":{"user":{"id":"6a736fdd-79a0-466a-9030-61b5ac3a3a0e","age":25,"sex":"M","location":"PT"}}}
DEBUG List({"user":{"id":"4d9e3582-2d35-4600-b070-e4d92e42c534","age":25,"sex":"M","location":"PT"}}, test 2, {"StartSurvey":{"user":{"id":"6a736fdd-79a0-466a-9030-61b5ac3a3a0e","age":25,"sex":"M","location":"PT"}}})

感谢您的帮助！

Answer 1

我不完全确定，但很可能你在阅读了等待下一个消息的最后一条消息后阻止（kafka流基本上是无限的）。为kafka使用者配置超时，因此如果一段时间没有消息，它将放弃。有consumer.timeout.ms属性（例如，将其设置为3000 ms），一旦达到等待限制，将导致ConsumerTimeoutException。

顺便说一句，我会将您的代码重写为：

def tryBatchRead(maxMessages: Int = 100): List[String] = {
  // `.take` works fine if collection has less elements than max
  val batchStream = stream.take(maxMessages)  

  // TODO: add try/catch section, according to the above comments
  // in scala we usually write a single joined for, instead of multiple nested ones
  val batch = for {
    messageAndTopic <- batchStream.take(maxMessages)
    msg <- messageAndTopic // are you sure you can iterate message and topic? 0_o
  } yield {
    println(m.offset.toString + " --- " + new String(m.message))
    msg
  }

  println("Number of messages: " + batch.length)

  // shutdown has to be done outside, it's bad idea to implicitly tear down streams in reading function
  batch 
}

Answer 2

我认为这是一种正常的行为，因为你在一个流上做了一个理论上无限大小的流（所以它永远不会结束，或者如果它等待IO的结果就会挂起......）。恕我直言我宁愿写 for（m＆lt; - messageAndTopic.take（maxMessages）.toList）而不是 for（m＆lt; - messageAndTopic）

关于理解的奇怪问题

2 个答案: