Question

我基本上使用Spark的documentation here中给出的示例，内置测试流，其中一个流提前3秒（最初使用kafka但遇到了同样的问题）。结果正确地返回了匹配列，但是一段时间后返回一个带有外部空值的相同键。

这是预期的行为吗？有没有办法在匹配时排除重复的外部空结果？

代码：

val testStream = session.readStream.format("rate")
  .option("rowsPerSecond", "5").option("numPartitions", "1").load()

val impressions = testStream
  .select(
    (col("value") + 15).as("impressionAdId"),
    col("timestamp").as("impressionTime"))

val clicks = testStream
  .select(
    col("value").as("clickAdId"),
    col("timestamp").as("clickTime"))


// Apply watermarks on event-time columns
val impressionsWithWatermark =
  impressions.withWatermark("impressionTime", "20 seconds")
val clicksWithWatermark =
  clicks.withWatermark("clickTime", "30 seconds")

// Join with event-time constraints
val result = impressionsWithWatermark.join(
  clicksWithWatermark,
  expr("""
clickAdId = impressionAdId AND
clickTime >= impressionTime AND
clickTime <= impressionTime + interval 10 seconds
"""),
  joinType = "leftOuter"      // can be "inner", "leftOuter", "rightOuter"
)

val query = result.writeStream.outputMode("update").format("console").option("truncate", false).start()
query.awaitTermination()

结果：

-------------------------------------------
Batch: 19
-------------------------------------------
+--------------+-----------------------+---------+-----------------------+
|impressionAdId|impressionTime         |clickAdId|clickTime              |
+--------------+-----------------------+---------+-----------------------+
|100           |2018-05-23 22:18:38.362|100      |2018-05-23 22:18:41.362|
|101           |2018-05-23 22:18:38.562|101      |2018-05-23 22:18:41.562|
|102           |2018-05-23 22:18:38.762|102      |2018-05-23 22:18:41.762|
|103           |2018-05-23 22:18:38.962|103      |2018-05-23 22:18:41.962|
|104           |2018-05-23 22:18:39.162|104      |2018-05-23 22:18:42.162|
+--------------+-----------------------+---------+-----------------------+

-------------------------------------------
Batch: 57
-------------------------------------------
+--------------+-----------------------+---------+-----------------------+
|impressionAdId|impressionTime         |clickAdId|clickTime              |
+--------------+-----------------------+---------+-----------------------+
|290           |2018-05-23 22:19:16.362|290      |2018-05-23 22:19:19.362|
|291           |2018-05-23 22:19:16.562|291      |2018-05-23 22:19:19.562|
|292           |2018-05-23 22:19:16.762|292      |2018-05-23 22:19:19.762|
|293           |2018-05-23 22:19:16.962|293      |2018-05-23 22:19:19.962|
|294           |2018-05-23 22:19:17.162|294      |2018-05-23 22:19:20.162|
|100           |2018-05-23 22:18:38.362|null     |null                   |
|99            |2018-05-23 22:18:38.162|null     |null                   |
|103           |2018-05-23 22:18:38.962|null     |null                   |
|101           |2018-05-23 22:18:38.562|null     |null                   |
|102           |2018-05-23 22:18:38.762|null     |null                   |
+--------------+-----------------------+---------+-----------------------+

Answer 1

很遗憾，您遇到的问题是SPARK-26154，其中有patch的正确性问题，但评论有些拖延。

鉴于补丁程序有点庞大，您可能不想尝试手动将补丁程序移植回您的版本。然后，我认为您最好的选择是要求提交者尽快审查补丁，并要求将使用的内容移植回版本行。

Spark Structured Streaming左外连接返回已匹配行的外空值

1 个答案: