Question

我正在尝试从EventHub读取数据，并写入2个delta湖表，下面是伪代码

// read from event hub
inputDF = spark.readstream().format(“eventhubs”).option(“consumerGroup”,”myapp”)

//write to 1 delta lake
inputDF.writestream().format(“delta”).option(“checkpointLocation”,”loc1”).start(“table_1”)

//write to 2 delta lake
inputDF.writestream().format(“delta”).option(“checkpointLocation”,”loc2”).start(“table_2”)

当我开始工作时，它失败，并在消息下方显示消息“无法找到sourceVersion”

ERROR: Query termination received for [id=5735eea9-a2c0-42bf-b368-0918985bff3e, runId=88c17d32-d5d9-46b6-bb9c-19f5ab8598c5], with exception: java.lang.IllegalStateException: Cannot find 'sourceVersion' in {"My_EventHub_Event_Name":{"2":25,"5":33,"4":35,"7":33,"1":26,"3":28,"6":30,"0":32}}
    at com.databricks.sql.transaction.tahoe.sources.DeltaSourceOffset$.validateSourceVersion(DeltaSourceOffset.scala:91)
    at com.databricks.sql.transaction.tahoe.sources.DeltaSourceOffset$.apply(DeltaSourceOffset.scala:74)
    at com.databricks.sql.transaction.tahoe.sources.DeltaSource.getBatch(DeltaSource.scala:269)

有什么想法要解决吗？

从EventHub读取并写入Delta Lake时找不到sourceVersion错误

0 个答案: