在我的Spark结构化流应用程序中,我正在对三个数据集进行unionByName
:
val allRecords = ds1
.unionByName(ds2)
.unionByName(ds3)
其中ds1
等是强类型数据集。有时,一个微型批次的一个数据集为空。发生这种情况时,我会收到警告:
WARN MicroBatchExecution: Could not report metrics as number leaves in trigger logical plan did not match that of the execution plan:
logical plan leaves: (size = 8) OMITTED
execution plan leaves: (size = 7), FileScan csv OMITTED
因为这是预期的情况,有没有办法避免收到此警告?