我在CoGroupFunction中使用广播变量时遇到错误。如果注释df['timestamp'].diff() # .select(1 < x < 10)
,则错误消失。你知道如何解决这个问题吗?是与变量定义或初始化相关的错误吗?
appProvider.value()
我收到此错误
public class UsageJobDS implements Serializable{
private static final Logger log = org.apache.log4j.LogManager.getLogger("myLogger");
Broadcast<Provider> appProvider;
void init(){
// init broadcast variable
....
}
public static void main(String[] args) {
UsageJobDS ujb = new UsageJobDS();
ujb.init();
ujb.run();
}
void run(){
KeyValueGroupedDataset<Long, Row> charges = usageCharges.groupByKey(x -> x.getLong(x.fieldIndex("si__subscription_id")), Encoders.LONG());
Dataset<ProcessEdr> cogg = edrs.cogroup(charges, rateEDRs, Encoders.bean(ProcessEdr.class));
log.warn("Count cogg " + cogg.count());
}
CoGroupFunction<Long, EDR2, Row, ProcessEdr> rateEDRs = (subscription_id, edrsIter, chargesIter) -> {
Logger log = org.apache.log4j.LogManager.getLogger("myLogger");
log.warn("inside rateEDRs function");
while (edrsIter.hasNext()) {
appProvider.value(); // HERE
}
return results.iterator();
};
}
答案 0 :(得分:0)
实际上,如果将cogroup函数定义更改为以下代码,则它可以正常工作。但是,错误原因仍然未知。
Dataset<ProcessEdr> cogg = edrs.cogroup(charges, (subscription_id, edrsIter, chargesIter) -> {
ArrayList<ProcessEdr> results = new ArrayList<>();
System.out.println("App Provider name" + appProvider.value().getIssuer_name());
return results.iterator();
}, Encoders.bean(ProcessEdr.class));