我正在使用带有java api的spark sql。我正在尝试广播数据集并使用boradcasted datset。 这是造成问题的相同代码。
Dataset<Rules> rulesDS= loadTrustRulesAsDataset("Rules.csv");
final Broadcast<Dataset<Rules>> broadcastTrustRulesDS = sqlcontext.broadcast(rulesDS);
Dataset<Rules> ds = broadcastTrustRulesDS.getValue();
ds.show();
正如评论部分所述,我更新了以下代码
Dataset<Rules> broadcastTrustRulesDS = org.apache.spark.sql.functions.broadcast(rulesDS);
Dataset<Rules> ds = broadcastTrustRulesDS.value();
ds.show();
在ds.show()
中抛出Nullpointer Exception这个ds.show()没有给出任何结果。
当我在日食中运行时,下面的消息显示在控制台中。
18/05/03 09:51:31 WARN NettyUtil: Found Netty's native epoll transport, but not running on linux-based operating system. Using NIO instead.
18/05/03 09:51:32 INFO Cluster: New Cassandra host localhost/127.0.0.1:9042 added
18/05/03 09:51:32 INFO CassandraConnector: Connected to Cassandra cluster: Test Cluster
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]
[Stage 16:=======================================> (7 + 3) / 10]