Spark:forEachPartition无效

时间:2016-05-30 08:22:38

标签: java apache-spark spark-streaming

我想使用foreachpartition在我的数据库中保存数据,但我注意到这个功能不起作用

RDD2.foreachRDD(new VoidFunction<JavaRDD<Object>>() {


            @Override
            public void call(JavaRDD<Object> t) throws Exception {


                t.foreachPartition(new VoidFunction<Iterator<Object>>() {

                    @Override
                    public void call(Iterator<Object> t) throws Exception {
                        System.out.println("test");
                    }   }
                );
          }});

当我运行此示例时,我的火花程序将在这些步骤中被阻止,而不显示其他RDD甚至打印测试

6/05/30 10:18:41 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
16/05/30 10:18:41 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/05/30 10:18:41 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2946 bytes)
16/05/30 10:18:41 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/05/30 10:18:41 INFO SparkContext: Starting job: foreachPartition at BrokerSpout.java:265
16/05/30 10:18:41 INFO RecurringTimer: Started timer for BlockGenerator at time 1464596321600
-------------------------------------------
Time: 1464596321500 ms
-------------------------------------------

16/05/30 10:18:41 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer()
16/05/30 10:18:41 INFO ReceiverTracker: Registered receiver for stream 0 from 10.25.30.41:59407
16/05/30 10:18:41 INFO InputInfoTracker: remove old batch metadata: 
16/05/30 10:18:41 INFO ReceiverSupervisorImpl: Starting receiver
16/05/30 10:18:41 INFO ReceiverSupervisorImpl: Called receiver onStart
16/05/30 10:18:41 INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped
16/05/30 10:18:42 INFO SparkContext: Starting job: foreachPartition at BrokerSpout.java:265
16/05/30 10:18:42 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
16/05/30 10:18:42 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks

正如你在我的记录中所看到的,它说我的火花正在等待接收器停止,但我的接收器一定不能停止,如果不是,如果我们必须停止发送器,火花流的目的是什么。

0 个答案:

没有答案