Spark"尚未开始任务"

时间:2016-08-24 20:48:53

标签: apache-spark rdd spark-dataframe

我是初学者,在独立模式下火花和运行火花。任务dataframe.count()挂起

    SparkConf conf = new SparkConf();
    conf.set("spark.driver.allowMultipleContexts", "true");
    conf.set("spark.executor.memory", "10g");
    conf.set("spark.dirver.maxResultSize","10g");
    conf.set("spark.driver.memory ", "10g");
    //Initialize sparkcontext
    Dataframt dt = //load data from reshift
    JavaRDD<String> rdd = sc.textFile(url);
    JavaPairRDD<String, String> pairRdd = rdd.mapToPair(SparkFunctionsImpl
                .strToMap());

   //dt.count()
   //pairrdd => map => collectAsMap()

Spark任务在count()和collectasMap()处挂起,并且不从那里继续。 enter image description here

看起来rdd.collectasMap和dataframe.count()正在并行执行,并且spark挂起,没有任务正在进行

0 个答案:

没有答案