Question

我们正在通过以下步骤开展大数据应用： 1）读取大约500 Mb大小的json文件。 2）经过一些操作后，我们创建5个orc文件并保存在hdfs上。如果我们只保存1个文件则需要80秒。对于orc保存文件，我们使用多线程。问题是5个orc文件保存需要花费太多时间约160秒。我们的代码如下所述。

    class SaveTask implements Runnable{
     private final String savePath;
     private Dataset<Row> data;
     public SaveTask(Dataset<Row> data, String savePath) {
      this.data = data;
      this.savePath = savePath;
}

@Override
public void run() {
    // TODO Auto-generated method stub
    System.out.println(Thread.currentThread().getName());
    try {

        data.write().mode(SaveMode.Append).format("orc").partitionBy("choiceId").save(this.savePath);
        System.out.println("Saved in HDFS::: " + savePath);

    } catch (final Exception e) {
    throw new RuntimeException(e);
    }

}

} 驱动程序是：

    public static void multiThreadRun(Dataset<Row> data){


    ExecutorService executor = Executors.newFixedThreadPool(5);
    for (int i = 0; i < 5; i++) {
        Runnable task = new SaveTask(data, "hdfs:/user/root/AIR_MAIN/AIR"+i+".snappy.orc");
        executor.execute(task);
    }  
    executor.shutdown();
    // Wait until all threads are finish
    try {
        executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
    } catch(InterruptedException e){

    }

    System.out.println("Finished all threads");

}

我们的火花提交脚本是：

    ./bin/spark-submit \
    --class com.ibm.spark.etl.SparkCCNMainJsonProcessor --master yarn \
    --conf spark.driver.cores=4 \
    --deploy-mode cluster \
    --executor-memory 6g \
    --num-executors 2 \
    /home/spark/deployment/multithreaded-spark-0.0.1-SNAPSHOT.jar

有关orc文件保存时间优化的任何帮助/建议???

Apache Spark Orc文件保存需要太多时间

0 个答案: