Question

我正在运行以下代码（包含大约10000行）：

IndexedRowMatrix qTmp = new IndexedRowMatrix(a.rows());
IndexedRowMatrix qTmpT = qTmp.toCoordinateMatrix().transpose().toIndexedRowMatrix();

我收到以下错误：

15/06/19 10:08:52 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (1 time so far)
15/06/19 10:08:54 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (2 times so far)
15/06/19 10:08:57 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (3 times so far)
15/06/19 10:09:02 ERROR Executor: Exception in task 0.0 in stage 9.0 (TID 9)
java.lang.OutOfMemoryError: Java heap space

我也试过用各种参数调整内存：

SparkConf conf = new SparkConf();
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.setAppName("test");
conf.setMaster("local[2]");
conf.set("spark.executor.memory", "6g");
conf.set("spark.driver.maxResultSize", "1g");
conf.set("spark.shuffle.spill", "false");
conf.set("spark.storage.memoryFraction", "0.3");
conf.set("spark.eventLog.enabled", "false");
conf.set("driver-memory", "4g")`;
JavaSparkContext sc = new JavaSparkContext(conf);

我是否有理由不断遇到此溢出错误？

Answer 1

您应该使用重写的java选项运行您的应用程序。

使用-Xms = [size] -Xmx -XX：MaxPermSize = [size] -XX：MaxNewSize = [size]

启动程序

例如 -

-Xms512m -Xmx1152m -XX：MaxPermSize = 256m -XX：MaxNewSize = 256m

Java Spark溢出内存映射OutOfMemoryError错误

1 个答案: