Question

我正在尝试运行一个用python编写的简单火花流工作：

#!/usr/bin/env python
from pyspark import SparkContext, SparkConf
from pyspark.streaming import StreamingContext

conf = SparkConf()
conf.setMaster("spark://master1:7077,master2:7077")
sc = SparkContext(conf=conf)
ssc = StreamingContext(sc, 1)

ssc.socketTextStream("master1", 9999).count().pprint()

ssc.start()
ssc.awaitTermination()

运行几秒钟后，任务失败。以下是我看到的例外情况：

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
    at com.esotericsoftware.kryo.io.Output.flush(Output.java:155)
    at com.esotericsoftware.kryo.io.Output.require(Output.java:135)
    at com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420)
    at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326)
    at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:153)
    at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:146)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
    at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:158)
    at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
    at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1190)
    at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1199)
    at org.apache.spark.storage.MemoryStore.putArray(MemoryStore.scala:132)
    at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:169)
    at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:143)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:791)
    at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:638)
    at org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler.storeBlock(ReceivedBlockHandler.scala:77)
    at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushAndReportBlock(ReceiverSupervisorImpl.scala:156)
    at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushArrayBuffer(ReceiverSupervisorImpl.scala:127)
    at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl$$anon$3.onPushBlock(ReceiverSupervisorImpl.scala:108)
    at org.apache.spark.streaming.receiver.BlockGenerator.pushBlock(BlockGenerator.scala:294)
    at org.apache.spark.streaming.receiver.BlockGenerator.org$apache$spark$streaming$receiver$BlockGenerator$$keepPushingBlocks(BlockGenerator.scala:266)
    at org.apache.spark.streaming.receiver.BlockGenerator$$anon$1.run(BlockGenerator.scala:108)

之后会启动一项新任务，因此作业会继续运行。但是，我想知道，我错过了什么。

更新

火花defaults.conf

spark.serializer                 org.apache.spark.serializer.KryoSerializer
spark.driver.memory              4g
spark.executor.memory            4g
spark.executor.extraJavaOptions  -XX:+PrintGCDetails
spark.deploy.recoveryMode        ZOOKEEPER
spark.deploy.zookeeper.url       master1:2181,master2:2181,master3:2181

Answer 1

尝试设置执行程序内存on the application itself：

conf = SparkConf()
conf.setMaster("spark://master1:7077,master2:7077")
conf.set("spark.executor.memory", "4g")

Spark Streaming：java.lang.OutOfMemoryError：Java堆空间

1 个答案: