如何处理这个PySpark问题?

时间:2019-03-01 14:40:45

标签: python apache-spark pyspark

每次尝试在大小为0的rdd上调用一个简单操作时,都会遇到以下问题。 每当我尝试在rdd中运行操作时,都会不断出现此内存堆映射错误:例如:rdd.isEmpty()或rdd.count()

*** Error in `/home/@@@@/miniconda3/envs/Project_3/bin/python': double free or corruption (!prev): 0x00005579db3108a0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fcfce2067e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fcfce20f37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fcfce21353c]
/home/@@@@/.local/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x2902c)[0x7fcfcba6102c]
/home/@@@@/.local/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x2c787)[0x7fcfcba64787]
/home/@@@@/miniconda3/envs/Project_3/lib/python3.6/site-packages/hdbscan/_hdbscan_tree.cpython-36m-x86_64-linux-gnu.so(+0x24f40)[0x7fcfbcc5ff40]
/home/@@@@/miniconda3/envs/Project_3/lib/python3.6/site-packages/hdbscan/_hdbscan_tree.cpython-36m-x86_64-linux-gnu.so(+0x2723f)[0x7fcfbcc6223f]
/home/@@@@/miniconda3/envs/Project_3/bin/python(_PyCFunction_FastCallDict+0x115)[0x5579d8873395]
/home/@@@@/miniconda3/envs/Project_3/bin/python(+0x19cc1c)[0x5579d88fcc1c]

...............
...............

7fcfce981000-7fcfce982000 rw-p 00004000 08:02 6319360                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so
7fcfce982000-7fcfce984000 r--p 00000000 08:02 3808611                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so
7fcfce984000-7fcfce988000 r-xp 00002000 08:02 3808611                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so
7fcfce988000-7fcfce989000 r--p 00006000 08:02 3808611                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so
7fcfce989000-7fcfce98a000 ---p 00007000 08:02 3808611                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so
7fcfce98a000-7fcfce98b000 r--p 00007000 08:02 3808611                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so
7fcfce98b000-7fcfce98d000 rw-p 00008000 08:02 3808611                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so
7fcfce98d000-7fcfce98e000 r--p 00000000 08:02 6319372                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
7fcfce98e000-7fcfce990000 r-xp 00001000 08:02 6319372                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
7fcfce990000-7fcfce991000 r--p 00003000 08:02 6319372                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
7fcfce991000-7fcfce992000 r--p 00003000 08:02 6319372                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
7fcfce992000-7fcfce994000 rw-p 00004000 08:02 6319372                    /home/@@@@/miniconda3/envs/Project_3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
7fcfce994000-7fcfce99b000 r--s 00000000 08:02 5250159                    /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7fcfce99b000-7fcfce99c000 r--p 00025000 08:02 6035518                    /lib/x86_64-linux-gnu/ld-2.23.so
7fcfce99c000-7fcfce99d000 rw-p 00026000 08:02 6035518                    /lib/x86_64-linux-gnu/ld-2.23.so
7fcfce99d000-7fcfce99e000 rw-p 00000000 00:00 0
7ffcc8e26000-7ffcc8e48000 rw-p 00000000 00:00 0                          [stack]
7ffcc8f86000-7ffcc8f89000 r--p 00000000 00:00 0                          [vvar]
7ffcc8f89000-7ffcc8f8b000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[Stage 54:>                                                         (0 + 1) / 1]2019-03-01 16:16:06 ERROR Executor:91 - Exception in task 0.0 in stage 54.0 (TID 302)
org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$3.applyOrElse(PythonRunner.scala:486)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$3.applyOrElse(PythonRunner.scala:475)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:593)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
    ....
    ....
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:578)
    ... 26 more
2019-03-01 16:16:06 WARN  TaskSetManager:66 - Lost task 0.0 in stage 54.0 (TID 302, localhost, executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$3.applyOrElse(PythonRunner.scala:486)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$3.applyOrElse(PythonRunner.scala:475)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:593)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
    ...
    ...
    at org.apache.spark.api.python.PythonRDD$$anonfun$3.apply(PythonRDD.scala:153)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:578)
    ... 26 more

2019-03-01 16:16:06 ERROR TaskSetManager:70 - Task 0 in stage 54.0 failed 1 times; aborting job

它一直给我这个错误!它一直给我这个错误! 它一直给我这个错误!

0 个答案:

没有答案