Question

我有一个数据框va，其中有两列vertex1和vertex2。 va有

6731710209 rows

我想计算vertex1和vertex2之间的相似度。为此，我具有以下UDF函数：

weight_udf = udf(edgeWeight, FloatType())
va = va.withColumn('similarity', weight_udf(va.vertex1, va.vertex2))

运行代码时，出现以下奇怪错误：

#
# A fatal error has been detected by the Java Runtime Environment:

    #
    #  SIGSEGV (0xb) at pc=0x00002b5fe21d0ba0, pid=67441, tid=0x00002b605e421700
    #
    # JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
    # Problematic frame:
    # J 12180 C2 org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap$1.next()Z (60 bytes) @ 0x00002b5fe21d0ba0 [0x00002b5fe21d0ac0+0xe0]
    #
    # Core dump written. Default location: /project/6008168/moudi/job/core or core.67441
    #
    # An error report file with more information is saved as:
    # /project/6008168/moudi/job/hs_err_pid67441.log
    [thread 47692633986816 also had an error]
    #
    # If you would like to submit a bug report, please visit:
    #   http://bugreport.java.com/bugreport/crash.jsp
    #
Aborted (core dumped)

我不知道是什么问题。有人可以帮忙吗？

错误：Java运行时环境已检测到致命错误

0 个答案: