Question

我想添加graphframes库。通常这个库是由（例如）添加的：

 pyspark --packages graphframes:graphframes:0.7.0-spark2.4-s_2.11

然后您应该得到类似的东西：

     Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.3.0
      /_/

Using Python version 3.7.0 (default, Sep 25 2018 18:19:16)
SparkSession available as 'spark'.
>>>

我可以导入图框：>>> import graphframes

问题在我执行spark-submit test.py时出现，其中test.py是：

import numpy as np
from operator import add
from pyspark.sql import *
from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import DoubleType, StringType, IntegerType
from pyspark.sql.functions import udf
from pyspark.sql.types import *
from pyspark.sql import SQLContext
import graphframes
from graphframes import *

if __name__ == "__main__":
    print ("==================================================================")
    print ("|                            main                                |")           
    print ("==================================================================")
    # create Spark context with Spark configuration
    conf = SparkConf().setMaster("local[*]").setAppName("test").set('spark.executor.memory', '60G').set('spark.driver.memory', '60G').set('spark.driver.maxResultSize', '10G')

    spark = SparkSession.builder.master("localhost").config(conf=conf).getOrCreate()

    start_time = time.time()
    sc = spark.sparkContext
    sqlContext = SQLContext(sparkContext = sc)

    # Create a Vertex DataFrame with unique ID column "id"
    v = spark.createDataFrame([("a", "Alice", 34),("b", "Bob", 36), ("c", "Charlie", 30),], ["id", "name", "age"])
    v.show()
    # Create an Edge DataFrame with "src" and "dst" columns
    e = spark.createDataFrame([("a", "b", "friend"),("b", "c", "follow"),("c", "b", "follow"),], ["src", "dst", "relationship"])

    # Create a GraphFrame
    g = GraphFrame(v, e)

    # Query: Get in-degree of each vertex.
    g.inDegrees.show()

    # Query: Count the number of "follow" connections in the graph.
    g.edges.filter("relationship = 'follow'").count()

我收到以下异常：

import graphframes
ModuleNotFoundError: No module named 'graphframes'

我认为问题与--packages有关，无法使python软件包从Spark客户端/驱动程序可用或无法加载。

我认为应将图框添加到python路径中。

1-如何解决此问题？

如何在Windows和Linux中将建议的解决方案应用于1）？

我尝试了以下操作：

下载图框jar
提取JAR内容
导航到“ graphframe”目录并将其内容压缩在其中。
将生成的zip复制到我的主目录：/ home / tam /

在我设置的.bashrc中：

export PYSPARK_PYTHONPYSPARK_PYTHON=/home/tam/.local/easybuild/software/2017/Core/miniconda3/4.3.27/envs/testEnv/bin/python
export PYTHONPATH=${PYTHONPATH}:/home/tam/graphframes.zip

spark-submit test.py，出现以下错误：

 Traceback (most recent call last):


 File "/home/tam/test.py", line 3, in <module>
    import graphframes
  File "/home/tam/graphframes/__init__.py", line 2, in <module>
    from .graphframe import GraphFrame
  File "/home/tam/graphframes/graphframe.py", line 26, in <module>
    from graphframes.lib import Pregel
  File "/home/tam/graphframes/lib/__init__.py", line 3, in <module>
    from .pregel import Pregel
  File "/home/tam/graphframes/lib/pregel.py", line 24, in <module>
    from pyspark.ml.wrapper import JavaWrapper, _jvm
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/spark/2.3.0/python/lib/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module>
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/spark/2.3.0/python/lib/pyspark.zip/pyspark/ml/base.py", line 24, in <module>
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/spark/2.3.0/python/lib/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module>
ModuleNotFoundError: No module named 'numpy'
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Answer 1

我通过以下方法解决了这个问题：

下载图框jar
提取JAR内容
导航到“ graphframe”目录并将其内容压缩在其中。
将生成的zip复制到我的主目录：/ home / tam /

然后在.bashrc-profile中：只需将graphframes.zip的路径添加到$ PYTHONPATH

然后：运行以下命令：

spark-submit --driver-class-path /graphframes-0.7.0-spark2.3-s_2.11.jar --jars“ /graphframes-0.7.0-spark2.3-s_2的路径。 11.jar” your_script.py

请注意：

-driver-class-path用于提及“额外”的罐子，以添加到spark作业的“ driver”中。
-driver-class-path只会将jars推送到驱动程序机器。如果要将罐子发送给“执行者”，则需要使用--jar

添加以上两个参数非常重要。否则，我们将收到以下错误：

py4j.protocol.Py4JJavaError: An error occurred while calling o94.loadClass.
: java.lang.ClassNotFoundException: org.graphframes.GraphFramePythonAPI
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:214)
        at java.lang.Thread.run(Thread.java:745)

有帮助的人

为什么--packages命令使python软件包不可用或无法从Spark客户端/驱动程序加载？

1 个答案: