Question

我（尝试）使用Python2在Manjaro Linux上运行pyspark。我已经创建了一个测试脚本来创建一个SparkContext实例并再次将其停止：

import findspark
findspark.init()
from pyspark import SparkContext

if __name__ == '__main__':
    sc = SparkContext()
    sc.stop()

我从带有python2 filename.py的终端启动它。这以前曾经起作用，但由于我不明白的原因，现在提出了以下问题：

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more
Traceback (most recent call last):
  File "mwe.py", line 22, in <module>
    sc = SparkContext()
  File "/opt/apache-spark/python/pyspark/context.py", line 115, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/apache-spark/python/pyspark/context.py", line 292, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/apache-spark/python/pyspark/java_gateway.py", line 93, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

我已经读过其他有关通过使用Java版本8而不是版本9或10来解决此问题的信息。但是，我似乎已经在运行版本8，因为archlinux-java status输出：

Available Java environments:
  java-10-openjdk
  java-8-jdk
  java-8-jre/jre
  java-8-openjdk/jre (default)

我不知道如何从这里开始，因此将不胜感激。

Answer 1

我遇到了同样的问题，我通过卸载java-10-openjdk

来解决了这个问题。

创建SparkContext时，Pyspark引发JNI错误

1 个答案: