Question

我正在我的本地计算机（Windows上的Ubuntu）和hortonworks HDP沙箱上测试此spark脚本。结果是一样的。

这是它的样子：

from pyspark.sql import SparkSession
from pyspark.sql.functions import split
from pyspark.ml.feature import Word2Vec
import sys

spark = SparkSession \
    .builder \
    .appName("Spark job") \
    .getOrCreate()

DIMENSIONALITY = 300


def main(argv):
    filename = argv[1]
    df = spark.read.text(filename)

    tokens = df.select(split(df.value, " ").alias("words"))

    tokens.show()
    tokens.printSchema()
    print(type(tokens))

    w2vec = Word2Vec(vectorSize=DIMENSIONALITY, inputCol="words", outputCol="vectors")
    model = w2vec.fit(tokens)

fit方法要求提供docs

中所示的DataFrame

fit调用之前的执行输出如下：

+--------------------+
|               words|
+--------------------+
|[Anca, TAPERL, Pr...|
|[Anca, , Protesi,...|
|[Anca, FITMORE, P...|
......
+--------------------+
only showing top 20 rows

root
 |-- words: array (nullable = true)
 |    |-- element: string (containsNull = true)

<class 'pyspark.sql.dataframe.DataFrame'>

最后，我收到错误：

Traceback (most recent call last):
  File "src/word2vec.py", line 33, in <module>
    main(sys.argv)
  File "src/word2vec.py", line 25, in main
    model = w2vec.fit(tokens)
  File "/home/proch/.local/lib/python2.7/site-packages/pyspark/ml/base.py", line 132, in fit
    return self._fit(dataset)
  File "/home/proch/.local/lib/python2.7/site-packages/pyspark/ml/wrapper.py", line 288, in _fit
    java_model = self._fit_java(dataset)
  File "/home/proch/.local/lib/python2.7/site-packages/pyspark/ml/wrapper.py", line 285, in _fit_java
    return self._java_obj.fit(dataset._jdf)
  File "/home/proch/.local/lib/python2.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/home/proch/.local/lib/python2.7/site-packages/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/home/proch/.local/lib/python2.7/site-packages/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o43.fit.
: java.lang.IllegalArgumentException
        at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
        at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
        at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
        at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:46)
        at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:449)
        at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:432)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
        at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:103)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:432)
        at org.apache.xbean.asm5.ClassReader.a(Unknown Source)
        at org.apache.xbean.asm5.ClassReader.b(Unknown Source)
        at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
        at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
        at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCle
aner.scala:262)
        at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCle
aner.scala:261)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:261)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2299)
        at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:388)
        at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:387)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.filter(RDD.scala:387)
        at org.apache.spark.mllib.feature.Word2Vec.learnVocab(Word2Vec.scala:189)
        at org.apache.spark.mllib.feature.Word2Vec.fit(Word2Vec.scala:309)
        at org.apache.spark.ml.feature.Word2Vec.fit(Word2Vec.scala:186)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.base/java.lang.Thread.run(Thread.java:844)

这里是什么问题？

谢谢

pyspark：在word2vec上调用fit函数时出错IllegalArgumentException

0 个答案: