Pyspark错误+方法__getnewargs __（[]）不存在

Question

我正在使用Spark版本2.2.0，＆amp; Python 2.7。我正在使用pyspark和amp;来连接BigSQL。试图检索数据。以下是我使用的代码

import cPickle as cpick
import numpy as np
import pandas as pd
import time
import sys
from pyspark.sql.session import SparkSession 
spark = SparkSession.builder.getOrCreate() 
spark_train_df = spark.read.jdbc("jdbc:db2://BigSQL URL:Port:sslConnection=true;","Schema.Table",
             properties={"user": "my userid", 
                      "password": "password",
                      'driver' : 'com.ibm.db2.jcc.DB2Driver'})
spark_train_df.registerTempTable('data_table')
# query to get columns necessary to create indexes
sql = "select * FROM data_table"
train_df = spark.sql(sql)

cmr_dict = { 'date': time.strftime('%a, %b %d, %Y'),
            'description': '`cmrs` contains data from data_table',
           'cmrs': train_df}

with open('cmrs.pkl', mode='wb') as fp:
    cpick.dump(cmr_dict, fp, cpick.HIGHEST_PROTOCOL)

运行代码后，我收到错误消息

Py4JError: An error occurred while calling o79.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:272)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:748)

Answer 1

不可能挑选Spark分布式对象。这些只是JVM结构的代理，更不用说它们不包含任何数据（仅描述计算）。

如果您想挑选数据，collect并序列化结果