Question

我有一个具有1000万条记录和150列的spark数据框。我正在尝试将其转换为熊猫DF。

x = df.toPandas()
# do some things to x

它以ordinal must be >= 1失败。我认为这是因为要一次处理才大。是否可以对每个块进行分块并将其转换为熊猫DF？

完整堆栈：

ValueError                                Traceback (most recent call last)
<command-2054265283599157> in <module>()
    158 from db.table where snapshot_year_month=201806""")
--> 159 ps = x.toPandas()
    160 # ps[["pol_nbr",
    161 # "pol_eff_dt",

/databricks/spark/python/pyspark/sql/dataframe.py in toPandas(self)
   2029                 raise RuntimeError("%s\n%s" % (_exception_message(e), msg))
   2030         else:
-> 2031             pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
   2032 
   2033             dtype = {}

/databricks/spark/python/pyspark/sql/dataframe.py in collect(self)
    480         with SCCallSiteSync(self._sc) as css:
    481             port = self._jdf.collectToPython()
--> 482         return list(_load_from_socket(port, BatchedSerializer(PickleSerializer())))
    483

Answer 1

假设您的表具有整数键/索引，则可以使用循环+查询读取大数据帧的块。

我远离df.toPandas()，这会带来很多开销。相反，我有一个辅助函数，可将pyspark实例列表的Row查询结果转换为pandas.DataFrame。

In [1]: from pyspark.sql.functions import col

In [2]: from pyspark.sql import SparkSession

In [3]: import numpy as np

In [4]: import pandas as pd

In [5]: def to_pandas(rows):
       :     row_dicts = [r.asDict() for r in rows]
       :     return pd.DataFrame.from_dict(row_dicts)
       :

要查看此功能的实际效果，让我们制作一个小的示例数据框。

In [6]: from string import ascii_letters
       : n = len(ascii_letters)
       : df = pd.DataFrame({'id': range(n),
       :                    'num': np.random.normal(10,1,n),
       :                    'txt': list(ascii_letters)})
       : df.head()
Out [7]:
   id        num txt
0   0   9.712229   a
1   1  10.281259   b
2   2   8.342029   c
3   3  11.115702   d
4   4  11.306763   e


In [ 8]: spark = SparkSession.builder.appName('Ops').getOrCreate()
       : df_spark = spark.createDataFrame(df)
       : df_spark
Out[ 9]: DataFrame[id: bigint, num: double, txt: string]

通过对索引进行过滤来收集大块。

In [10]: chunksize = 25
       : for i in range(0, n, chunksize):
       :     chunk = (df_spark.
       :               where(col('id').between(i, i + chunksize)).
       :               collect())
       :     pd_df = to_pandas(chunk)
       :     print(pd_df.num.mean())
       :
9.779573360741152
10.23157424753804
9.550750629366462

Spark Dataframe中的块topandas

1 个答案: