Question

我有一个csv文件，大约有200多个cols和1mil +的行。当我从csv转换为python时，出现错误：

     csv_file = 'bigcut.csv'
     chunksize = 100_000
     parquet_file ='output.parquet'
     parser=argparse.ArgumentParser(description='Process Arguments')
     parser.add_argument("--fname",action="store",default="",help="specify <run/update>")
     args=parser.parse_args()
     argFname=args.__dict__["fname"]
     csv_file=argFname
     csv_stream = pd.read_csv(csv_file, encoding = 'utf-8',sep=',', >chunksize=chunksize, low_memory=False)
     for i, chunk in enumerate(csv_stream):
        print("Chunk", i)
        if i==0:
           parquet_schema = pa.Table.from_pandas(df=chunk).schema
           parquet_writer = pq.ParquetWriter(parquet_file, parquet_schema, compression='snappy')
        table = pa.Table.from_pandas(chunk, schema=parquet_schema)
        parquet_writer.write_table(table)
     parquet_writer.close()

我跑步时会产生以下错误

    File "pyconv.py", line 25, in <module>
table = pa.Table.from_pandas(chunk, schema=parquet_schema)
    File "pyarrow/table.pxi", line 1217, in pyarrow.lib.Table.from_pandas
    File "/home/cloud-user/pydev/py36-venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 387, in dataframe_to_arrays
convert_types))
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
yield fs.pop().result()
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
    File "/home/cloud-user/pydev/py36-venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 376, in convert_column
raise e
    File "/home/cloud-user/pydev/py36-venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 370, in convert_column
return pa.array(col, type=ty, from_pandas=True, safe=safe)
     File "pyarrow/array.pxi", line 169, in pyarrow.lib.array
     File "pyarrow/array.pxi", line 69, in pyarrow.lib._ndarray_to_array
     File "pyarrow/error.pxi", line 81, in pyarrow.lib.check_status
      pyarrow.lib.ArrowInvalid: ("'utf-32-le' codec can't decode bytes in position 0-3: code point not in range(0x110000)", 'Conversion failed for column agent_number__c with type float64')

我是新的pandas / pyarrow / python，如果有人有任何建议，我下一步应该做什么调试工作将受到赞赏。

Answer 1

'utf-32-le'编解码器无法解码位置0-3的字节

似乎库正在尝试对utf-32-le中的数据进行解码，而您将csv数据读取为utf-8。

因此，您将不得不以某种方式告诉读者（pyarrow.lib）以utf-8的身份读取（我不知道Python / Parquet，因此我无法提供执行此操作的确切代码）。 / p>

Answer 2

csv大约有3mils的记录。我设法发现了1个潜在的问题。

列1的col的数据类型为字符串/文本。它们中的大多数以某种方式是数字的，但是其中一些是与文本混合的，例如，其中许多是1000,230,400等，但是却很少输入，例如5k，100k，29k。

因此代码不喜欢它尝试将其设置为数字/整数。

你能建议吗？

使用python错误从大型csv转换为实木复合地板

2 个答案: