Question

将Pandas数据帧保存到内存中的gzipped csv在Python 2.7（Pandas 0.22.0）中就像这样：

from io import BytesIO
import gzip
import pandas as pd
df = pd.DataFrame.from_dict({'a': ['a', 'b', 'c']})
s = BytesIO()
f = gzip.GzipFile(fileobj=s, mode='wb', filename='file.csv')
df.to_csv(f)
s.seek(0)
content = s.getvalue()

但是，在Python 3.6（Pandas 0.22.0）中，调用to_csv时相同的代码会引发错误：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib/python3.6/site-packages/pandas/core/frame.py", line 1524, in to_csv
    formatter.save()
  File "lib/python3.6/site-packages/pandas/io/formats/format.py", line 1652, in save
    self._save()
  File "lib/python3.6/site-packages/pandas/io/formats/format.py", line 1740, in _save
    self._save_header()
  File "lib/python3.6/site-packages/pandas/io/formats/format.py", line 1708, in _save_header
    writer.writerow(encoded_labels)
  File "miniconda3/lib/python3.6/gzip.py", line 260, in write
    data = memoryview(data)
TypeError: memoryview: a bytes-like object is required, not 'str'

我该如何解决这个问题？我是否需要以某种方式更改GzipFile对象to_csv才能正确处理它？</ p>

为了澄清，我想在内存中创建gzip文件（content变量），以便稍后可以使用Boto 3 put_object将其保存到Amazon S3。

Answer 1

您可以利用StringIO：

from io import StringIO
buf = StringIO()
df.to_csv(buf)
f = gzip.GzipFile(fileobj=s, mode='wb', filename='file.csv')
f.write(buf.getvalue().encode())
f.flush()

还要注意添加的f.flush()-根据我的经验，GzipFile在某些情况下可能不会随机刷新数据，从而导致存档损坏。

或者作为基于您的代码的完整示例：

from io import BytesIO
import gzip
import pandas as pd
from io import StringIO
df = pd.DataFrame.from_dict({'a': ['a', 'b', 'c']})
s = BytesIO()
buf = StringIO()
f = gzip.GzipFile(fileobj=s, mode='wb', filename='file.csv')
df.to_csv(buf)
f.write(buf.getvalue().encode())
f.flush()
s.seek(0)
content = s.getvalue()

Answer 2

Roland Pihlakas 的回答有效，除了 gzip 文件不完整（尽管刷新）。它需要在调用 bytesio.getvalue() 之前关闭。修改代码：

df= structure(list(fruit = structure(1:3, .Label = c("Orange", "Banana", 
                                                     "Kiwi"), class = "factor"), term = c("calories", "calories", "calories"), 
                   calories = c(47, 89, 61
                   )), row.names = c(NA, -3L), class = "data.frame")


df %>%
  ggplot(aes(calories, fruit)) + 
  geom_point(mapping=aes(x=calories, y=fruit), size=5)

Pandas to_csv到Python 3中的GzipFile不起作用

2 个答案: