Question

我正在尝试了解如何将多行csv文件写入谷歌云存储。我没有关注documentation

靠近这里： Unable to read csv file uploaded on google cloud storage bucket

示例：

from google.cloud import storage
from oauth2client.client import GoogleCredentials
import os

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "<pathtomycredentials>"

a=[1,2,3]

b=['a','b','c']

storage_client = storage.Client()
bucket = storage_client.get_bucket("<mybucketname>")

blob=bucket.blob("Hummingbirds/trainingdata.csv")

for eachrow in range(3):
    blob.upload_from_string(str(a[eachrow]) + "," + str(b[eachrow]))

这可以让你在谷歌云存储上获得一行

3,c

显然每次都会打开一个新文件并写下该行。

好的，添加新行delim怎么样？

for eachrow in range(3):
    blob.upload_from_string(str(a[eachrow]) + "," + str(b[eachrow]) + "\n")

添加换行符，但从头开始再次写入。

有人可以说明这种方法是什么吗？我可以将所有行组合成一个字符串，或者写一个临时文件，但这看起来非常难看。

也许打开文件？

Answer 1

from google.cloud import storage
from oauth2client.client import GoogleCredentials
import os

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "<pathtomycredentials>"

a=[1,2,3]

b=['a','b','c']

storage_client = storage.Client()
bucket = storage_client.get_bucket("<mybucketname>")

blob=bucket.blob("Hummingbirds/trainingdata.csv")

# build up the complete csv string
csv_string_to_upload = ''

for eachrow in range(3):
    # add the lines
    csv_string_to_upload = csv_string_to_upload + str(a[eachrow]) + ',' + b[eachrow] + '\n'

# upload the complete csv string
blob.upload_from_string(
            data=csv_string_to_upload,
            content_type='text/csv'
        )

Answer 2

blob.upload_from_string(data)方法创建一个新对象，其内容正是字符串data的内容。它会覆盖现有对象而不是附加。

最简单的解决方案是将整个CSV写入临时文件，然后使用blob.upload_from_filename(filename)函数将该文件上传到GCS。

Answer 3

请参考以下答案，希望对您有帮助。

import pandas as pd
 data = [['Alex','Feb',10],['Bob','jan',12]]
 df = pd.DataFrame(data,columns=['Name','Month','Age'])
 print df

输出

   Name Month  Age
0  Alex   Feb   10
1   Bob   jan   12

添加行

row = ['Sally','Oct',15]
df.loc[len(df)] = row
print df

输出

     Name Month  Age
 0   Alex   Feb   10
 1    Bob   jan   12
 2  Sally   Oct   15

使用gsutil写入/复制到GCP存储桶

  df.to_csv('text.csv', index = False)
 !gsutil cp 'text.csv' 'gs://BucketName/folderName/'

将csv写入谷歌云存储

3 个答案: