将csv写入来自DSX python 2.7 notebook的Ibm bluemix对​​象存储

Question

我正在尝试从DSX Python笔记本中将pandas数据帧作为CSV写入Bluemix对​​象存储。我首先将数据帧保存为“本地”CSV文件。然后我有一个例程尝试将文件写入对象存储。我得到413响应 - 对象太大了。该文件只有大约3MB。这是我的代码，基于我在这里找到的JSON示例：http://datascience.ibm.com/blog/working-with-object-storage-in-data-science-experience-python-edition/

sys.path.insert(0, os.path.abspath('../../'))
sys.path.insert(1, '<your local full path>/google_appengine')
sys.path.insert(1, '<your local full path>/google_appengine/lib/yaml/lib')

if 'google' in sys.modules:
    del sys.modules['google']

非常感谢任何帮助或指示。

Answer 1

tutorial的这段代码片段对我来说很好（对于一个12 MB的文件）。

from io import BytesIO  
import requests  
import json  
import pandas as pd

def put_file(credentials, local_file_name):  
    """This functions returns a StringIO object containing
    the file content from Bluemix Object Storage V3."""
    f = open(local_file_name,'r')
    my_data = f.read()
    url1 = ''.join(['https://identity.open.softlayer.com', '/v3/auth/tokens'])
    data = {'auth': {'identity': {'methods': ['password'],
            'password': {'user': {'name': credentials['username'],'domain': {'id': credentials['domain_id']},
            'password': credentials['password']}}}}}
    headers1 = {'Content-Type': 'application/csv'}
    resp1 = requests.post(url=url1, data=json.dumps(data), headers=headers1)
    resp1_body = resp1.json()
    for e1 in resp1_body['token']['catalog']:
        if(e1['type']=='object-store'):
            for e2 in e1['endpoints']:
                        if(e2['interface']=='public'and e2['region']=='dallas'):
                            url2 = ''.join([e2['url'],'/', credentials['container'], '/', local_file_name])
    s_subject_token = resp1.headers['x-subject-token']
    headers2 = {'X-Auth-Token': s_subject_token, 'accept': 'application/json'}
    resp2 = requests.put(url=url2, headers=headers2, data = my_data )
    print resp2

我使用以下方法创建了一个随机的pandas数据帧：

df = pd.DataFrame(np.random.randint(0,100,size=(1000000, 4)), columns=list('ABCD'))

将其保存到csv

df.to_csv('myPandasData_1000000.csv',index=False)

然后将其放到对象存储

put_file(credentials_1,'myPandasData_1000000.csv')

您可以通过单击对象库中任何对象的credentials_1来获取insert to code -> Insert credentials对象。