Question

我试图访问Watson Data Platform目录中的csv文件。我使用了DSX笔记本中的代码生成功能：Insert to code＆gt; Insert StreamingBody object。

生成的代码是：

import os
import types
import pandas as pd
import boto3

def __iter__(self): return 0

# @hidden_cell
# The following code accesses a file in your IBM Cloud Object Storage. It includes your credentials.
# You might want to remove those credentials before you share your notebook.

os.environ['AWS_ACCESS_KEY_ID'] = '******'
os.environ['AWS_SECRET_ACCESS_KEY'] = '******'
endpoint = 's3-api.us-geo.objectstorage.softlayer.net'

bucket = 'catalog-test'

cos_12345 = boto3.resource('s3', endpoint_url=endpoint)
body = cos_12345.Object(bucket,'my.csv').get()['Body']

# add missing __iter__ method so pandas accepts body as file-like object
if not hasattr(body, "__iter__"): body.__iter__ = types.MethodType(__iter__, body)

df_data_2 = pd.read_csv(body)
df_data_2.head()

当我尝试运行此代码时，我得到：

/usr/local/src/conda3_runtime.v27/4.1.1/lib/python3.5/site-packages/botocore/endpoint.py in create_endpoint(self, service_model, region_name, endpoint_url, verify, response_parser_factory, timeout, max_pool_connections)
    270         if not is_valid_endpoint_url(endpoint_url):
    271 
--> 272             raise ValueError("Invalid endpoint: %s" % endpoint_url)
    273         return Endpoint(
    274             endpoint_url,

ValueError: Invalid endpoint: s3-api.us-geo.objectstorage.service.networklayer.com

奇怪的是，如果我为SparkSession设置生成代码，则使用相同的端点但是spark代码运行正常。

如何解决此问题？

我假设其他Softlayer端点会遇到同样的问题，所以我也在这里列出它们以确保这个问题也适用于其他软件层位置：

s3-api.us-geo.objectstorage.softlayer.net
s3-api.dal-us-geo.objectstorage.softlayer.net
s3-api.sjc-us-geo.objectstorage.softlayer.net
s3-api.wdc-us-geo.objectstorage.softlayer.net
s3.us-south.objectstorage.softlayer.net
s3.us-east.objectstorage.softlayer.net
s3.eu-geo.objectstorage.softlayer.net
s3.ams-eu-geo.objectstorage.softlayer.net
s3.fra-eu-geo.objectstorage.softlayer.net
s3.mil-eu-geo.objectstorage.softlayer.net
s3.eu-gb.objectstorage.softlayer.net

Answer 1

解决方案是在端点前加https://，从......改为

此

endpoint = 's3-api.us-geo.objectstorage.softlayer.net'

到

endpoint = 'https://s3-api.us-geo.objectstorage.softlayer.net'

Answer 2

对于IBM Cloud Object Storage，它应该是import ibm_boto3而不是import boto3。原始boto3用于访问AWS，后者使用不同的身份验证。也许这两者对端点值有不同的解释。

ValueError：无效的端点：s3-api.xxxx.objectstorage.service.networklayer.com

2 个答案: