Question

I have a requirement of moving data from aws s3 to vertica using aws roles.

I have created a table with admin user and created a policy with s3 full access , attached it to our vertica role and that in turn is attached vertica instances running on AWS.

VERTICA VERSION

Vertica Analytic Database v8.1.1-3

COMMAND:

COPY scratch.my_test SOURCE S3(url='https://s3.amazonaws.com/xxxx-xx/test.csv');

My Table:

Has only one column , name which is varchar(64)

My test.csv

name --> column

test --> this is my value

ERROR:

SQL Error [6085] [VP001]: [Vertica]VJDBC ERROR: Unexpected exception calling process() User Function in UDSource [Vertica]VJDBC ERROR: Unexpected exception calling process() User Function in UDSource com.vertica.util.ServerException: [Vertica]VJDBC ERROR: Unexpected exception calling process() User Function in UDSource

POLICY:

I also test with this policy , i got the same error.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "s3:", "Resource": "" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": "s3:", "Resource": [ "arn:aws:s3:::xxx-xx", "arn:aws:s3:::xxx-xx/" ] } ] }

Someone please point where i am wrong, and also some suggestions. Thanks!

Answer 1

我一点也不了解Vertica，但问题可能出在您未正确指定S3对象的URL。您已要求Vertica从HTTP URL https://s3.amazonaws.com/xxxx-xx/test.csv加载。此请求中没有身份验证。就像您从浏览器中访问URL一样，您将被拒绝访问。

我相信您应该以{{1}}的形式提供URL。

我假设您的IAM策略正确无误，并指示s3://xxxx-xx/test.csv和arn:aws:s3:::xxxx-xx（请注意后者上的结尾星号，以便将其应用于xxxx-xx存储桶下的所有对象）。 / p>

Answer 2

感谢您抽出宝贵时间检查我的问题，当我添加您建议的网址时，出现以下错误

SQL Error [6783] [VP001]: [Vertica][VJDBC](6783) ERROR: Filesystem does not support glob s3://xxx-xx/test.csv
  [Vertica][VJDBC](6783) ERROR: Filesystem does not support glob s3://xxx-xx/test.csv
    com.vertica.util.ServerException: [Vertica][VJDBC](6783) ERROR: Filesystem does not support glob s3://xxx-xx/test.csv

这帮助我发现，我应该将s3存储桶安装到vertica中

https://github.com/vertica/aws-lambda-vertica-loader#set-up-s3-bucket-mounts（此处是步骤）

讨论（https://forum.vertica.com/discussion/205657/load-data-from-amazon-s3-bucket-to-vertica）

How to send data from S3 to vertica using IAM ROLE?

2 个答案: