通过专用子网访问跨区域s3端点

时间:2019-04-10 08:05:16

标签: amazon-s3 amazon-emr amazon-vpc vpc private-subnet

我有一个EMR,它正在eu-west-1 私有子网中旋转。我已经在路由表中为S3定义了一个网关端点。我必须访问AWS提供的 public 存储桶/位置:s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,它给出以下错误。我认为这是因为不允许通过网关端点进行跨区域访问。 我可以访问同一地区的其他存储桶。是否有解决方法可以通过NAT访问?路由表已经有一个NAT,但是请求却不通过该地址。

2019-04-10T05:17:06.849Z INFO Ensure step 1 jar file s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar
INFO Failed to download: s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar
java.lang.RuntimeException: Error whilst fetching 's3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar'
    at aws157.instancecontroller.util.S3Wrapper.fetchS3HadoopFileToLocal(S3Wrapper.java:412)
    at aws157.instancecontroller.util.S3Wrapper.fetchHadoopFileToLocal(S3Wrapper.java:351)
    at aws157.instancecontroller.master.steprunner.HadoopJarStepRunner$Runner.<init>(HadoopJarStepRunner.java:243)
    at aws157.instancecontroller.master.steprunner.HadoopJarStepRunner.createRunner(HadoopJarStepRunner.java:152)
    at aws157.instancecontroller.master.steprunner.HadoopJarStepRunner.createRunner(HadoopJarStepRunner.java:146)
    at aws157.instancecontroller.master.steprunner.StepExecutor.runStep(StepExecutor.java:136)
    at aws157.instancecontroller.master.steprunner.StepExecutor.run(StepExecutor.java:70)
    at aws157.instancecontroller.master.steprunner.StepExecutionManager.enqueueStep(StepExecutionManager.java:248)
    at aws157.instancecontroller.master.steprunner.StepExecutionManager.doRun(StepExecutionManager.java:195)
    at aws157.instancecontroller.master.steprunner.StepExecutionManager.access$000(StepExecutionManager.java:33)
    at aws157.instancecontroller.master.steprunner.StepExecutionManager$1.run(StepExecutionManager.java:94)
Caused by: com.amazonaws.AmazonClientException: Unable to execute HTTP request: connect timed out
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:618)
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3826)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1143)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1021)
    at aws157.instancecontroller.util.S3Wrapper.copyS3ObjectToFile(S3Wrapper.java:303)
    at aws157.instancecontroller.util.S3Wrapper.getFile(S3Wrapper.java:287)
    at aws157.instancecontroller.util.S3Wrapper.fetchS3HadoopFileToLocal(S3Wrapper.java:399)
    ... 10 more

1 个答案:

答案 0 :(得分:1)

S3网关终结点永远不会尝试路由跨区域流量,但是NAT网关应该自动处理此流量。假定存在NAT网关的断言,则Unable to execute HTTP request: connect timed out表示NAT网关(或与之关联的设置)配置错误。

如评论中所述,此处的特定问题是NAT网关是在打算服务的同一子网上提供的。这不是有效的配置,因为在这种情况下,NAT网关会尝试通过自身访问Internet ...,因为它是从部署子网中获取默认路由的。

  

要创建NAT网关,必须指定NAT网关应驻留的 public 子网。

     

...

     

创建NAT网关后,必须更新与一个或多个私有子网关联的路由表,以将Internet绑定的流量指向NAT网关。这使您的专用子网中的实例可以与Internet通信。 (添加了重点)

     

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html#nat-gateway-basics