当我尝试在IntelliJ中运行FLink程序时,我不断收到此消息
org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
在寻找解决方案时,我遇到了https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/aws.html#aws-access-key-id-and-secret-access-key-not-specified 它谈论将flink指向hadoop等。那么,我需要在本地安装hadoop吗?
答案 0 :(得分:2)
通常不需要Hadoop运行Flink。但我认为您正在使用s3。 Flink通过所谓的Hadoop兼容性模式支持S3。为了使它起作用,您可能需要一些其他依赖项,并且还应将core-site.xml
添加到您将在其中定义的项目中:
<configuration>
<property>
<name>fs.s3.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>[some-key]</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>[some-key]</value>
</property>
</configuration>