提供程序com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem无法实例化

Question

我正在尝试在Spark上安装Google Cloud Storage。我使用-libjars在hadoop类路径中添加了“ gcs-connector-latest-hadoop2.jar”。我仍然遇到GoogleCloudStorageFileSystem类未找到错误。

$ hadoop fs -libjars /PATH/gcs-connector-hadoop2-latest.jar -ls /HDFS_PATH
20/02/05 05:41:33 WARN fs.FileSystem: Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem

这里我想念什么吗？

Answer 1

如果您有权访问dataproc集群的主节点，则可以在此处/usr/lib/hadoop/lib

添加gcs连接器和符号链接。