Question

我设法将具有3个数据节点的Hadoop设置为一个小型集群，并且一切正常。尝试通过S3A协议访问AWS存储桶时，出现此错误：

hadoop fs -ls s3a：// my-bucket /

-ls: Fatal internal error
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class 
org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2395)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3208)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3240)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3291)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3259)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:470)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
    at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
    at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:245)
    at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:228)
    at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
    at org.apache.hadoop.fs.shell.Command.run(Command.java:175)
    at org.apache.hadoop.fs.FsShell.run(FsShell.java:317)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
    at org.apache.hadoop.fs.FsShell.main(FsShell.java:380)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2299)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2393)
    ... 16 more

我做错了什么？如何解决？

P.S。如果完全公开，请在Amazon上进行存储。任何人都可以从中下载。

按照以下说明在hadoop/core-site.xml中配置了Amazon凭证：Hadoop-AWS module: Integration with Amazon Web Services

Answer 1

根据您共享的链接，问题似乎与CLASSPATH中缺少的JAR文件有关。您可以检查它是否可以访问。如果不是，则可以复制所需的JARS，如下所示，使其与Hadoop版本匹配并重试。

sudo cp hadoop/share/hadoop/tools/lib/$AWS_JAVA_SDK_VERSION.jar hadoop/share/hadoop/common/lib/

sudo cp hadoop/share/hadoop/tools/lib/$AWS_HADOOP_VERSION.jar hadoop/share/hadoop/common/lib/

Hadoop 2.9.2 AWS

1 个答案: