Question

我试图使用spring数据hadoop向HDFS写一个简单的文本。但是我在写作时遇到了一个未知问题。

线程中的异常＆＃34; main＆＃34; org.springframework.data.hadoop.store.StoreException：存储输出上下文尚未初始化;嵌套异常是java.io.IOException：意外的HTTP响应：代码= 404！= 200，op = GETFILESTATUS，消息=未找到 org.springframework.data.hadoop.store.support.OutputStoreObjectSupport.getOutputContext（OutputStoreObjectSupport.java:135）在 org.springframework.data.hadoop.store.output.AbstractDataStreamWriter.getOutput（AbstractDataStreamWriter.java:131）在 org.springframework.data.hadoop.store.output.TextFileWriter.write（TextFileWriter.java:132）在com.mstack.app.MainApp.someMethod（MainApp.java:37）at com.mstack.app.MainApp.main（MainApp.java:32）引起： java.io.IOException：意外的HTTP响应：code = 404！= 200， op = GETFILESTATUS，message =未找到 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse（WebHdfsFileSystem.java:347）在 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access 200 $（WebHdfsFileSystem.java:90）在 org.apache.hadoop.hdfs.web.WebHdfsFileSystem $ AbstractRunner.runWithRetry（WebHdfsFileSystem.java:613）在 org.apache.hadoop.hdfs.web.WebHdfsFileSystem $ AbstractRunner.access $ 100（WebHdfsFileSystem.java:463）在 org.apache.hadoop.hdfs.web.WebHdfsFileSystem $ AbstractRunner $ 1.run（WebHdfsFileSystem.java:492）在java.security.AccessController.doPrivileged（Native Method）at javax.security.auth.Subject.doAs（Subject.java:422）at org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1657）在 org.apache.hadoop.hdfs.web.WebHdfsFileSystem $ AbstractRunner.run（WebHdfsFileSystem.java:488）在 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus（WebHdfsFileSystem.java:848）在 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus（WebHdfsFileSystem.java:858）在org.apache.hadoop.fs.FileSystem.exists（FileSystem.java:1424）at org.springframework.data.hadoop.store.support.OutputStoreObjectSupport.findInitFiles（OutputStoreObjectSupport.java:111）在 org.springframework.data.hadoop.store.support.OutputStoreObjectSupport.initOutputContext（OutputStoreObjectSupport.java:93）在 org.springframework.data.hadoop.store.support.OutputStoreObjectSupport.getOutputContext（OutputStoreObjectSupport.java:133） ... 4更多引起：java.io.IOException：Content-Type＆＃34; text / plain＆＃34; 与＆＃34; application / json＆＃34;不兼容（解析=＆＃34; text / plain＆＃34;）at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse（WebHdfsFileSystem.java:320）在 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse（WebHdfsFileSystem.java:343） ......还有18个

我的application-context.xml： -

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/hadoop"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:beans="http://www.springframework.org/schema/beans"
    xmlns:hdp="http://www.springframework.org/schema/hadoop" xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">


    <hdp:configuration id="hadoopConfigBean">
        fs.defaultFS=${hdp.fs}
    </hdp:configuration>

    <context:annotation-config />
    <beans:bean id="textFileWriter"
        class="org.springframework.data.hadoop.store.output.TextFileWriter">
        <beans:constructor-arg index="0" ref="hadoopConfigBean"></beans:constructor-arg>
        <beans:constructor-arg index="1"
            type="org.apache.hadoop.fs.Path" value="/user/mhduser"></beans:constructor-arg>
        <beans:constructor-arg index="2" type="org.springframework.data.hadoop.store.codec.CodecInfo" >
        <beans:null></beans:null>
        </beans:constructor-arg>
    </beans:bean>

    <context:property-placeholder location="hadoop-configs.properties" />
</beans:beans>

主要课程： -

public class MainApp {

    @Autowired
    TextFileWriter textFileWriter;

    public static void main(String[] args) {
        ApplicationContext context = new ClassPathXmlApplicationContext("/META-INF/spring/application-context.xml",
                MainApp.class);
        System.out.println("Context loaded...");
        MainApp obj = new MainApp();
        context.getAutowireCapableBeanFactory().autowireBean(obj);
        obj.someMethod();
    }

    private void someMethod() {
        try {
            textFileWriter.write("Something");
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

任何帮助将不胜感激！感谢

Answer 1

我们在这里有一些样本https://github.com/spring-projects/spring-hadoop-samples。不考虑任何特定的存储，所以我创建了一个基于Spring Boot的简单要点，可以通过CLI运行。 https://gist.github.com/jvalkeal/8145f0618f25c1d25d19f4e1e89de1e6

另外值得看看我们在单元测试https://github.com/spring-projects/spring-hadoop/tree/master/spring-hadoop-store/src/test/java/org/springframework/data/hadoop/store

中做了些什么

使用spring数据写入HDFS时出现问题hadoop

1 个答案: