Question

我们尝试在当前环境中执行jdbchdfs作业

我们的环境是：
spring-xd版本：1.2.1
hadoop分布：PHD3.0

job create rcd_prod --definition "jdbchdfs --driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver --url='jdbc:sqlserver://hostname:49552;database=databasename' --username=User --password=xxxxxx --testOnBorrow=false --delimiter=^ --sql='select * from tablename' --fsUri=hdfs://HOSTNAME-HA --fileExtension=csv --directory=/PATH/TO/DIR/ --fileName=XYZ --partitions=1 --makeUnique=true"

执行jdbchdfs作业时出错

以下错误

java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2005)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1136)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1132)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1132)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1423)
    at org.springframework.xd.batch.item.hadoop.AbstractHdfsItemWriter.initializeCounterIfNecessary(AbstractHdfsItemWriter.java:66)
    at org.springframework.xd.batch.item.hadoop.HdfsTextItemWriter.write(HdfsTextItemWriter.java:60)
    at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy94.write(Unknown Source)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:175)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:151)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:274)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:199)
    at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletSte

导致此异常的原因是什么？我们如何解决此问题？

Answer 1

当您运行多个jdbchdfs作业时，通常会遇到此问题。问题是核心Hadoop库关闭了整个JVM的连接 - 因此任何并行运行并打开HDFS文件的作业都可能会关闭彼此的文件系统连接。

这已从1.3.1.RC修订。

http://github.com/spring-projects/spring-xd/pull/1805

JDBCHDFS作业因Filesystem关闭异常而失败

1 个答案: