Spring XD Distibuted Runtime -JDBCHDFS JOB Getting＆＃39;无法初始化阅读器＆＃39;

时间：2015-10-21 21:20:32

标签： java spring spring-mvc hadoop spring-xd

我们正尝试使用spring-xd dirt环境中的批处理作业将外部数据源表（sql server 2012）拉入hadoop系统。

运行以下作业后： 

job create jobname --definition "jdbchdfs --driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver --url='jdbc:sqlserver://hostname:port;database=databasename' --username=user --password=password --testOnBorrow=false --delimiter=; --sql='select t1.* from (select ROW_NUMBER() OVER (ORDER BY GroupBy4KPI,Name  DESC) AS ID, GETDATE() as time,Name,budget,CASE WHEN bmon='BudMnt1' THEN CONVERT(DATE, '01-JAN-2015') WHEN bmon='BudMnt2' THEN CONVERT(DATE, '01-FEB-2015') WHEN bmon='BudMnt3' THEN CONVERT(DATE, '01-MAR-2015') WHEN bmon='BudMnt4' THEN CONVERT(DATE, '01-APR-2015') WHEN bmon='BudMnt5' THEN CONVERT(DATE, '01-MAY-2015') WHEN bmon='BudMnt6' THEN CONVERT(DATE, '01-JUN-2015') WHEN bmon='BudMnt7' THEN CONVERT(DATE, '01-JUL-2015') WHEN bmon='BudMnt8' THEN CONVERT(DATE, '01-AUG-2015') WHEN bmon='BudMnt9' THEN CONVERT(DATE, '01-SEP-2015') WHEN bmon='BudMnt10' THEN CONVERT(DATE, '01-OCT-2015') WHEN bmon='BudMnt11' THEN CONVERT(DATE, '01-NOV-2015') WHEN bmon='BudMnt12' THEN CONVERT(DATE, '01-DEC-2015')END AS bmon from tablename UNPIVOT (budget FOR bmon IN(BudMnt1,BudMnt2,BudMnt3,BudMnt4,BudMnt5,BudMnt6,BudMnt7,BudMnt8,BudMnt9,BudMnt10,BudMnt11,BudMnt12)) as p) t1' --fsUri=hdfs://hostname-HA --fileExtension=csv --directory=/dir/to/hdfs --fileName=xyz --partitions=1 --makeUnique=true"

在失败的工作状态后尝试跟进：

我们正在运行spring-xd的分布式模式。 所以，尝试使用以下命令部署不同的容器

job deploy jobname --properties "module.jdbchdfs.criteria=host.equals('host1.acme.com')"

job deploy jobname --properties "module.jdbchdfs.criteria=host.equals('host2.acme.com')"

获取＆＃39;退出说明＆＃39; 

org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
    at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:147)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy471.open(Unknown Source)
    at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:96)
    at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:310)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
    at org.springframework.batch.integration.partition.StepExecutionRequestHandler.handle(StepExecutionRequestHandler.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:112)
    at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:129)
    at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
    at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:342)
    at org.springframework.expression.spel.ast.CompoundExpressio

获取更多错误 

org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
    at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:198)
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:165)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128)
    at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy52.run(Unknown Source)
    at org.springframework.batch.integration.launch.JobLaunchingMessageHandler.launch(JobLaunchingMessageHandler.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.refl
用于调试的

容器日志是： 

2015-10-21T16:40:59-0400 1.2.1.RELEASE INFO CuratorFramework-0-EventThread container.DeploymentListener - Undeploying module [ModuleDescriptor@45e9256d moduleName = 'jdbchdfs', moduleLabel = 'jdbchdfs', group = 'jobname', sourceChannelName = [null], sinkChannelName = [null], index = 0, type = job, parameters = map['fileExtension' -> 'csv', 'partitions' -> '1', 'testOnBorrow' -> 'false', 'driverClassName' -> 'com.microsoft.sqlserver.jdbc.SQLServerDriver', 'password' -> 'password', 'url' -> 'jdbc:sqlserver://hostname:port;database=databasename', 'delimiter' -> ';', 'username' -> 'user', 'sql' -> ''select t1.* from (select ROW_NUMBER() OVER (ORDER BY GroupBy4KPI,Name DESC) AS ID, GETDATE() as time,Name,budget,CASE WHEN bmon='BudMnt1' THEN CONVERT(DATE, '01-JAN-2015') WHEN bmon='BudMnt2' THEN CONVERT(DATE, '01-FEB-2015') WHEN bmon='BudMnt3' THEN CONVERT(DATE, '01-MAR-2015') WHEN bmon='BudMnt4' THEN CONVERT(DATE, '01-APR-2015') WHEN bmon='BudMnt5' THEN CONVERT(DATE, '01-MAY-2015') WHEN bmon='BudMnt6' THEN CONVERT(DATE, '01-JUN-2015') WHEN bmon='BudMnt7' THEN CONVERT(DATE, '01-JUL-2015') WHEN bmon='BudMnt8' THEN CONVERT(DATE, '01-AUG-2015') WHEN bmon='BudMnt9' THEN CONVERT(DATE, '01-SEP-2015') WHEN bmon='BudMnt10' THEN CONVERT(DATE, '01-OCT-2015') WHEN bmon='BudMnt11' THEN CONVERT(DATE, '01-NOV-2015') WHEN bmon='BudMnt12' THEN CONVERT(DATE, '01-DEC-2015')END AS bmon from dbo.tablename UNPIVOT (budget FOR bmon IN(BudMnt1,BudMnt2,BudMnt3,BudMnt4,BudMnt5,BudMnt6,BudMnt7,BudMnt8,BudMnt9,BudMnt10,BudMnt11,BudMnt12)) as p) t1'', 'directory' -> '/path/to/hdfs/', 'fsUri' -> 'hdfs://HOSTNAME-HA', 'fileName' -> 'rev', 'makeUnique' -> 'true'], children = list[[empty]]]
2015-10-21T16:40:59-0400 1.2.1.RELEASE INFO DeploymentsPathChildrenCache-0 container.DeploymentListener - Path cache event: path=/deployments/modules/allocated/ee07615a-4051-477d-962f-400582136d23/trev_bycorpclass_stg0.job.jdbchdfs.1, type=CHILD_REMOVED
2015-10-21T16:47:30-0400 1.2.1.RELEASE INFO DeploymentsPathChildrenCache-0 container.DeploymentListener - Path cache event: path=/deployments/modules/allocated/ee07615a-4051-477d-962f-400582136d23/kpi_stg0.job.jdbchdfs.1, type=CHILD_ADDED
2015-10-21T16:47:31-0400 1.2.1.RELEASE WARN DeploymentsPathChildrenCache-0 config.ReleaseStrategyFactoryBean - No annotated method found; falling back to SequenceSizeReleaseStrategy, target:org.springframework.batch.integration.partition.MessageChannelPartitionHandler@311776a3, methodName:null
2015-10-21T17:49:10-0400 1.2.1.RELEASE INFO DeploymentsPathChildrenCache-0 container.DeploymentListener - Path cache event: path=/deployments/modules/allocated/ee07615a-4051-477d-962f-400582136d23/jobname.job.jdbchdfs.1, type=CHILD_ADDED
2015-10-21T17:49:10-0400 1.2.1.RELEASE INFO DeploymentsPathChildrenCache-0 container.DeploymentListener - Deploying job 'jobname'
2015-10-21T17:49:10-0400 1.2.1.RELEASE INFO DeploymentsPathChildrenCache-0 container.DeploymentListener - Deploying module [ModuleDescriptor@5ef4ee27 moduleName = 'jdbchdfs', moduleLabel = 'jdbchdfs', group = 'jobname', sourceChannelName = [null], sinkChannelName = [null], index = 0, type = job, parameters = map['fileExtension' -> 'csv', 'partitions' -> '1', 'testOnBorrow' -> 'false', 'driverClassName' -> 'com.microsoft.sqlserver.jdbc.SQLServerDriver', 'password' -> 'password', 'url' -> 'jdbc:sqlserver://hostname:port;database=databasename', 'delimiter' -> ';', 'username' -> 'user', 'sql' -> 'select ROW_NUMBER() OVER (ORDER BY GroupBy4KPI,Name DESC) AS ID, GETDATE() as time from tablename', 'directory' -> '/path/to/hdfs/', 'fsUri' -> 'hdfs://HOSTNAME-HA', 'fileName' -> 'rev', 'makeUnique' -> 'true'], children = list[[empty]]]
2015-10-21T17:49:11-0400 1.2.1.RELEASE WARN DeploymentsPathChildrenCache-0 config.ReleaseStrategyFactoryBean - No annotated method found; falling back to SequenceSizeReleaseStrategy, target:org.springframework.batch.integration.partition.MessageChannelPartitionHandler@6cc0c6f2, methodName:null

问题出现了：

当我们在job defintition中提供sql选项时 由于select * from tablename unpivot () as up spring-xd无法初始化读者， 但与此同时，如果我们将供应 select * from tablename，成功执行了工作* 

**Any help would be much appreciated ?**

0 个答案:

没有答案
相关问题
最新问题