Question

为什么itemReader方法始终发送要在CustomItemProcessor中处理的完全相同的文件名？

据我所知，由于我将读者安排为@Scope并且我设置了超过1块，我期待＆＃34;返回s＆＃34;从String数组向前移动到下一个值。

让我用读者方法中的调试示例澄清我的问题：

1 - 变量stringArray用3个文件名填充（f1.txt，f2.txt和f3.txt）

2 - ＆＃34;返回s＆＃34;是用s = f1.txt

引起的

3 - ＆＃34;返回s＆＃34;在引发customItemProcessor方法之前再次唤起（完全直到这里，因为chunk = 2）

4 - 看着它再次包含f1.txt（与我的预期不同。我期望f2.txt）

5和6 - 运行具有相同名称f1.tx的处理器（如果＆＃34的第二个回合，它应该正常工作;返回s＆＃34;将包含f2.txt）

7 - writer方法按预期工作（processedFiles包含两次在customItemProcessor f1.txt和f1.txt中处理的两个名称，因为同一名称被处理了两次）

CustomItemReader

public class CustomItemReader implements ItemReader<String> {
       @Override
       public String read() throws Exception, UnexpectedInputException,

                     ParseException, NonTransientResourceException {
              String[] stringArray;

              try (Stream<Path> stream = Files.list(Paths.get(env

                           .getProperty("my.path")))) {

                     stringArray = stream.map(String::valueOf)

                                  .filter(path -> path.endsWith("out"))

                                  .toArray(size -> new String[size]);

              }



              //*** the problem is here

              //every turn s variable receives the first file name from the stringArray

              if (stringArray.length > 0) {

                     for (String s : stringArray) {

                           return s;



                     }

              } else {

                     log.info("read method - no file found");

                     return null;
              }
              return null;
       }

CustomItemProcessor

public class CustomItemProcessor implements ItemProcessor<String , String> {

       @Override

       public String process(String singleFileToProcess) throws Exception {

              log.info("process method: " + singleFileToProcess);

              return singleFileToProcess;

       }

}

CustomItemWriter

public class CustomItemWriter implements ItemWriter<String> {



       private static final Logger log = LoggerFactory

                     .getLogger(CustomItemWriter.class);



       @Override

       public void write(List<? extends String> processedFiles) throws Exception {



              processedFiles.stream().forEach(

                           processedFile -> log.info("**** write method"

                                         + processedFile.toString()));



              FileSystem fs = FileSystems.getDefault();



              for (String s : processedFiles) {

                     Files.deleteIfExists(fs.getPath(s));

              }

       }

配置

@Configuration

@ComponentScan(...

@EnableBatchProcessing

@EnableScheduling

@PropertySource(...

public class BatchConfig {





       @Autowired

       private JobBuilderFactory jobBuilderFactory;



       @Autowired

       private StepBuilderFactory stepBuilderFactory;



       @Autowired

       private JobRepository jobRepository;



    @Bean

    public TaskExecutor getTaskExecutor() {

        return new TaskExecutor() {

            @Override

            public void execute(Runnable task) {



            }

        };

    }



       //I can see the number in chunk reflects how many time customReader is triggered before triggers customProcesser     

       @Bean

       public Step step1(ItemReader<String> reader,

                     ItemProcessor<String, String> processor, ItemWriter<String> writer) {

              return stepBuilderFactory.get("step1").<String, String> chunk(2)

                           .reader(reader).processor(processor).writer(writer)

                           .allowStartIfComplete(true).build();



       }



       @Bean

       @Scope

       public ItemReader<String> reader() {

              return new CustomItemReader();



       }



       @Bean

       public ItemProcessor<String, String> processor() {

              return new CustomItemProcessor();



       }



       @Bean

       public ItemWriter<String> writer() {

              return new CustomItemWriter();



       }



       @Bean

       public Job job(Step step1) throws Exception {

              return jobBuilderFactory.get("job1").incrementer(new RunIdIncrementer()).start(step1).build();

       }

调度

@Component

public class QueueScheduler {



       private static final Logger log = LoggerFactory

                     .getLogger(QueueScheduler.class);



    private Job job;

    private JobLauncher jobLauncher;



    @Autowired

    public QueueScheduler(JobLauncher jobLauncher, @Qualifier("job") Job job){

        this.job = job;

        this.jobLauncher = jobLauncher;

   }

   @Scheduled(fixedRate=60000)

   public void runJob(){

          try{
       jobLauncher.run(job, new JobParameters());

          }catch(Exception ex){

                 log.info(ex.getMessage());

          }

   }

}

Answer 1

您的问题是，您依靠内部循环来迭代这些项目，而不是让Spring Batch通过多次调用ItemReader#read为您执行此操作。

我建议将读者更改为以下内容：

public class JimsItemReader implements ItemStreamReader {
   private String[] items;
   private int curIndex = -1;

   @Override
   public void open(ExecutionContext ec) {
       curIndex = ec.getInt("curIndex", -1);
       String[] stringArray;

       try (Stream<Path> stream = Files.list(Paths.get(env.getProperty("my.path")))) {
           stringArray = stream.map(String::valueOf)
                               .filter(path -> path.endsWith("out"))
                               .toArray(size -> new String[size]);

        }
   }

   @Override
   public void update(ExecutionContext ec) {
       ec.putInt("curIndex", curIndex);
   }

   @Override
   public String read() {
       if (curIndex < items.length) {
           curIndex++;
           return items[curIndex];
       } else {
           return null;
       }
   }
}

上面的示例应该在读取数组时遍历数组中的项目。它也应该是可重新启动的，因为我们将索引存储在ExecutionContext中，所以如果在失败后重新启动作业，您将从中断的地方重新启动。

为什么intemReader总是向CustomItemProcessor发送完全相同的值

1 个答案: