Question

我在使用spring集成配置的spring批处理应用程序中将多行日志消息作为单个消息读取时遇到问题，此应用程序必须将多行日志消息（示例异常堆栈跟踪）作为单个消息读取，稍后它必须处理和分类消息以进一步索引。每一行都由其时间戳（上面提到的模式，即DATE_PATTERN）标识，并且它可以继续多行，我试图继续读取消息，直到我通过覆盖看到另一个时间戳当第二行到达preProcess方法时，来自SimpleRecordSeparatorPolicy的isEndOfRecord方法我为isEndOfRecord返回true但是这没有按预期工作，任何人都可以通过识别时间戳模式帮助我读取提到的日志文件吗？

我使用org.springframework.batch.item.file.FlatFileItemReader和org.springframework.batch.item.file.mapping.PassThroughLineMapper作为映射器。

请参阅完整信息，

1）日志消息文件：sample-message-test.log

2013-10-19 07:05:32.253 [My First Class..] LOG LEVEl  first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 
first-message-line-2 first-message-line-2 first-message-line-2 
first-message-line-3 first-message-line-3 first-message-line-3 
first-message-line-4 first-message-line-4 first-message-line-4 
first-message-line-5 first-message-line-5 
first-message-line-6 
2013-10-19 07:05:32.257 [My Second Class..] LOG LEVEl  second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 
second-message-line-2 second-message-line-2 second-message-line-2 
second-message-line-3 second-message-line-3 second-message-line-3 
second-message-line-4 second-message-line-4 second-message-line-4 
second-message-line-5 second-message-line-5 
second-message-line-6
2013-10-19 07:05:32.259 [My Third Class..] LOG LEVEl  third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 
third-message-line-2 third-message-line-2 third-message-line-2 
third-message-line-3 third-message-line-3 third-message-line-3 
third-message-line-4 third-message-line-4 third-message-line-4 
third-message-line-5 third-message-line-5 
third-message-line-6

2）批量配置文件

<batch:job id="fileReadingJob">
        <batch:step id="flatFileReadingStep">
            <batch:tasklet >
                <batch:chunk reader="reader" writer="writer" commit-interval="10" />
            </batch:tasklet>
        </batch:step>
    </batch:job>

    <bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader"  scope="step">
        <property name="lineMapper">
            <bean class="org.springframework.batch.item.file.mapping.PassThroughLineMapper"/>
        </property>
        <property name="bufferedReaderFactory">
            <bean class="org.springframework.batch.item.file.DefaultBufferedReaderFactory"/>
        </property>
        <property name="recordSeparatorPolicy" >
            <bean class="com.batchlog.explorer.batchio.FlatFileRecordSeperationPolicy"/>
        </property>
        <property name="resource" value="file:///#{systemProperties['logfolder']}/#{jobParameters['inputfile']}" />
    </bean>
    <bean id="writer" class="com.batchlog.explorer.batchio.FlatFileWriter" scope="step"/>
........

3）

public class FlatFileRecordSeperationPolicy extends SimpleRecordSeparatorPolicy {

    public static final String STARTING_OF_THE_LINE = "-STARTING_OF_THE_LINE-";
    public static final String CONTINUATION_OF_THE_FILE  = "-CONTINUATION_OF_THE_FILE-";
    public static final String END_OF_THE_LINE = "-END_OF_THE_LINE-";
    public static final String END_OF_THE_LINE_CHARACER = " \n ";
    public static final String DATE_PATTERN ="^(?>\\d\\d){1,2}-(?:0?[1-9]|1[0-2])-(\\s)?(?:2[0123]|[01][0-9]):? (?:[0-5][0-9])(?::?(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?))?(?:Z|[+-](?:2[0123]|[01][0-9])(?::?(?:[0-5][0-9])))?.*?";


    @Override
        public boolean isEndOfRecord(String line) {
            if(line.matches(DATE_PATTERN) || line.startsWith(STARTING_OF_THE_LINE)
                            || line.contains(CONTINUATION_OF_THE_FILE) || line.startsWith(END_OF_THE_LINE)){
                if(isNextLineStarts(line) || line.startsWith(END_OF_THE_LINE)){
                    return true;//to break line
                }
        }
        return false; //to conitnue line

    private boolean isNextLineStarts(String preProcessOfLine){
            if(preProcessOfLine.contains(CONTINUATION_OF_THE_FILE) && !preProcessOfLine.endsWith(CONTINUATION_OF_THE_FILE)){
                String[] lines = preProcessOfLine.split(CONTINUATION_OF_THE_FILE);
                if(lines[1].trim().matches(DATE_PATTERN)){
                    return true;
                }
            }
            return false;
    }
    @Override
        public String preProcess(String line) {
            if(line.matches(DATE_PATTERN) && !line.contains(CONTINUATION_OF_THE_FILE)){
                line = new StringBuilder(STARTING_OF_THE_LINE).append(line).toString();
            }else if(line.startsWith(STARTING_OF_THE_LINE) && !line.contains(CONTINUATION_OF_THE_FILE)){
                line =  new StringBuilder(line.substring(STARTING_OF_THE_LINE.length())).append(CONTINUATION_OF_THE_FILE).toString();
            }else if(line.contains(CONTINUATION_OF_THE_FILE) && !line.endsWith(CONTINUATION_OF_THE_FILE)){
                String[] lines = line.split(CONTINUATION_OF_THE_FILE);
                if(lines[1].trim().matches(DATE_PATTERN)){
                    line = new StringBuilder(END_OF_THE_LINE).append(lines[0]).toString();//.append(lines[1]).toString();
                }else{
                    line = new StringBuilder(lines[0]).append(lines[1]).append(CONTINUATION_OF_THE_FILE).toString();
                }
            }
                return super.preProcess(line);
    }
    @Override
        public String postProcess(String record) {
            if(record.startsWith(END_OF_THE_LINE)){
                record = new StringBuilder(record.substring(END_OF_THE_LINE.length())).toString();
            }else if(record.contains(CONTINUATION_OF_THE_FILE) && !record.endsWith(CONTINUATION_OF_THE_FILE)){
                String[] lines = record.split(CONTINUATION_OF_THE_FILE);
                if(lines[1].trim().matches(DATE_PATTERN)){
                    record = new StringBuilder(END_OF_THE_LINE).append(lines[0]).toString();
                }else{
                    record = new StringBuilder(lines[0]).append(lines[1]).toString();
                }
            }
            return super.postProcess(record);
    }

Answer 1

按multiorder-line example中所述或this post中所述编写您自己的ItemReader。

Answer 2

你的问题没有涉及RecordSeparatorPolicy.isEndOfRecord（String）范例。当排列的结尾放在最后一行时，isEndOfRecored可以很好地工作例如，在DefaultRecordSeparatorPolicy中，它确保您具有偶数引号。最后一个报价包含在所需记录中。在你的情况下，你会过度阅读一行。

使用postProcess和preProcess的基本思路可能有效，但是当你到达EOL并且readline返回null时，你仍然从最后一行的FlatFileItemReader得到FlatFileParseException，请参阅FlatFileItemReader中的applyRecordSeparatorPolicy（String line）。

  private String applyRecordSeparatorPolicy(String line) throws IOException {

        String record = line;
        while (line != null && !recordSeparatorPolicy.isEndOfRecord(record)) {
            line = this.reader.readLine();
            if (line == null) {

                if (StringUtils.hasText(record)) {
                    // A record was partially complete since it hasn't ended but
                    // the line is null
                    throw new FlatFileParseException("Unexpected end of file before record complete", record, lineCount);
                }
                else {
                    // Record has no text but it might still be post processed
                    // to something (skipping preProcess since that was already
                    // done)
                    break;
                }
            }
            else {
                lineCount++;
            }
            record = recordSeparatorPolicy.preProcess(record) + line;
        }

        return recordSeparatorPolicy.postProcess(record);

    }

在这种情况下，输出文件将缺少基于commit-interval和isEndOfRecord逻辑的行。

所以基本上我建议使用不同的方法，bellabax解决方案是否适合您？

Spring Batch - 读取多行日志消息

2 个答案: