使用时间戳本身作为拆分器拆分已分组的带时间戳的日志数据?

时间:2016-08-02 02:06:26

标签: java

我想编写可以读取日志文件并将其拆分为多个事件的代码,使用时间戳作为拆分器(因为每个日志条目都以时间戳开头)。我想分割的日志样本如下。我也想保留时间戳。

所以如果这是我的输入:

01 Aug 2016 04:48:13,311 ERROR [pool-2-thread-12436] com.orders.queue.OrdersQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Exception while calling /purchases: x-company-status : UNKNOWN_ERROR response status: Internal Server Error 
01 Aug 2016 04:48:13,311  WARN [pool-2-thread-12436] com.orders.queue.OrdersQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Failed to process order, will be re-tried: ADD2500051FR
01 Aug 2016 04:48:13,332  INFO [pool-2-thread-12436] com.delegate - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Getting Email from Primary email
01 Aug 2016 04:48:13,363  WARN [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Message processing failed QueueMessage [payload=ADD2500051FR, delaySeconds=0, sqsId=51f70e3f-554a-463b-8384-0b2c25a90450, stringAttributes={features=adac2911-0578-4bcd-b8c3-783481a48e1d, accept-language=FR_FR, request-id=836ac8b6-515d-4414-b4c6-ddd8a52ef497}]
com.orders.exception.orderserviceException: Error in Calling PUT purchase from main service 
    at com.OrderServiceDelegate.handleInternalServerErrors(OrderServiceDelegate.java:352)
    at com.OrderServiceDelegate.sendOrderForProcessing_aroundBody0(OrderServiceDelegate.java:113)
    at com.OrderServiceDelegate.sendOrderForProcessing_aroundBody1$advice(OrderServiceDelegate.java:37)
    at com.OrderServiceDelegate.sendOrderForProcessing(OrderServiceDelegate.java:1)
    at com.orders.queue.OrdersQueueWorker.doWork(OrdersQueueWorker.java:168)
    at com.queue.SQSQueueWorker.lambda$0(SQSQueueWorker.java:149)
    at com.queue.SQSQueueWorker.dt_access$492(SQSQueueWorker.java)
    at com.queue.SQSQueueWorker$$dtt$$Lambda$8/852112146.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
01 Aug 2016 04:48:13,365  INFO [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Order will be re-tried after 300 seconds: ADD2500051FR 
01 Aug 2016 04:48:15,600  INFO [myScheduler-3] com.queue.SQSQueueWorker - x-company-requestid=sqs-worker service_name=orders_v3 Processing messages message_number=1
01 Aug 2016 04:48:15,600  INFO [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Received msg from SQS:QueueMessage [payload=428CB476547214700268914651663, delaySeconds=0, sqsId=7f4dcbbe-90c4-4e56-b4ab-50332597b5d8, stringAttributes={features=FIS-JEM, accept-language=EN_US, request-id=a2c31da4-517f-40ec-8587-624f97393659}]

然后我的输出应该是(水平线描绘一个条目结束而另一个条目开始的位置):

01 Aug 2016 04:48:13,311 ERROR [pool-2-thread-12436] com.orders.queue.OrdersQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Exception while calling /purchases: x-company-status : UNKNOWN_ERROR response status: Internal Server Error
//--------------------------------------------
01 Aug 2016 04:48:13,311  WARN [pool-2-thread-12436] com.orders.queue.OrdersQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Failed to process order, will be re-tried: ADD2500051FR
//--------------------------------------------
01 Aug 2016 04:48:13,332  INFO [pool-2-thread-12436] com.delegate - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Getting Email from Primary email
//--------------------------------------------
01 Aug 2016 04:48:13,363  WARN [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Message processing failed QueueMessage [payload=ADD2500051FR, delaySeconds=0, sqsId=51f70e3f-554a-463b-8384-0b2c25a90450, stringAttributes={features=adac2911-0578-4bcd-b8c3-783481a48e1d, accept-language=FR_FR, request-id=836ac8b6-515d-4414-b4c6-ddd8a52ef497}]
com.orders.exception.orderserviceException: Error in Calling PUT purchase from main service 
    at com.OrderServiceDelegate.handleInternalServerErrors(OrderServiceDelegate.java:352)
    at com.OrderServiceDelegate.sendOrderForProcessing_aroundBody0(OrderServiceDelegate.java:113)
    at com.OrderServiceDelegate.sendOrderForProcessing_aroundBody1$advice(OrderServiceDelegate.java:37)
    at com.OrderServiceDelegate.sendOrderForProcessing(OrderServiceDelegate.java:1)
    at com.orders.queue.OrdersQueueWorker.doWork(OrdersQueueWorker.java:168)
    at com.queue.SQSQueueWorker.lambda$0(SQSQueueWorker.java:149)
    at com.queue.SQSQueueWorker.dt_access$492(SQSQueueWorker.java)
    at com.queue.SQSQueueWorker$$dtt$$Lambda$8/852112146.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
//--------------------------------------------
01 Aug 2016 04:48:13,365  INFO [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Order will be re-tried after 300 seconds: ADD2500051FR
//--------------------------------------------
01 Aug 2016 04:48:15,600  INFO [myScheduler-3] com.queue.SQSQueueWorker - x-company-requestid=sqs-worker service_name=orders_v3 Processing messages message_number=1
//--------------------------------------------
01 Aug 2016 04:48:15,600  INFO [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Received msg from SQS:QueueMessage [payload=428CB476547214700268914651663, delaySeconds=0, sqsId=7f4dcbbe-90c4-4e56-b4ab-50332597b5d8, stringAttributes={features=FIS-JEM, accept-language=EN_US, request-id=a2c31da4-517f-40ec-8587-624f97393659}]

我想我需要某种正则表达式,但我没有使用正则表达式通过Java代码进行这种拆分的经验。

我还发现了以下相关问题,但我并不了解那里建议的解决方案: java regex: capture multiline sequence between tokens

1 个答案:

答案 0 :(得分:0)

以下代码使用您提供的日志并生成所需的输出。我在代码中包含了解释程序流程的注释。

<强>代码:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.regex.Pattern;


public class Driver {
    private static final String LINE_BREAK = System.lineSeparator();

    // path to the logfile
    private static final String LOG_FILE = "xin/xin.log";

    // The regex below matches lines that begin with "00 Mth 0000 00:00:00,000".
    private static Pattern pattern = Pattern.compile("^[0-9][0-9]\\s[A-Z][a-z][a-z]\\s[0-9][0-9][0-9][0-9]\\s[0-9][0-9]:[0-9][0-9]:[0-9][0-9],[0-9][0-9][0-9]");

    public static void main(String[] args) {
        BufferedReader reader = null;
        StringBuilder sb = new StringBuilder();
        String line = "";
        boolean firstRun = true;

        try {
            // instantiated the buffered reader
            reader = new BufferedReader(new FileReader(new File(LOG_FILE)));
        } catch(FileNotFoundException e) {
            e.printStackTrace();
        }

        // loop until all lines are read from log file 
        while(true) {
            try {
                // get next line from log file
                line = reader.readLine();
                // if no more content in log file then break out of loop
                if(line == null) {
                    break;
                }
            } catch (IOException e) {
                e.printStackTrace();
            }

            // we don't want a line-break before the first line
            if(firstRun) {
                firstRun = false;
                // append the first line to the string builder
                sb.append(line);
            } else {
                // we've handled the first line, so now we
                // append a line-break to the string builder
                // before appending the next line
                sb.append(LINE_BREAK);

                // if the line matches the timestamp pattern then
                // append another line-break
                if(pattern.matcher(line).find()) {
                    sb.append(LINE_BREAK);
                }
                // add the line to the string builder
                sb.append(line);
            }

        }

        try {
            // close the buffered reader
            reader.close();
        } catch (IOException e) {
            e.printStackTrace();
        }

        // print string builder contents to standard out
        System.out.println(sb.toString());
    }

}

<强>输出:

01 Aug 2016 04:48:13,311 ERROR [pool-2-thread-12436] com.orders.queue.OrdersQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Exception while calling /purchases: x-company-status : UNKNOWN_ERROR response status: Internal Server Error 

01 Aug 2016 04:48:13,311  WARN [pool-2-thread-12436] com.orders.queue.OrdersQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Failed to process order, will be re-tried: ADD2500051FR

01 Aug 2016 04:48:13,332  INFO [pool-2-thread-12436] com.delegate - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Getting Email from Primary email

01 Aug 2016 04:48:13,363  WARN [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Message processing failed QueueMessage [payload=ADD2500051FR, delaySeconds=0, sqsId=51f70e3f-554a-463b-8384-0b2c25a90450, stringAttributes={features=adac2911-0578-4bcd-b8c3-783481a48e1d, accept-language=FR_FR, request-id=836ac8b6-515d-4414-b4c6-ddd8a52ef497}]
com.orders.exception.orderserviceException: Error in Calling PUT purchase from main service 
    at com.OrderServiceDelegate.handleInternalServerErrors(OrderServiceDelegate.java:352)
    at com.OrderServiceDelegate.sendOrderForProcessing_aroundBody0(OrderServiceDelegate.java:113)
    at com.OrderServiceDelegate.sendOrderForProcessing_aroundBody1$advice(OrderServiceDelegate.java:37)
    at com.OrderServiceDelegate.sendOrderForProcessing(OrderServiceDelegate.java:1)
    at com.orders.queue.OrdersQueueWorker.doWork(OrdersQueueWorker.java:168)
    at com.queue.SQSQueueWorker.lambda$0(SQSQueueWorker.java:149)
    at com.queue.SQSQueueWorker.dt_access$492(SQSQueueWorker.java)
    at com.queue.SQSQueueWorker$$dtt$$Lambda$8/852112146.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

01 Aug 2016 04:48:13,365  INFO [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Order will be re-tried after 300 seconds: ADD2500051FR 

01 Aug 2016 04:48:15,600  INFO [myScheduler-3] com.queue.SQSQueueWorker - x-company-requestid=sqs-worker service_name=orders_v3 Processing messages message_number=1

01 Aug 2016 04:48:15,600  INFO [pool-2-thread-12436] com.queue.SQSQueueWorker - x-company-requestid=836ac8b6-515d-4414-b4c6-ddd8a52ef497-sqs service_name=orders_v3 Received msg from SQS:QueueMessage [payload=428CB476547214700268914651663, delaySeconds=0, sqsId=7f4dcbbe-90c4-4e56-b4ab-50332597b5d8, stringAttributes={features=FIS-JEM, accept-language=EN_US, request-id=a2c31da4-517f-40ec-8587-624f97393659}]

如果您需要任何其他帮助,只需对此答案发表评论,我们会尽力提供帮助。