Question

我有一个这样的记录器文件：

2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.commerce.common.utils.APIUtils - Enrichment data updated successful for partnumber : 13794017 with status : 201
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.commerce.common.utils.APIUtils - Enrichment data updated successful for partnumber : 13794017 with status : 201
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.commerce.common.utils.APIUtils - Enrichment data updated successful for partnumber : 13794017 with status : 201
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute() 13794017 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=1.808 sec
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute() 13794017 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=1.808 sec
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute() 13794017 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=1.808 sec
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute()  Scene7 update for 13794017 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute()  Scene7 update for 13794017 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute()  Scene7 update for 13794017 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=
2016-06-11 07:34:01.543 98460 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute EXIT
2016-06-11 07:34:01.543 98460 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute EXIT
2016-06-11 07:34:01.543 98460 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute EXIT
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.commerce.common.utils.APIUtils - Enrichment data updated successful for partnumber : 17696532 with status : 500
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.commerce.common.utils.APIUtils - Enrichment data updated successful for partnumber : 17696532 with status : 500
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.commerce.common.utils.APIUtils - Enrichment data updated successful for partnumber : 17696532 with status : 500
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute() 17696532 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=1.808 sec
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute() 17696532 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=1.808 sec
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute() 17696532 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=1.808 sec
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute()  Scene7 update for 17696532 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute()  Scene7 update for 17696532 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=
2016-06-11 07:34:01.542 98459 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute()  Scene7 update for 17696532 itemStatus itemsProcessed=1, itemsUpdated=1, timeTaken=
2016-06-11 07:34:01.543 98460 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute EXIT
2016-06-11 07:34:01.543 98460 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute EXIT
2016-06-11 07:34:01.543 98460 [Thread-23-Job-parser-bolt] INFO  JobLoader c.t.c.w.b.JobParserBolt - execute EXIT

有数百个此类日志重复出现，其中包含不同的部分号和状态码。我想将状态代码不是201的不同partnumbers存储到一个单独的文件中，以便我们可以轻松地监视它。虽然我想计算所有201个成功职位。因此，我想要的示例输出应该如下所示：

No. of partnumbers with Status 201: 1
Partnumbers with Status 500: 17696532, ... , ...
Partnumbers with Status 401: ... ,...

我首先使用awk，但是解析并不那么容易。另请注意，同一部件号出现多次，如何添加支票，以便我不会多次计算单个部件号。

我的代码到现在为止：

awk -F'Enrichment data updated successful for partnumber :' '{print $2}' file.log |rev | cut -c 4- | rev

我想首先像这样提取partnumber，但是我无法应用检查来避免多个partnumber问题并将它的相应状态代码与它相关联。

Answer 1

这是使用awk解决的问题。请参阅内联注释以获得解释。

awk '/Enrichment data updated successful for partnumber/ {
    # store the results as a multidimensional array with the first key
    # being the status and the key of the second array being the product
    # number. This removes duplicates because array keys must be unique
    arr[$NF][$16]++
}
END {
    # iterate over the 201 status items and count them
    for (item in arr[201]) {
        count++
    }
    print "No. of partnumbers with Status 201: " count

    # iterate over the status array
    for (status in arr) {
        # skip 201 status
        if (status == 201)
            continue
        # join the array by "," for printing
        # taken from http://stackoverflow.com/a/13648609/1032785
        joined = sep = ""
        for (product in arr[status]) {
            joined = joined sep product
            sep = ","
        }

        print "Partnumbers with Status " status ": " joined
    }
}
' foo.log

这会在您的示例日志文件中生成以下输出，我在其中添加了一些其他行：

No. of partnumbers with Status 201: 1
Partnumbers with Status 401: 17623039
Partnumbers with Status 500: 17696532, 17696539

Answer 2

非 awk ，使用 datamash 和 pee ：

echo -n "No. of partnumbers with Status 201: " ; \
grep "status : " file.log | pee \
    'grep    ": 201" | datamash -W -s countunique 16'  \
    'grep -v ": 201" | datamash -W -s -g20 unique 16 | \
        sed "s/^[0-9]*/Partnumbers with Status &:/;s/,/, /g"'

输出，（使用OP的样本数据）：

No. of partnumbers with Status 201: 1
Partnumbers with Status 500:    17696532

读取日志文件以提取各种字段并计算出现的次数

2 个答案: