Question

我是shell脚本的新手。我有这样的日志文件。

2018-01-18T15:55:15,637 INFO  [HiveServer2-Handler-Pool: Thread-37([])]: 
thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(317)) - Client 
protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
2018-01-18T15:55:15,648 INFO  [HiveServer2-Handler-Pool: Thread-37([])]: 
session.SessionState (SessionState.java:createPath(749)) - Created HDFS

我试图像这样过滤掉所需的列。

cat hive-server2.log | grep 's3\|user\|query'

2018-01-18T16:20:39,464 WARN  [67272380-f3e9-40da-8e8e-a209c05eb4fe HiveServer2-Handler-Pool: Thread-37([])]: util.CurrentUserGroupInformation (CurrentUserGroupInformation.java:getGroupNameFromUser(52)) - user aa (auth:PROXY) via hive (auth:SIMPLE) has no primary groupName, setting groupName to be aa.
2018-01-18T16:23:25,389 INFO  [HiveServer2-Background-Pool: Thread-63([])]: ql.Driver (Driver.java:execute(1735)) - Executing command(queryId=hive_20180118162325_5ad8be3f-80e7-468d-bb47-1bdc2d2fb624):      2018-01-18T16:23:25,393 INFO  [HiveServer2-Background-Pool: Thread-63([])]: ql.Driver (Driver.java:execute(2050)) - Completed executing command(queryId=hive_20180118162325_5ad8be3f-80e7-468d-bb47-1bdc2d2fb624); Time taken: 0.004 seconds
 2018-01-18T16:23:25,972 INFO  [67272380-f3e9-40da-8e8e-a209c05eb4fe HiveServer2-Handler-Pool: Thread-49([])]: s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(1210)) - Opening 's3://f4340808-220a-424c-ba67-3f2383ea42ea-c000.csv' for reading

以上打印所有已过滤的关键字。现在我需要像这样保存。

IN .txt文件

column names - TimeStamp,User,Query,file path
2018-01-18T16:23:25,972,select * from bv limit 5,Opening 
's3://aaaa' for reading

我不知道如何在上面的输出中提取此列值。任何帮助将不胜感激。

Answer 1

Awk 静态日志格式的解决方案：

awk 'BEGIN{ print "TimeStamp,User,Query,S3 bucket path" }
     /\<user\>/{ u=$10 }
     /Executing command\(queryId/{ sub(/.*queryId=[^[:space:]]+: /,""); q=$0 }
     /s3:\/\//{ print $1,u,q,$10 }' OFS=',' hive-server2.log

输出：

TimeStamp,User,Query,S3 bucket path
2018-01-18T16:23:25,972,a8197zz,select * from pfeevent limit 5,'s3://3m-his-dev-cayuga/Demo-Enterprise/TrustedZone/Published/Enrichment/Core/PFE/PFEEvent/part-00000-f4340808-220a-424c-ba67-3f2383ea42ea-c000.snappy.parquet'

提取登录shell脚本

1 个答案: