awk中分隔符之间的换行符

时间：2017-12-11 07:17:56

标签： regex bash awk sed

我正在解析一个包含nginx GET请求体的字符串的文件。有时它在同一请求的两个部分之间包含换行符，因此我无法使用awk解析此类请求。

我有awk -F'delimeter1: |delimiter2'的两个分隔符，也许我可以告诉awk这些分隔符之间可以有换行符，所以它会将这两行作为一个处理？

提前致谢。

示例输入（Java错误是随机示例）：

[2017-12-04 20:53:07] [ERROR] [ID-XX] Get: sr=342x487&c64=(not set)&c1=Phones, MP3s, GPS&v=1&c33=427&d28=
Like
&je=0&s4d=4-b&c32=(not set)&ua=Opera/9.80 (Android; Opera Mini/32.0.2254/77.161; U; uk) Presto/2.12.423 Version/12.16&time=04/Dec/2017:20:52:02 +0200&qtype=get
com.test.app. java.lang.StringIndexOutOfBoundsException: String index out of range: -1
          at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75)
          at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93)
          at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)
          at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350)
          at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302)
          at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113)
          at javax.swing.JComponent.getFontMetrics(JComponent.java:1626)
          at javax.swing.text.WrappedPlainView.updateMetrics(WrappedPlainView.java:318)
          at javax.swing.text.WrappedPlainView.updateChildren(WrappedPlainView.java:297)
          at javax.swing.text.WrappedPlainView.insertUpdate(WrappedPlainView.java:463)
          at javax.swing.plaf.basic.BasicTextUI$RootView.insertUpdate(BasicTextUI.java:1610)
          at javax.swing.plaf.basic.BasicTextUI$UpdateHandler.insertUpdate(BasicTextUI.java:1869)
          at javax.swing.text.AbstractDocument.fireInsertUpdate(AbstractDocument.java:201)
          at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:748)
          at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:707)
          at javax.swing.text.PlainDocument.insertString(PlainDocument.java:130)
          at javax.swing.text.DefaultEditorKit.read(DefaultEditorKit.java:273)
          at javax.swing.JEditorPane.setText(JEditorPane.java:1416)
  Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
          at java.lang.String.substring(String.java:1967) ~[?:1.8.0_151]
          ... 12 more
[2017-12-04 21:03:07] [ERROR] [ID-YY] Get: sr=342x487&c64=(not set)&c1=Phones, MP3s, GPS&v=1&em=Exception: Error: [$sc:ind] Aborting!&ua=Opera/9.80 (Android; Opera Mini/30.0.2254/77.161; U; ru) Presto/2.12.423 Version/12.16&time=04/Dec/2017:21:03:07 +0200&qtype=get
com.test.app. java.lang.StringIndexOutOfBoundsException: String index out of range: -1
          at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75)
          at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93)
          at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)
          at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350)
          at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302)
          at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113)
          at javax.swing.JComponent.getFontMetrics(JComponent.java:1626)
          at javax.swing.text.WrappedPlainView.updateMetrics(WrappedPlainView.java:318)
          at javax.swing.text.WrappedPlainView.updateChildren(WrappedPlainView.java:297)
          at javax.swing.text.WrappedPlainView.insertUpdate(WrappedPlainView.java:463)
          at javax.swing.plaf.basic.BasicTextUI$RootView.insertUpdate(BasicTextUI.java:1610)
          at javax.swing.plaf.basic.BasicTextUI$UpdateHandler.insertUpdate(BasicTextUI.java:1869)
          at javax.swing.text.AbstractDocument.fireInsertUpdate(AbstractDocument.java:201)
          at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:748)
          at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:707)
          at javax.swing.text.PlainDocument.insertString(PlainDocument.java:130)
          at javax.swing.text.DefaultEditorKit.read(DefaultEditorKit.java:273)
          at javax.swing.JEditorPane.setText(JEditorPane.java:1416)
  Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
          at java.lang.String.substring(String.java:1967) ~[?:1.8.0_151]
          ... 12 more
[2017-12-04 19:40:02] [ERROR] [ID-ZZ] Get: el=search&dl=https://market.com/?dt=Market – Electronics Store | Web Store (Market.com)&id=104777577&a=770227875&t=pageview&ua=Mozilla/5.0 (Linux; Android 7.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36&time=04/Dec/2017:19:39:04 +0200&qtype=get
com.test.app. java.lang.StringIndexOutOfBoundsException: String index out of range: -1
          at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75)
          at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93)
          at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)
          at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350)
          at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302)
          at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113)
          at javax.swing.JComponent.getFontMetrics(JComponent.java:1626)
          at javax.swing.text.WrappedPlainView.updateMetrics(WrappedPlainView.java:318)
          at javax.swing.text.WrappedPlainView.updateChildren(WrappedPlainView.java:297)
          at javax.swing.text.WrappedPlainView.insertUpdate(WrappedPlainView.java:463)
          at javax.swing.plaf.basic.BasicTextUI$RootView.insertUpdate(BasicTextUI.java:1610)
          at javax.swing.plaf.basic.BasicTextUI$UpdateHandler.insertUpdate(BasicTextUI.java:1869)
          at javax.swing.text.AbstractDocument.fireInsertUpdate(AbstractDocument.java:201)
          at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:748)
          at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:707)
          at javax.swing.text.PlainDocument.insertString(PlainDocument.java:130)
          at javax.swing.text.DefaultEditorKit.read(DefaultEditorKit.java:273)
          at javax.swing.JEditorPane.setText(JEditorPane.java:1416)
  Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
          at java.lang.String.substring(String.java:1967) ~[?:1.8.0_151]
          ... 12 more

所需的输出（一行中的打印ID和正文（在“”中）并将&替换为_&_）：

ID-XX "sr=342x487_&_c64=(not set)_&_c1=Phones, MP3s, GPS_&_v=1_&_c33=427_&_d28=Like_&_je=0_&_s4d=4-b_&_c32=(not set)_&_ua=Opera/9.80 (Android; Opera Mini/32.0.2254/77.161; U; uk) Presto/2.12.423 Version/12.16_&_time=04/Dec/2017:20:52:02 +0200_&_qtype=get"
ID-YY "sr=342x487_&_c64=(not set)_&_c1=Phones, MP3s, GPS_&_v=1_&_em=Exception: Error: [$sc:ind] Aborting!_&_ua=Opera/9.80 (Android; Opera Mini/30.0.2254/77.161; U; ru) Presto/2.12.423 Version/12.16_&_time=04/Dec/2017:21:03:07 +0200_&_qtype=get"
ID-ZZ "el=search_&_dl=https://example.market.com/?dt=Market – Electronics Store | Web Store (Market.com)_&_id=104777577_&_a=770227875_&_t=pageview_&_ua=Mozilla/5.0 (Linux; Android 7.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36_&_time=04/Dec/2017:19:39:04 +0200_&_qtype=get"

没有太多那些撕裂的请求正文字符串，大多数都在一行中，正如预期的那样。此外，只有GET请求有错误，因此搜索模式不应包含Get（这不是必需的）。

2 个答案:

答案 0 :(得分：1)

Awk 解决方案：

awk 'f{ if (/^\[/) { printf "\042\n"; f=0 } else printf("%s", $0) }
     / Get:/{ 
         f=1; gsub(/[\[\]]/, "", $4); id=$4; sub(/^.* Get: /, "");
         gsub("&", "_&_"); printf "%s \042%s",id,$0 
     }
     END{ if (f) printf "\042\n" }' file
  • / Get:/ - 遇到＆＃34;获取请求＆＃34; 行
    • f=1 - f是表示从属/内部处理的标记
    • id=$4 - 捕获 ID 字段（例如ID-XX）

输出：

ID-XX "sr=342x487_&_c64=(not set)_&_c1=Phones, MP3s, GPS_&_v=1_&_c33=427_&_d28=Like&je=0&s4d=4-b&c32=(not set)&ua=Opera/9.80 (Android; Opera Mini/32.0.2254/77.161; U; uk) Presto/2.12.423 Version/12.16&time=04/Dec/2017:20:52:02 +0200&qtype=get"
ID-YY "sr=342x487_&_c64=(not set)_&_c1=Phones, MP3s, GPS_&_v=1_&_em=Exception: Error: [$sc:ind] Aborting!_&_ua=Opera/9.80 (Android; Opera Mini/30.0.2254/77.161; U; ru) Presto/2.12.423 Version/12.16_&_time=04/Dec/2017:21:03:07 +0200_&_qtype=get"
ID-ZZ "el=search_&_dl=https://market.com/?dt=Market – Electronics Store | Web Store (Market.com)_&_id=104777577_&_a=770227875_&_t=pageview_&_ua=Mozilla/5.0 (Linux; Android 7.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36_&_time=04/Dec/2017:19:39:04 +0200_&_qtype=get"

答案 1 :(得分：0)

我知道您希望保留ERROR行并格式化它们。

我们不知道分隔符是什么。

奇怪的是，让你保持最后一线。

你可以试试这个sed 

"file:/storage/emulated/0/Pictures/JPEG_20171211_162442_-1697405777.jpg".
相关问题
最新问题