Question

我有一些从powershell导出到CSV的事件日志。如何获取和扩展消息字段，以便我可以在elasticsearch中使用其中的字段？我现在正在使用pandas将数据发送到elasticsearch。

当前拆分方法如下：

sec_events['action'] = sec_events.join(sec_events['message'].str.split('\\s\\s\\s\\s', 1, expand=True).apply(pd.Series))[0]
sec_events['message'] = sec_events.join(sec_events['message'].str.split('\\s\\s\\s\\s', 1, expand=True).apply(pd.Series))[1]

这输出我的方法，如：

0,mycompname.domm.f.f.a,4688,successaudit,microsoft-windows-security-auditing,3/7/2017 10:38:16 am,3/7/2017 10:38:16 am,NONE,"subject:   security id:  s-1-5-18   account name:  mycompname$   account domain:  domm   logon id:  0x3e7    process information:   new process id:  0x1a54   new process name: c:\windows\system32\ipconfig.exe   token elevation type: %%1936   creator process id: 0x1b38   process command line:     token elevation type indicates the type of token that was assigned to the new process in accordance with user account control policy.    type 1 is a full token with no privileges removed or groups disabled.  a full token is only used if user account control is disabled or if the user is the built-in administrator account or a service account.    type 2 is an elevated token with no privileges removed or groups disabled.  an elevated token is used when user account control is enabled and the user chooses to start the program using run as administrator.  an elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the administrators group.    type 3 is a limited token with administrative privileges removed and administrative groups disabled.  the limited token is used when user account control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using run as administrator.",a new process has been created.
1,mycompname.domm.f.f.f,4656,failureaudit,microsoft-windows-security-auditing,3/7/2017 10:38:05 am,3/7/2017 10:38:05 am,NONE,subject:   security id:  s-1-5-18   account name:  mycompname$   account domain:  f   logon id:  0x3e7    object:   object server:  security   object type:  key   object name:  \registry\machine\system\controlset001\services\policyagent\parameters\cache   handle id:  0x0    process information:   process id:  0x1b14   process name:  c:\windows\system32\reg.exe    access request information:   transaction id:  {00000000-0000-0000-0000-000000000000}   accesses:  %%4432      %%4435         access reasons:  -   access mask:  0x9   privileges used for access check: -   restricted sid count: 0,a handle to an object was requested.

如何从导出的csv中将windows事件日志的消息字段转换为elasticsearch / json样式格式的字段？

我正在寻找类似下面的json格式，所以我可以摄入logstash：

{message: {
    subject:  ,
    security id: s-1-5-18,
    account name: mycompname$,
    account domain:  domm,
    logon id:  0x3e7,
    process information:  ,
    new process id:  0x1a54,
    new process name: c:\windows\system32\ipconfig.exe,
    token elevation type: %%1936,
    creator process id: 0x1b38,
    process command line:  ,
    process command line:  ,
    string1: token elevation type indicates the type of token that was assigned to the new process in accordance with user account control policy.
             type 1 is a full token with no privileges removed or groups disabled.
             a full token is only used if user account control is disabled or if the user is the built-in administrator account or a service account.
             type 2 is an elevated token with no privileges removed or groups disabled.
             an elevated token is used when user account control is enabled and the user chooses to start the program using run as administrator.
             an elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the administrators group.
             type 3 is a limited token with administrative privileges removed and administrative groups disabled.
             the limited token is used when user account control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using run as administrator.


    action: a new process has been created.

    }
}

我尝试过以下几项：

sec_events['message'] = sec_events.message.replace([':\s+,', '\[', '\]', ':\s+', ',\s+','\s\s\s'],['":"none","', '{"', '"}', '":"', '","','","'], regex=True)

我似乎无法正常工作。任何帮助都会很棒，谢谢你。

我注意到以下分割，

3 exact white spaces = transition from field to field 
6 exact white spaces = break between the accesses field 
2 exact white spaces = transition from field name to data if the data is there.

Answer 1

问题最终成为匹配之间没有拿到结束逗号的匹配。

示例：

,Token elevation type: %%1936,
creator process id: 0x1b38,
process command line:  C:\windows\system32\cmd.exe,
process command line:  ,

匹配只是

 Key                    Value
 Token elevation type   %%1936
 process command line   C:\windows\system32\cmd.exe

弹出搜索的Windows事件日志消息字段

1 个答案: