我正在迭代从Windows主机提取的许多导出的安全事件日志，例如下面的示例数据框：
"MachineName","EventID","EntryType","Source","TimeGenerated","TimeWritten","UserName","Message"
"mycompname","5156","SuccessAudit","Microsoft-Windows-Security-Auditing","4/26/2017 10:47:41 AM","4/26/2017 10:47:41 AM",,"The Windows Filtering Platform has permitted a connection. Application Information: Process ID: 4 Application Name: System Network Information: Direction: %%14592 Source Address: 192.168.10.255 Source Port: 137 Destination Address: 192.168.10.238 Destination Port: 137 Protocol: 17 Filter Information: Filter Run-Time ID: 83695 Layer Name: %%14610 Layer Run-Time ID: 44"
"mycompname","4688","SuccessAudit","Microsoft-Windows-Security-Auditing","4/26/2014 10:47:03 AM","4/26/2014 10:47:03 AM",,"A new process has been created. Subject: Security ID: S-1-5-18 Account Name: mycompname$ Account Domain: mydomain Logon ID: 0x3e7 Process Information: New Process ID: 0x1b04 New Process Name: C:\Windows\SysWOW64\Macromed\Flash\FlashPlayerUpdateService.exe Token Elevation Type: %%1936 Creator Process ID: 0x300 Process Command Line: C:\windows\SysWOW64\Macromed\Flash\FlashPlayerUpdateService.exe Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy. Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account. Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group. Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator."
"mycompname","4673","SuccessAudit","Microsoft-Windows-Security-Auditing","4/26/2014 10:47:00 AM","4/26/2014 10:47:00 AM",,"A privileged service was called. Subject: Security ID: S-1-5-18 Account Name: mycompname$ Account Domain: mydomain Logon ID: 0x3e7 Service: Server: NT Local Security Authority / Authentication Service Service Name: LsaRegisterLogonProcess() Process: Process ID: 0x308 Process Name: C:\Windows\System32\lsass.exe Service Request Information: Privileges: SeTcbPrivilege"
我正在将其转换为从＆＃34; Message＆＃34;中提取关键：值对。列并将键转换为如下所示的列
def myfunc(folder):
file = ''.join(glob2.glob(folders + "\\*security*"))
df = pd.read_csv(file)
df.message = df.message.replace(["[ ]{6}", "[ ]{3}"],[","," ||"], regex=True)
message_results = df.message.str.extractall(r"\|([^\|]*?):(.*?)\|").reset_index()
message_results.columns = ["org_index", "match", "keys", "vals"]
# PART THAT TAKES THE LONGEST
p = pd.pivot_table(message_results, values="vals", columns=['keys'], index=["org_index"], aggfunc=np.sum)
df = df.join(p).fillna("NONE")
以上功能的输出：
MachineName,EventID,EntryType,Source,TimeGenerated,TimeWritten,UserName,Message, Application Information, Filter Information, Network Information, Process, Process Information, Service, Service Request Information, Subject,Account Domain,Account Name,Application Name,Creator Process ID,Destination Address,Destination Port,Direction,Filter Run-Time ID,Layer Name,Logon ID,New Process ID,New Process Name,Process Command Line,Process ID,Process Name,Protocol,Security ID,Server,Service Name,Source Address,Source Port,Token Elevation Type
mycompname,5156,SuccessAudit,Microsoft-Windows-Security-Auditing,4/26/2017 10:47:41 AM,4/26/2017 10:47:41 AM,NONE,The Windows Filtering Platform has permitted a connection. || Application Information: ||Process ID: 4 ||Application Name: System || Network Information: ||Direction: %%14592 ||Source Address: 192.168.10.255 ||Source Port: 137 ||Destination Address: 192.168.10.238 ||Destination Port: 137 ||Protocol: 17 || Filter Information: ||Filter Run-Time ID: 83695 ||Layer Name: %%14610 ||Layer Run-Time ID: 44, , , ,NONE,NONE,NONE,NONE,NONE,NONE,NONE, System ,NONE, 192.168.10.238 , 137 , %%14592 , 83695 , %%14610 ,NONE,NONE,NONE,NONE, 4 ,NONE, 17 ,NONE,NONE,NONE, 192.168.10.255 , 137 ,NONE
mycompname,4688,SuccessAudit,Microsoft-Windows-Security-Auditing,4/26/2017 10:47:03 AM,4/26/2017 10:47:03 AM,NONE,"A new process has been created. || Subject: ||Security ID: S-1-5-18 ||Account Name: mycompname$ ||Account Domain: mydomain ||Logon ID: 0x3e7 || Process Information: ||New Process ID: 0x1b04 ||New Process Name: C:\Windows\SysWOW64\Macromed\Flash\FlashPlayerUpdateService.exe ||Token Elevation Type: %%1936 ||Creator Process ID: 0x300 ||Process Command Line: C:\windows\SysWOW64\Macromed\Flash\FlashPlayerUpdateService.exe || Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy. || Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account. || Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group. || Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.",NONE,NONE,NONE,NONE, ,NONE,NONE, , mydomain , MEADWK4216DC190$ ,NONE, 0x300 ,NONE,NONE,NONE,NONE,NONE, 0x3e7 , 0x1b04 , C:\Windows\SysWOW64\Macromed\Flash\FlashPlayerUpdateService.exe , C:\windows\SysWOW64\Macromed\Flash\FlashPlayerUpdateService.exe ,NONE,NONE,NONE, S-1-5-18 ,NONE,NONE,NONE,NONE, %%1936
mycompname,4673,SuccessAudit,Microsoft-Windows-Security-Auditing,4/26/2017 10:47:00 AM,4/26/2017 10:47:00 AM,NONE,A privileged service was called. || Subject: ||Security ID: S-1-5-18 ||Account Name: mycompname$ ||Account Domain: mydomain ||Logon ID: 0x3e7 || Service: ||Server: NT Local Security Authority / Authentication Service ||Service Name: LsaRegisterLogonProcess() || Process: ||Process ID: 0x308 ||Process Name: C:\Windows\System32\lsass.exe || Service Request Information: ||Privileges: SeTcbPrivilege,NONE,NONE,NONE, ,NONE, , , , mydomain , mycompname$ ,NONE,NONE,NONE,NONE,NONE,NONE,NONE, 0x3e7 ,NONE,NONE,NONE, 0x308 , C:\Windows\System32\lsass.exe ,NONE, S-1-5-18 , NT Local Security Authority / Authentication Service , LsaRegisterLogonProcess() ,NONE,NONE,NONE
程序的功能有效但在较大的数据集（大约150000行）上的代码的p = pivot_table部分上却非常慢。
我目前正在使用concurrent.futures.ThreadPoolExecutor（maxworkers = 1000）迭代文件的每次读取，如下所示：
with concurrent.futures.ThreadPoolExecutor(max_workers=1000) as pool:
for folder in path:
if os.path.isdir(folder):
try:
print(folder)
pool.submit(myfunc(folder), 1000)
except:
print('error')
如何加快我的功能的数据透视表部分？
另外，有没有什么方法可以加速来自pandas的pivot_table调用？
对此的任何帮助将不胜感激。谢谢。
答案 0 :(得分：3)
您的代码有许多语法错误
pool.submit(myfunc(folder), 1000)
pool.submit method将函数作为第一个参数。
从我看到你的函数myfunc没有返回任何东西，绝对不是函数。
即便如此，根据我的理解，您正在尝试启动所有读取相同文件夹的1000工作人员，然后创建数据框。
在任何线程场景中，工作线程数应接近您运行的计算机上可用的核心数。这是常识，我不会引用任何内容。
产生1000名工人是很多开销，并且可能是你的慢功能的来源。你的所有工人似乎都在做同样的事情，这当然意味着你做了1000次同样的工作。
因此，根据您的编写，代码，我理解您正在尝试创建一个巨大的密钥空间，允许您切入任何指标并深入到数据集中。
您使用我看到的单列进行此操作。您应该将它们分成不同的列。正如评论者所暗示的那样，大熊猫有可以使用的分类列，但即使没有它们，如果关键部分位于不同的列中，键空间的索引也会小得多。您当前的数据集很可能几乎每行都有一个单独的键，因此不会将多行聚合在一起，从而使数据透视表与原始数据集的大小相同。
TLDR;
将您的键列拆分为多个列，最好是分类列。