Question

我希望让我的脚本更快，并让我的输出更快。我正在处理我的csv中的大量数据，如果我输入完整的文件来完成脚本，则需要大约一个小时。

 $csv = Import-Csv 'U:\Local Group Members.csv' |
Where-Object {($_.Name0 -eq "administrators") -and ($_.caption0 -match "Microsoft Windows 10 Enterprise|Microsoft Windows 7 Enterprise|Microsoft Windows 7 Professional|Microsoft Windows 8 Enterprise|Microsoft Windows 8 Pro|Microsoft Windows 8.1 Enterprise|Microsoft Windows 8.1 Pro")} |
  Select-Object "Netbios_name0", "Name0", "Account0","category0","Domain0","Unique Account Name","Type0","caption0", "Excluded"
#Modify each line
Foreach ($row in $csv) {
If ($row.Type0 -eq 'Domain') { 
    $row."Unique Account Name" = "$($row.Domain0) - $($row.Account0)" 
    If ($row."Unique Account Name" -in @('ACCOUNTS - DODSCAN'.'ACCOUNTS - Domain Admins','ACCOUNTS - LADM_WS_Admins','ACCOUNTS - Tech Enterprise'))
         {$row."Excluded" = "True"}
    Else {$row."Excluded" = "False"}        
}
Else {
    $row."Unique Account Name" = "$($row.Netbios_name0) - $($row.Account0)"
    If ($row."Account0" -in @('esrxadm1n_esi','#Update','medco_tech','medco_admin'))
         {$row."Excluded" = "True"}
    Else {$row."Excluded" = "False"}
}
    Write-Host $row."Unique Account Name"
    Write-Host $row."Excluded"    
}

#Export CSV
 $csv | Export-Csv U:\$(get-date -f yyyy-MM-dd-hh-mm)-TestOutput1.csv -NoTypeInformation

任何人都可以帮助我加快速度吗？

Answer 1

我在评论中谈到的例子：

byte[] flags = new byte[]{6};

EnumSet<GoalType> goalTypes = getEnumValuesForFlags(GoalType.class, flags);

System.out.println(goalTypes); // [BREAK, PLACE]

Answer 2

如果你把所有这些都放在同一行？

$tab1 = @('ACCOUNTS - DODSCAN', 'ACCOUNTS - Domain Admins','ACCOUNTS - LADM_WS_Admins','ACCOUNTS - Tech Enterprise')
$tab2 = @('esrxadm1n_esi','#Update','medco_tech','medco_admin')

Import-Csv 'U:\Local Group Members.csv' | Select "administrators", "Netbios_name0", "Name0", "Account0","category0","Domain0","Unique Account Name","Type0","caption0", "Excluded" | %{

 if ($_.Name0 -eq "administrators" -and $_.caption0 -match "Microsoft Windows 10 Enterprise|Microsoft Windows 7 Enterprise|Microsoft Windows 7 Professional|Microsoft Windows 8 Enterprise|Microsoft Windows 8 Pro|Microsoft Windows 8.1 Enterprise|Microsoft Windows 8.1 Pro")
 {
    If ($_.Type0 -eq 'Domain') 
    { 
        $_."Unique Account Name" = "$($_.Domain0) - $($_.Account0)"
        $_."Excluded" = $_."Unique Account Name" -in $tab1   
    }
    Else 
    {
        $_."Unique Account Name" = "$($_.Netbios_name0) - $($_.Account0)"
        $_."Excluded" = $_."Account0" -in $tab2
    }

  $_

}

} | Export-Csv U:\$(get-date -f yyyy-MM-dd-hh-mm)-TestOutput1.csv -NoTypeInformation

Answer 3

不是让2个Foreach块在同一个对象上循环（$ csv中的$ row），为什么不组合它们呢？如果您的CSV很大，为什么要经历两次？

let photoResponse = response?.result.value

if let items = photoResponse?.items {
    print ("Items found: #\(items.count)")
   // do something with items here
}

另外，在顶部进行匹配的地方：

Foreach ($row in $csv) {
If ($row.Type0 -eq 'Domain') { 
    $row."Unique Account Name" = "$($row.Domain0) - $($row.Account0)" 
    Write-Host $row."Unique Account Name"
} Else { 
    $row."Unique Account Name" = "$($row.Netbios_name0) - $($row.Account0)"
 }
If (($row.Type0 -eq 'Local') -and ($row.Account0 -in @('esrxadm1n_esi','#Update','medco_tech','medco_admin')) -and ($row."Unique Account Name" -in @('ACCOUNTS - DODSCAN'.'ACCOUNTS - Domain Admins','ACCOUNTS - LADM_WS_Admins','ACCOUNTS - Tech Enterprise')))  { 
    $row."Excluded" = "True" 
    Write-Host $row."Excluded"
 } Else { 
    $row."Excluded" = "False" 
 }
}

Answer 4

我在这里有一个~600k的事件日志行CSV，我可以测试：

import-csv events.csv | select * | foreach { $_ } | Export-csv out.csv

需要5分15秒（SSD上约为180Mb）。使用write-host，它可以推动最多9分36秒。所以这可能是你的CSV很大（每行有很多信息？），一次把它全部加载到内存中？或写主机行。

尝试将其更改为流媒体方式：

Import-Csv 'U:\Local Group Members.csv' |
    Where-Object {

        ($_.Name0 -eq "administrators") -and
        ($_.caption0 -match "Microsoft Windows (10 Enterprise|7 Enterprise|7 Professional|8 Enterprise|8 Pro|8.1 Enterprise|8.1 Pro)")

    } | ForEach-Object { 

        If ($row.Type0 -eq 'Domain')
        {
            $row."Unique Account Name" = $row.Domain0 + ' - ' + $row.Account0
            $row."Excluded" = ($row."Unique Account Name" -in @('ACCOUNTS - DODSCAN'.'ACCOUNTS - Domain Admins','ACCOUNTS - LADM_WS_Admins','ACCOUNTS - Tech Enterprise'))
        }
        Else
        {
            $row."Unique Account Name" = $row.Netbios_name0 + ' - ' + $row.Account0
            $row."Excluded" = ($row."Account0" -in @('esrxadm1n_esi','#Update','medco_tech','medco_admin'))
        }

    } | Select-Object "Netbios_name0", "Name0", "Account0",
                      "category0","Domain0","Unique Account Name",
                      "Type0","caption0", "Excluded" |
    Export-Csv U:\$(get-date -f yyyy-MM-dd-hh-mm)-TestOutput1.csv -NoTypeInformation

我还在一些冗余的“if（事情测试是真的）然后（真实）其他如果（事物测试是假的）然后（假）”进入“（事物测试）”，删除子表达式，缩短了正则表达式，并将选择结果移动到指南'过滤器的早期，选择迟到'。

但是，它比PS方法明显更快，将涉及将CSV处理从PowerShell下放到.Net，或者将其视为文本文件并首先以某种方式进行某些过滤。

Powershell CSV脚本更快

4 个答案: