Question

我的情况与this question中描述的情况非常相似（但是批量处理，而不是shell）。我做了一个简单的批处理脚本来遍历一个磁贴的行并使用python脚本从服务器下载数据（该过程本身比简单下载更复杂，它必须通过API进行身份验证并获取多个URL）。

第一个版本如下：

for /F "tokens=*" %%A in (client_name_list.txt) do python download_metadata.py "%%A"

它等待每次迭代完成以继续前进的方式，所以我将其更新为以下内容：

for /F "tokens=*" %%A in (client_name_list.txt) do start cmd /C python download_metadata.py "%%A"

第二个版本按我想要的方式执行，但是，因为文件client_name_list.txt大约有30,000行，所以很多命令提示开始产生，计算机会在几秒钟内冻结。

如何限制CMD运行实例的数量（例如10）并使脚本等到有一个＆＃34;免费CMD插槽＆＃34;去下一行？

Answer 1

改编自my answer to "Parallel execution of shell processes"。点击链接获取解释。

@echo off
setlocal enableDelayedExpansion

:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
  set "lockHandle=1"
  set "showOutput=1"
) else (
  set "lockHandle=1^>nul 9"
  set "showOutput="
)

:: Define the maximum number of parallel processes to run.
set "maxProc=10"

:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
  set "lock="
  for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
    set "lock=%%T"
    goto :break
  )
  :break
  set "lock=%temp%\lock%lock%_%random%_"

:: Initialize the counters
  set /a "startCount=0, endCount=0"

:: Clear any existing end flags
  for /l %%N in (1 1 %maxProc%) do set "endProc%%N="

:: Launch the commands in a loop
  set launch=1
  for /f "delims=" %%A in (client_name_list.txt) do (
    if !startCount! lss %maxProc% (
      set /a "startCount+=1, nextProc=startCount"
    ) else (
      call :wait
    )
    set cmd!nextProc!=%%A
    if defined showOutput echo -------------------------------------------------------------------------------
    echo !time! - proc!nextProc!: starting %%A
    2>nul del %lock%!nextProc!
    %= Redirect the lock handle to the lock file. The CMD process will     =%
    %= maintain an exclusive lock on the lock file until the process ends. =%
    start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 python download_metadata.py "%%A"
  )
  set "launch="

:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
  :: redirect stderr to null to suppress any error message if redirection
  :: within the loop fails.
  for /l %%N in (1 1 %startCount%) do 2>nul (
    %= Redirect an unused file handle to the lock file. If the process is    =%
    %= still running then redirection will fail and the IF body will not run =%
    if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
      %= Made it inside the IF body so the process must have finished =%
      if defined showOutput echo ===============================================================================
      echo !time! - proc%%N: finished !cmd%%N!
      if defined showOutput type "%lock%%%N"
      if defined launch (
        set nextProc=%%N
        exit /b
      )
      set /a "endCount+=1, endProc%%N=1"
    )
  )
  if %endCount% lss %startCount% (
    1>nul 2>nul ping /n 2 ::1
    goto :wait
  )

2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Done

Answer 2

在for循环的每次迭代中，您都可以计算CMD任务打开的数量。如果该值小于限制，则启动新任务，否则等待一个插槽空闲。

@echo off
set $Limit=11

setlocal enabledelayedexpansion

for /F "tokens=*" %%A in (client_name_list.txt) do (call:wait %%A)
exit/b

:wait
set "$cmd="
for /f %%a in ('tasklist ^| findstr /i "cmd"') do set /a $cmd+=1
if !$cmd! lss %$Limit% (
       start cmd /C python download_metadata.py "%1"
       goto:eof)
ping localhost -n 2 >nul
goto:wait

限制批处理脚本中生成的进程数

2 个答案: