Question

我在服务器上有三千个文件。我可以通过REST API调用一次检索一个。我写了一个命令来检索这些文件。它工作得很好，但是在大约200次下载后我的登录超时。

我想并行而不是连续下载所有这些文件。理想情况下，我想一次检索文件1-200，同时检索200-400，同时检索400-600等。

所以我的尝试：

FOR /L %i in (0,1,200) do wget --no-check-certificate --content-disposition  --load-cookies cookies.txt \ -p https://username:password@website.APICall.com/download/%i

如何将其转换为我想要创建的并行调用？

感谢。

Answer 1

安装了Cygwin和GNU Parallel后，您可以使用以下命令下载持续运行200个并行下载的3000个文件：

seq 3000 | parallel -j 200 wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:password@website.APICall.com/download/{}

Answer 2

~~I doubt your command works, because the iterator variable needs a double percent as far as I know, i.e. %i needs to be %%i.~~

Concerning the parallelization, you can try this:

FOR /L %%i IN (0,1,200) DO ( 
    start wget --no-check-certificate --content-disposition --load-cookies cookies.txt -p "https://username:password@website.APICall.com/download/%%i"
)

It will, for your first 200 downloads, spawn a seperate process (and shell window!) for every download. Doing so will cause a lot of load on the server and I'm not sure this is really the way to go forward. But it does what you've asked for.

Edit: The above note holds for using the command in a .bat file, if you're executing this on the shell directly, a single percent is sufficient.

Answer 3

不要经历Cygwin的麻烦;试图将Windows变成UNIX是一个复杂的问题，并增加了依赖层。使用PowerShell。

如果您可以在超时前下载200个文件，请将其分解为三个作业：

invoke-command -asjob -scriptblock {$files = @(1..200);$files | foreach-object{ & wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:password@website.APICall.com/download/$_}};
invoke-command -asjob -scriptblock {$files = @(201..400);$files | foreach-object{ & wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:password@website.APICall.com/download/$_}};
invoke-command -asjob -scriptblock {$files = @(601..400);$files | foreach-object{ & wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:password@website.APICall.com/download/$_}};

或者获取Invoke-Parallel并像这样使用它：

$filenames = @(1..600);    
invoke-parallel -InputObject $servers -throttle 200 -runspaceTimeout 30 -ScriptBlock { & wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:password@website.APICall.com/download/$_}

另一个（可能是最好的）选项是使用invoke-webrequest，但我不知道它是否适用于您的Cookie要求。

免责声明：由于我目前无法访问Windows或您的网址，因此无法在内存中工作。

Answer 4

另一种to the GNU parallel method是好的＆＃39; xargs选项-P：

$ seq 3000 | xargs -i '{}' -n 1 -P 200 wget <url_start>{}<url_end>

同时运行带有循环变量的命令，最多N个

4 个答案: