我想进行并行下载,但问题wget输出的文件名不正确。
url.txt
http://example.com/file1.zip?arg=tereef&arg2=okook
http://example.com/file2.zip?arg=tereef&arg2=okook
命令
xargs -P 4 -n 1 wget <url.txt
输出文件名
file1.zip?arg=tereef&arg2=okook
file2.zip?arg=tereef&arg2=okook
预期产出
file1.zip
file2.zip
我是bash的新手,请建议我如何输出正确的文件名,请不要建议for
循环或&
,因为它会阻止。
谢谢
答案 0 :(得分:1)
您可以使用必须导出的bash函数在当前shell之外看到
function mywget()
{
wget -O ${1%%\?*} "'$1'"
}
export -f mywget
xargs -P 4 -n 1 -I {} bash -c "mywget '{}'" < url.txt
答案 1 :(得分:0)
处理您的输入以生成所需的命令,然后通过xargs运行它。
perl -ne
- 迭代输入文件的行并执行内联程序
-e:执行perl one-liner
-n:遍历所有输入行,依次为$ _分配每个输入行。
xargs -P 4 -n 1 -i -t wget "{}"
-P 4:一次最多4个进程
-n 1:一次消耗一个输入行
-i:使用替换字符串&#34; {}&#34;
-t:在执行命令之前打印命令
perl -ne '
chomp(my ($url) = $_); # Remove trailing newline
my ($name) = $url =~ m|example.com/(.+)\?|; # Grab the filename
print "$url -O $name\n"; # Print all of the wget params
' url.txt | xargs -P 4 -n 1 -i -t wget "{}"
<强>输出强>
wget http://example.com/file1.zip?arg=tereef&arg2=okook -O file1.zip
wget http://example.com/file2.zip?arg=tereef&arg2=okook -O file2.zip
--2016-07-21 22:24:44-- http://example.com/file2.zip?arg=tereef&arg2=okook%20-O%20file2.zip
--2016-07-21 22:24:44-- http://example.com/file1.zip?arg=tereef&arg2=okook%20-O%20file1.zip
Resolving example.com (example.com)... Resolving example.com (example.com)... 93.184.216.34, 2606:2800:220:1:248:1893:25c8:1946
93.184.216.34, Connecting to example.com (example.com)|93.184.216.34|:80... 2606:2800:220:1:248:1893:25c8:1946
Connecting to example.com (example.com)|93.184.216.34|:80... connected.
connected.
HTTP request sent, awaiting response... HTTP request sent, awaiting response... 404 Not Found
2016-07-21 22:24:44 ERROR 404: Not Found.
404 Not Found
2016-07-21 22:24:44 ERROR 404: Not Found.
答案 2 :(得分:0)
使用GNU Parallel,它看起来像这样:
parallel -P 4 wget -O '{= s/\?.*//;s:.*/:: =}' {} <url.txt