Pipe in for loop breaks double quoted variables

时间：2015-05-08 10:02:27

标签： batch-file for-loop pipe environment-variables double-quotes

Situation: Using batchscript to retrieve certain values from a JSON.
I've got the following batchscript:

www­-.ageofauti­sm.com

What does it do?
Upon having entered a npo.nl program-url, the first for-loop strips the url down to the prid:my-super-server.ageofauti­sm.com. In the second for-loop curl.exe retrieves the JSON...:

@ECHO off

ECHO Enter npo.nl program-url :
SET url=
SET /P url=
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 for example
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
    FOR /F "delims=" %%B IN ('curl.exe -s http://e.omroep.nl/metadata/aflevering/%%A ^| jq.exe -R -r -s ".[1+index(\"^(\"): rindex(\"^)\")]"') DO (
        FOR /F "delims=" %%C IN ('ECHO %%B ^| jq.exe -r .start') DO SET ss=%%C
        FOR /F "delims=" %%C IN ('ECHO %%B ^| jq.exe -r .eind') DO SET to=%%C
        FOR /F "delims=" %%C IN ('ECHO %%B ^| jq.exe -r .tijdsduur') DO SET t=%%C
        ECHO Start:    !ss!
        ECHO End:      !to!
        ECHO Duration: !t!
    )
)
ENDLOCAL
PAUSE

...and sends it through a pipe to jq.exe which removes the non-JSON-data POMS_VPRO_850040 and parseMetadata({"STATUS":"OK","VERSION":"1.11.12","prid":"VPWON_1232766","titel":"Schuim & As","aflevering_titel":"","info":"Schuim & As met Jelle Brandt Corstius","ratio":"16:9","medium":"tv","gidsdatum":"2015-05-03","tijdsduur":"00:05:27","start":"00:23:13","eind":"00:28:40","url":"","webcast":1,"images":[{"size":"640x480","ratio":"4:3","url":"http:\/\/images.poms.omroep.nl\/image\/sx480\/c640x480\/606030.jpg"},{"size":"720x405","ratio":"16:9","url":"http:\/\/images.poms.omroep.nl\/image\/sx405\/c720x405\/606030.jpg"}],"omroepen":[{"naam":"VPRO"}],"pubopties":["adaptive","h264_bb","h264_sb","h264_std"],"tt888":"ja","serie":{"srid":"VPWON_1232748","serie_titel":"Buitenhof","serie_url":null},"sitestat":{"baseurl":"http:\/\/b.scorecardresearch.com\/p?c1=2&c2=17827132&ns_site=po-totaal","programurl":"uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503","programurlpost":"category=uitzendinggemist&thema=informatief&po_source=video","baseurl_subtitle":"http:\/\/nl.sitestat.com\/klo\/po\/s","subtitleurl":"uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.buitenhof","subtitleurlpost":"category=uitzendinggemist&po_source=video&po_sitetype=webonly"},"reclame":"http:\/\/pubads.g.doubleclick.net\/gampad\/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&sz=_sz_&correlator=_correlator_&iu=\/9233\/_site_\/buitenhof&url=_url_&cust_params=genre%3Dinformatief%2Cnieuws%2Factualiteiten%26dur%3D3284%26prid%3DVPWON_1232766%26srid%3DVPWON_1232748%26player%3D_player_","streamSense":{"episode":"buitenhof","program":"buitenhof","station":"nederland_1","sitestatname":"uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503","sko":"TRUE","sko_dt":"20150503","sko_pr":"buitenhof","sko_stid":"1","sko_ty":"tv.seg","sko_prid":"vpwon1232766","sko_t":"1210","sko_cl":"3284"}}) //epc and leaves the single line intact. This is for 2 reasons: 1) with non-JSON-data present we can't process anything, and 2) for-loops process only 1 line at a time.
The subsequent jq.exe's retrieve values for the specified objects without double quotes.
As long as curl.exe and jq.exe are in the same directory as the batchscript, or in the %path%-variable, this is working all fine:

parseMetadata(

Now I want to call curl.exe and jq.exe from another map. One with spaces in it:

) //epc

For the 2nd for-loop this causes problems:

Start:     00:23:13
End:       00:28:40
Duration:  00:05:27

While SET curl="C:\map with spaces\curl.exe" SET jq="C:\map with spaces\jq.exe" @ECHO off ECHO Enter npo.nl program-url : SET url= SET /P url= :: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 for example SETLOCAL ENABLEDELAYEDEXPANSION FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO ( FOR /F "delims=" %%B IN ('%curl% -s http://e.omroep.nl/metadata/aflevering/%%A ^| %jq% -R -r -s ".[1+index(\"^(\"): rindex(\"^)\")]"') DO ( FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -r .start') DO SET ss=%%C FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -r .eind') DO SET to=%%C FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -r .tijdsduur') DO SET t=%%C ECHO Start: !ss! ECHO End: !to! ECHO Duration: !t! ) ) ENDLOCAL PAUSE does work, it seems 'C:\map' is not recognized as an internal or external command, operable program or batch file. doesn't. So for some reason things go wrong as soon as 2 variables in a pipe are parsed.

Well, no more pipe then:

'ECHO %%X ^| %jq%'

Now curl.exe and jq.exe each in a for-loop. At first this seems to work fine. The 3 values are echoed, but then things go wrong:

'%curl% ^| %jq%'

Like I said before; for-loops parse and process only 1 line at a time. The non-JSON-data SET curl="C:\map with spaces\curl.exe" SET jq="C:\map with spaces\jq.exe" @ECHO off ECHO Enter npo.nl program-url : SET url= SET /P url= :: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 for example SETLOCAL ENABLEDELAYEDEXPANSION FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO ( FOR /F "delims=" %%B IN ('%curl% -s http://e.omroep.nl/metadata/aflevering/%%A') DO ( FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -R -r -s ".[1+index(\"^(\"): rindex(\"^)\")]"') DO ( FOR /F "delims=" %%D IN ('ECHO %%C ^| %jq% -r .start') DO SET ss=%%D FOR /F "delims=" %%D IN ('ECHO %%C ^| %jq% -r .eind') DO SET to=%%D FOR /F "delims=" %%D IN ('ECHO %%C ^| %jq% -r .tijdsduur') DO SET t=%%D ECHO Start: !ss! ECHO End: !to! ECHO Duration: !t! ) ) ) ENDLOCAL PAUSE on the 2nd line causes the for-loop to start over, which goes horribly wrong as you can see. That's the reason for the pipe between curl and jq in the code above. To output 1 single line to process. Sadly that didn't work either...sigh.

Of course using temporary files is a last resort when curl and jq are still in a map with spaces in it, but I prefer to use variables, so I'm trying to solve the pipe-issue. I've tried 'usebackq' in the for-loop using backticks around the command instead of single-quotes for instance, but to no avail.
So far I haven't found a solution. Does anyone have an explanation for this behaviour and how to solve it?

2 个答案:

答案 0 :(得分：3)

感谢Dave Benham's answer on a related issue我找到了解决方案！
它似乎是WinXP中一个特定的FOR / F错误，猜猜是什么，这里我还在使用WinXP。为了修复主要攻击者，curl-pipe-jq-for-loop，我必须在整个命令之后将^"放在和之前。因此整个批量脚本，我也进一步改进了：

@ECHO off
CLS

:: NPO JSON-extractor geschreven door Reino Wijnsma, 2015 (reino@degeelebosch.nl)

SET batchname=NPO JSON-extractor
SET version=1.1
TITLE %batchname% %version%

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq-1.5rc1.exe"

:Check
IF EXIST %curl% (
    IF EXIST %jq% (
        GOTO Input
    ) ELSE (
        ECHO 'jq.exe' niet gevonden.
        ECHO.
        PAUSE
        GOTO :eof
    )
    GOTO Input
) ELSE (
    ECHO 'curl.exe' niet gevonden.
    ECHO.
    PAUSE
    GOTO :eof
)

:Input
ECHO Voer npo.nl programmalink in :
SET url=
SET /P url=
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 bijvoorbeeld
IF "%url%"=="" GOTO :eof

SETLOCAL ENABLEDELAYEDEXPANSION
FOR %%A IN ("%url%") DO (
    FOR /F "delims=" %%B IN ('^"%curl% -s http://e.omroep.nl/metadata/aflevering/%%~nxA ^| %jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"^"') DO (
        ECHO.
        ECHO JSON:
        FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% .') DO ECHO %%C
        ECHO.
        FOR /F "tokens=1-3" %%C IN ('ECHO %%B ^| %jq% -r "[.tijdsduur,.start,.eind] | @tsv"') DO (
            ECHO Tijdsduur: %%C
            IF NOT "%%D"=="" (
                SET ss=%%D
                SET to=%%E
                SET /A "_ss=((1!ss:~0,2!-100)*3600)+((1!ss:~3,2!-100)*60)+(1!ss:~6,2!-100)"
                SET /A "_to=((1!to:~0,2!-100)*3600)+((1!to:~3,2!-100)*60)+(1!to:~6,2!-100)"
                ECHO Start:     %%D (!_ss!s^)
                ECHO Einde:     %%E (!_to!s^)
            )
        )
    )
)
ECHO.
ENDLOCAL
GOTO Input


未来参考的重要说明：

jq-syntax:           jq -R -r -s  '.[1+index("("):   rindex(")")]'
cmd-shell:           jq -R -r -s  ".[1+index(\"(\"): rindex(\")\")]"
for-loop:           'jq -R -r -s  ".[1+index(\"(\"): rindex(\"^)\")]"'
for-loop (path): '^"%jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"^"'

- 对于cmd-shell，您必须使用换行符\来转义双引号。
- 虽然两个右括号是jq语法的一部分，但双引号之间的1不是。所以当在for循环中，为了防止这个for循环关闭，你必须用^来逃避这个循环。
- 当jq的可执行路径放在带双引号的变量中时，为了规避WinXP错误，你还必须将整个命令放在^"之间，因为括号现在被认为是特殊字符！此解决方法与Vista及更高版本兼容。 （另见DosTips.com）

答案 1 :(得分：0)

如果无法测试，我建议您尝试使用以下编写的批处理代码：

@ECHO off
SET "curl=C:\map with spaces\curl.exe"
SET "jq=C:\map with spaces\jq.exe"
ECHO Enter npo.nl program-url :
SET "url="
SET /P "url="
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 for example
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
    FOR /F "delims=" %%B IN ('"%curl%" -s http://e.omroep.nl/metadata/aflevering/%%A') DO (
        FOR /F "delims=" %%C IN ('ECHO %%B ^| "%jq%" -R -r -s ".[1+index(\"^(\"): rindex(\"^)\")]"') DO (
            FOR /F "delims=" %%D IN ('ECHO %%C ^| "%jq%" -r .start') DO SET "ss=%%D"
            FOR /F "delims=" %%D IN ('ECHO %%C ^| "%jq%" -r .eind') DO SET "to=%%D"
            FOR /F "delims=" %%D IN ('ECHO %%C ^| "%jq%" -r .tijdsduur') DO SET "t=%%D"
            ECHO Start:    !ss!
            ECHO End:      !to!
            ECHO Duration: !t!
        )
    )
)
ENDLOCAL
PAUSE

与代码的不同之处在于将字符串赋值给环境变量时双引号的位置。

命令 set 通常与参数 variable = value 一起使用，整个字符串就是参数。

命令

SET "curl=C:\map with spaces\curl.exe"

将命令 set 的整个参数放入双引号中。因此，环境变量curl使用字符串C:\map with spaces\curl.exe定义，而没有为其分配双引号。

作为在命令 set 的整个参数字符串上使用双引号的额外好处，现在通常不会在命令行末尾显示尾随空格和制表符。

但是使用

SET curl="C:\map with spaces\curl.exe"

导致完全不同的行为。命令 set 的整个参数 variable = value 现在不用双引号括起来。由于行中第一个双引号的位置不同，命令 set 现在创建环境变量curl，并使用双引号将字符串"C:\map with spaces\curl.exe"分配给它在命令行的末尾可能还包含现有的空格和制表符。

下面的批处理代码被复制到批处理文件中并执行，表明存在差异：

@echo off
set "Var1=String with spaces and "double quotes""   
set Var2="String with spaces and "double quotes""
set Var3="String with spaces and "double quotes" and 3 trailing spaces"   
echo Var1=#%Var1%#
echo Var2=#%Var2%#
echo Var3=#%Var3%#
pause

3个变量输出上的字符#用于显示指定字符串真正开始和结束的位置。

输出结果为：

Var1=#String with spaces and "double quotes"#
Var2=#"String with spaces and "double quotes""#
Var3=#"String with spaces and "double quotes" and 3 trailing spaces"   #

在定义Var1的行尾有3个尾随空格，但它们会被忽略，因为双引号包含命令 set 的整个参数字符串。

在定义Var2的行上没有尾随空格，但是所有4个双引号现在都是指定字符串的一部分，而不仅仅是字符串double quotes部分的两个双引号。

最后定义Var3的行也有3个尾随空格，这些空格也分配给变量，这在引用环境变量值的命令行中通常是不需要的。

即使变量名和值都包含空格，最好始终使用set "variable=value"。这种表示法只是保护不可见的尾随空格或制表符也被分配给环境变量。

由于带有包含空格的路径的curl.exe现在已分配给没有双引号的环境变量curl，因此必须在包含%curl%的完整字符串周围使用双引号，这是这里只是"%curl%"。 %jq%也是如此，因为未在较长的字符串中使用，因此只能使用"%jq%"。

最后一个提示：批处理文件的调试通常非常简单。将第一行从@ECHO off更改为@ECHO ON或删除/注释第一行会导致执行批处理文件，同时显示命令行解释器实际执行的内容。现在，通过查看已处理的命令行，可以快速找到代码中的错误。

相关问题
最新问题