如何使HTTP :: Proxy与HTTPS URL一起使用？

时间：2018-10-09 14:15:02

标签： perl http-proxy

在下面的代码示例中，我使用HTTP::Proxy启动代理服务器，并尝试使用它来请求HTTPS URL，但是代理服务器实际上没有发出请求，或者没有返回请求。响应。但是，如果我使URL使用HTTP（不安全），则请求成功。我已经安装了IO::Socket::SSL和LWP::UserAgent::https（确实是秘密部门！），但是仍然无法通过代理获得HTTPS请求。如何让HTTP::Proxy使用HTTPS URL？

这是我的代码：

#!/usr/bin/env perl

use strict;
use warnings;
use Data::Printer;
use HTTP::Proxy ':log';
use Mojo::UserAgent ();

my $URL = 'https://www.yahoo.com';
my $PROXY_PORT = 8667;

my $pid = fork();

if ($pid) { # I am the parent
    print "Press ^c to kill proxy server...\n";
    my $proxy = HTTP::Proxy->new( port => $PROXY_PORT );
    $proxy->logmask(ALL);
    $proxy->via(q{});
    $proxy->x_forwarded_for(0);
    $proxy->start;

    waitpid $pid, 0;
}
elsif ($pid == 0) { # I am the child
    sleep 3; # Allow the proxy server to start

    my $ua = Mojo::UserAgent->new;
    $ua->proxy
        ->http("http://127.0.0.1:$PROXY_PORT")
        ->https("http://127.0.0.1:$PROXY_PORT");

    my $tx = $ua->get($URL);

    if ($tx->error) {
        p $tx->error;
    }
    else {
        print "Success!\n";
    }
}
else {
    die 'Unknown result after forking';
}

将上述脚本另存为testcase-so.pl并运行它：

$ MOJO_CLIENT_DEBUG=1 ./testcase-so.pl
Press ^c to kill proxy server...
-- Blocking request (https://www.yahoo.com)
-- Connect c66a92739c09c76fa24029e8079808c7 (https://www.yahoo.com:443)
-- Client >>> Server (https://www.yahoo.com)
CONNECT www.yahoo.com:443 HTTP/1.1\x0d
User-Agent: Mojolicious (Perl)\x0d
Content-Length: 0\x0d
Host: www.yahoo.com\x0d
Accept-Encoding: gzip\x0d
\x0d

-- Client >>> Server (https://www.yahoo.com)

[Tue Oct  9 12:02:54 2018] (12348) PROCESS: Forked child process 12352
[Tue Oct  9 12:02:54 2018] (12352) SOCKET: New connection from 127.0.0.1:45312
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: CONNECT www.yahoo.com:443
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: Accept-Encoding: gzip
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: Host: www.yahoo.com
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: User-Agent: Mojolicious (Perl)
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: Content-Length: 0
[Tue Oct  9 12:02:54 2018] (12352) RESPONSE: 200 OK
[Tue Oct  9 12:02:54 2018] (12352) RESPONSE: Date: Tue, 09 Oct 2018 12:02:54 GMT
[Tue Oct  9 12:02:54 2018] (12352) RESPONSE: Transfer-Encoding: chunked
[Tue Oct  9 12:02:54 2018] (12352) RESPONSE: Server: HTTP::Proxy/0.304
-- Client <<< Server (https://www.yahoo.com)
HTTP/1.1 200 OK\x0d
Date: Tue, 09 Oct 2018 12:02:54 GMT\x0d
Transfer-Encoding: chunked\x0d
Server: HTTP::Proxy/0.304\x0d
\x0d

[Tue Oct  9 12:03:14 2018] (12352) CONNECT: Connection closed by the client
[Tue Oct  9 12:03:14 2018] (12352) PROCESS: Served 1 requests
[Tue Oct  9 12:03:14 2018] (12352) CONNECT: End of CONNECT proxyfication
\ {
    message   "Proxy connection failed"
}
[Tue Oct  9 12:03:15 2018] (12348) PROCESS: Reaped child process 12349
[Tue Oct  9 12:03:15 2018] (12348) PROCESS: 1 remaining kids: 12352
[Tue Oct  9 12:03:15 2018] (12348) PROCESS: Reaped child process 12352
[Tue Oct  9 12:03:15 2018] (12348) PROCESS: 0 remaining kids:
^C[Tue Oct  9 12:04:04 2018] (12348) STATUS: Processed 2 connection(s)
$

将$URL切换为不使用https：

$ MOJO_CLIENT_DEBUG=1 ./testcase-so.pl
Press ^c to kill proxy server...
-- Blocking request (http://www.yahoo.com)
-- Connect f792ee97a0362ab493575d8116e69e59 (http://127.0.0.1:8667)
-- Client >>> Server (http://www.yahoo.com)
GET http://www.yahoo.com HTTP/1.1\x0d
Accept-Encoding: gzip\x0d
Content-Length: 0\x0d
Host: www.yahoo.com\x0d
User-Agent: Mojolicious (Perl)\x0d
\x0d

[Tue Oct  9 12:09:38 2018] (12656) PROCESS: Forked child process 12659
-- Client >>> Server (http://www.yahoo.com)

[Tue Oct  9 12:09:38 2018] (12659) SOCKET: New connection from 127.0.0.1:58288
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: GET http://www.yahoo.com
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: Accept-Encoding: gzip
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: Host: www.yahoo.com
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: User-Agent: Mojolicious (Perl)
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: Content-Length: 0
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: 301 Moved Permanently
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Cache-Control: no-store, no-cache
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Date: Tue, 09 Oct 2018 14:10:01 GMT
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Transfer-Encoding: chunked
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Via: http/1.1 media-router-fp1006.prod.media.bf1.yahoo.com (ApacheTrafficServer [c s f ])
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Location: https://www.yahoo.com/
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Server: ATS
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Content-Language: en
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Content-Length: 8
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Content-Type: text/html
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Content-Security-Policy: sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-popups-to-escape-sandbox allow-presentation; report-uri https://csp.yahoo.com/beacon/csp?src=ats&site=frontpage&region=US&lang=en-US&device=desktop&yrid=&partner=;
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: X-Frame-Options: SAMEORIGIN
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: X-XSS-Protection: 1; report="https://csp.yahoo.com/beacon/csp?src=fp-hpkp-www"
-- Client <<< Server (http://www.yahoo.com)
HTTP/1.1 301 Moved Permanently\x0d
Cache-Control: no-store, no-cache\x0d
Date: Tue, 09 Oct 2018 14:10:01 GMT\x0d
Transfer-Encoding: chunked\x0d
Via: http/1.1 media-router-fp1006.prod.media.bf1.yahoo.com (ApacheTrafficServer [c s f ])\x0d
Location: https://www.yahoo.com/\x0d
Server: ATS\x0d
Content-Language: en\x0d
Content-Length: 8\x0d
Content-Type: text/html\x0d
Content-Security-Policy: sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-popups-to-escape-sandbox allow-presentation; report-uri https://csp.yahoo.com/beacon/csp?src=ats&site=frontpage&region=US&lang=en-US&device=desktop&yrid=&partner=;\x0d
X-Frame-Options: SAMEORIGIN\x0d
X-XSS-Protection: 1; report="https://csp.yahoo.com/beacon/csp?src=fp-hpkp-www"\x0d
\x0d

-- Client <<< Server (http://www.yahoo.com)
8\x0d
redirect\x0d
0\x0d
\x0d

Success!
[Tue Oct  9 12:09:38 2018] (12659) SOCKET: Getting request failed: Client closed
[Tue Oct  9 12:09:39 2018] (12656) PROCESS: Reaped child process 12657
[Tue Oct  9 12:09:39 2018] (12656) PROCESS: 1 remaining kids: 12659
[Tue Oct  9 12:09:39 2018] (12656) PROCESS: Reaped child process 12659
[Tue Oct  9 12:09:39 2018] (12656) PROCESS: 0 remaining kids:
^C[Tue Oct  9 12:09:45 2018] (12656) STATUS: Processed 2 connection(s)
$

1 个答案:

答案 0 :(得分：4)

HTTP :: Proxy中存在一个错误，因为它对CONNECT请求返回错误的响应：

-- Client <<< Server (https://www.yahoo.com)
HTTP/1.1 200 OK\x0d
Date: Tue, 09 Oct 2018 12:02:54 GMT\x0d
Transfer-Encoding: chunked\x0d
Server: HTTP::Proxy/0.304\x0d
\x0d

对CONNECT请求的响应不能有任何主体，这意味着它不应具有像Transfer-Encoding: chunked那样声明主体的HTTP标头。此错误发生在所有使用CONNECT发出HTTP/1.1请求的客户端上。如果CONNECT是用HTTP/1.0完成的，那么问题就消失了，因为Transfer-Encoding: chunked尚未用HTTP/1.0定义，因此HTTP :: Proxy无法发送它。

当尝试将curl与HTTP :: Proxy一起使用时，会发生相同的问题，因此，这不仅仅是Mojo :: UserAgent的问题。我已经对HTTP :: Proxy进行了补丁以正确响应。有关详细信息和您需要应用的（小的）差异，请参见this pull request。

相关问题
最新问题