无法在CURL PHP中设置主机

时间:2015-03-30 09:51:17

标签: php curl web-scraping web-crawler

我无法在curl中设置主机。如果我使用以下代码

,它仍然显示为localhost
function wget($url)
        {

            $agent= 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
            $curlHeaders = array (
                    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                    'Accept-Encoding: gzip, deflate',
                    'Accept-Language: en-US,en;q=0.5',
                    'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
                    'Connection: Keep-Alive',
                    'Pragma: no-cache',
                    'Referer: http://example.com/',
                    'Host: hostname',
                    'Cache-Control: no-cache',
                    'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'

            );
            $ch = curl_init();
            curl_setopt ($ch, CURLOPT_HTTPHEADER, $curlHeaders);
            curl_setopt ($ch, CURLOPT_HEADER, TRUE);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            curl_setopt($ch, CURLOPT_USERAGENT, $agent);
            curl_setopt($ch, CURLOPT_URL,$url);
            $result=curl_exec($ch);
            return $result;
        }

我使用fiddler来跟踪网络请求。我发现主机仍然是localhost enter image description here

如果我在浏览器中加载此相同的链接,我会在fiddler

中获得以下内容

enter image description here

我需要访问我指定的域名。我怎样才能做到这一点? 注意:我知道主机名不应包含协议。

替代地

另外我想知道是否有可能通过终端在浏览器中看到网站的源代码?

5 个答案:

答案 0 :(得分:3)

假设我们尝试欺骗Host标头,请完全省略Host标头,然后让curl对其进行排序。在这种情况下,只需删除'Host: hostname',,因为您已经使用curl_setopt($ch, CURLOPT_URL, $url);将curl自动设置为底部附近的代码。

如果您真的想自己设置Host标题,那么只需替换

即可
'Host: hostname',

"Host: ". parse_url($url, PHP_URL_HOST),

(注意:此功能不适用于相对URL。)

答案 1 :(得分:2)

尝试这样,

  curl_init('XXX.XXX.XXX.XXX');
  curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));

答案 2 :(得分:1)

如果你使用的是windows和xampp,那么尝试使用虚拟主机而不是localhost,然后它就会开始工作,我做了同样的事情。

答案 3 :(得分:1)

根据HTTP快速规范读取,我假设您的问题正在发生,因为发送了不正确的Host标头。我能够使用以下代码下载一些网站:

function wget($url, $follow = true) {

    $host = parse_url($url);

    $agent       = 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
    $curlHeaders = array(
        'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        'Accept-Encoding: gzip, deflate',
        'Accept-Language: en-US,en;q=0.5',
        'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
        'Connection: Keep-Alive',
        'Pragma: no-cache',
        'Referer: http://example.com/',
        'Host: ' . $host['host'] . (isset($host['port']) ? ':' . $host['port'] : null), // building host header
        'Cache-Control: no-cache',
        'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'
    );
    $ch          = curl_init();
    curl_setopt($ch, CURLOPT_HTTPHEADER, $curlHeaders);
    curl_setopt($ch, CURLOPT_HEADER, TRUE);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $follow); // following redirects or not
    curl_setopt($ch, CURLOPT_USERAGENT, $agent);
    curl_setopt($ch, CURLOPT_URL, $url);
    $result      = curl_exec($ch);
    return $result;
}

echo(wget('http://example.com'));

无论如何,这个功能不是通用版本。我个人会在重定向请求之间添加保存cookie等。基本更改在“主机”标题行内。我正在根据提供给功能的完整Host构建正确的$url标头。

答案 4 :(得分:-1)

将完整网址设为CURLOPT_UR L。