如何使用cURL获取网站

时间:2016-03-10 07:15:39

标签: php curl web-scraping libcurl

我无法在php中使用cURL获取数据我可以使用curl命令在终端中执行此操作,如下所示 -

curl http://www.universalapplianceparts.com/search.aspx?find=w10130694

此命令在终端中提供预期结果。

我想用curl

在php中获得相同的结果

下面是我的PHP代码 -

$url = "http://www.universalapplianceparts.com/search.aspx?find=W10130694";
$ch = curl_init();  
    curl_setopt($ch, CURLOPT_URL,"http://www.universalapplianceparts.com/");
    curl_setopt($ch, CURLOPT_POST, true);  
    curl_setopt($ch, CURLOPT_HEADER, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $url); 
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    $output = curl_exec ($ch); 
    curl_close ($ch); 
    var_dump($output);

请让我知道为什么我没有得到相同的结果使用PHP curl因为我进入终端。

谢谢

3 个答案:

答案 0 :(得分:3)

试试这个::

    $url = "http://www.universalapplianceparts.com/search.aspx?find=W10130694";
$ch1= curl_init();
curl_setopt ($ch1, CURLOPT_URL, $url );
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch1,CURLOPT_VERBOSE,1);
curl_setopt($ch1, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0)');
curl_setopt ($ch1, CURLOPT_REFERER,'http://www.google.com');  //just a fake referer
curl_setopt($ch1, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch1,CURLOPT_POST,0);
curl_setopt($ch1, CURLOPT_FOLLOWLOCATION, 20);

$htmlContent= curl_exec($ch1);
echo $htmlContent;

答案 1 :(得分:0)

试试这个:

 $url = "http://www.universalapplianceparts.com/search.aspx?find=W10130694";
                $options = Array(
    CURLOPT_RETURNTRANSFER => TRUE,  
    CURLOPT_FOLLOWLOCATION => TRUE,  
    CURLOPT_AUTOREFERER => TRUE,
    CURLOPT_HTTPHEADER => array("Content-Type: application/text"),
    CURLOPT_TIMEOUT => 120,  
    CURLOPT_MAXREDIRS => 10,
    CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8",
    CURLOPT_URL => $url,  
    );

    $ch = curl_init(); 

    curl_setopt_array($ch, $options);  

    $data = curl_exec($ch);

    curl_close($ch);   

答案 2 :(得分:0)

我尝试了您的代码并获得与长度相关的错误,因此您需要在curl请求中添加内容长度。尝试下面的代码希望它能帮到你:

$url = "http://www.universalapplianceparts.com/search.aspx?find=W10130694";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/text"));
curl_setopt($ch, CURLOPT_USERAGENT, array("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8"));
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Length: 0'));
curl_setopt($ch, CURLOPT_POST, 1);
$result = curl_exec($ch);
var_dump($result);