Question

我正在尝试抓取一个网页并从中解析一些数据。但每次我试图刮，我只得到 http响应标题。这是我用来从网站上获取数据的代码..

$host = 'Host: dealnews.com';
$user_agent = 'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0';
$accept = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
$accept_language = 'Accept-Language: en-US,en;q=0.5';
$accept_encoding = 'Accept-Encoding: gzip, deflate';
$connection = 'Connection: keep-alive';
$cookie = 'Cookie=front_page_sort=hotness; dnvta=%7B%22uid%22%3A%22VkA1VlBBb0tNcXdBQVF6UlJrTUFBQUJN%22%2C%22vid%22%3A%22VkA1bGx3b0tNcXdBQVF6bW53QUFBQUEt%22%2C%22fvts%22%3A1475237180%2C%22lvts%22%3A1475241453%2C%22ref%22%3A%22%22%2C%22usid%22%3A0%2C%22ct%22%3A2%2C%22cr%22%3A1475237180%7D; last_visit=1475241457; _ceg.s=oebjle; _ceg.u=oebjle; _ga=GA1.2.185245695.1475237222; __gads=ID=1921ec3c3fe54b1b:T=1475237222:S=ALNI_MZJZEuNpmg3Aq5e007E7iFjwuQ0nw; original_eref=DIRECT; _gat=1; mp_dealnews_mixpanel=%7B%22distinct_id%22%3A%20%221577afe52c549-01b1cfdcc8ca548-13666c4a-100200-1577afe52c620c%22%7D';

$requestHeaders = array ( $host, $user_agent, $accept, $accept_encoding, $accept_language, $connection, $cookie );

$ch = curl_init ( 'http://dealnews.com/2-LED-Window-Candles-w-Color-Changing-Bulbs-for-4-2-s-h/1797165.html?iref=rss-dealnews-todays-edition' );
curl_setopt ( $ch, CURLOPT_TIMEOUT, 100 );
curl_setopt ( $ch, CURLOPT_CONNECTTIMEOUT, 100 );
curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt ( $ch, CURLOPT_SSL_VERIFYHOST, false );
curl_setopt ( $ch, CURLOPT_SSL_VERIFYPEER, false );
curl_setopt ( $ch, CURLOPT_HEADER, TRUE );
curl_setopt ( $ch, CURLOPT_ENCODING, "gzip" );
curl_setopt ( $ch, CURLOPT_HTTPHEADER, $requestHeaders );
$data = curl_exec ( $ch );
if (! $data) {
    die ( "Error: " . curl_error ( $ch ) . " Error no: " . curl_errno ( $ch ) );
}
curl_close ( $ch );
$htmlContent = str_get_html ( $data );
echo $htmlContent;

但它给了我下面给出的错误..

HTTP/1.1 302 Found Date: Fri, 30 Sep 2016 13:50:44 GMT Server: Apache X-Powered-By: PHP/5.5.9-1ubuntu4.19 Status: 302 Found Location: /lw/landing.html?uri=%2F2-LED-Window-Candles-w-Color-Changing-Bulbs-for-4-2-s-h%2F1797165.html%3Firef%3Drss-dealnews-todays-edition Content-Encoding: gzip Vary: Accept-Encoding Content-Length: 20 X-Cnection: close Content-Type: text/html; charset=utf-8

所以有人可以帮助我解决我在这个错误的地方

Answer 1

你需要

curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);

标题302是重定向信息。

Answer 2

如果您正在使用PHP查看ScreenScrape，我已使用PHP Simple HTML DOM Parser libarary成功完成了它。它非常简单易用。我知道该网站看起来有点陈旧，但我去年的代码仍然运行良好。还没有CRON错误。

如何使用PHP抓取网页？

2 个答案: