Question

我可以使用此脚本成功抓取this page上的所有项目：

$html = file_get_contents($list_url);
$doc = new DOMDocument();
libxml_use_internal_errors(TRUE);

    if(!empty($html))
    {
        $doc->loadHTML($html);
        libxml_clear_errors(); // remove errors for yucky html
        $xpath = new DOMXPath($doc);

        /* FIND LINK TO PRODUCT PAGE */

        $products = array();

        $row = $xpath->query($product_location);

        /* Create an array containing products */
        if ($row->length > 0)
        {            
            foreach ($row as $location)
            {
                $product_urls[] = $product_url_root . $location->getAttribute('href');
            }
        }
        else { echo "product location is wrong<br>";}

        $imgs = $xpath->query($photo_location);

        /* Create an array containing the image links */
        if ($imgs->length > 0)
        {            
            foreach ($imgs as $img)
            {
                $photo_url[] = $photo_url_root . $img->getAttribute('src');
            }
        }
        else { echo "photo location is wrong<br>";}

        $was = $xpath->query($was_price_location);

        /* Create an array containing the was price */
        if ($was->length > 0)
        {
            foreach ($was as $price)
            {
                $stripped = preg_replace("/[^0-9,.]/", "", $price->nodeValue);
                $was_price[] = "&pound;".$stripped;
            }
        }
        else { echo "was price location is wrong<br>";}

        $now = $xpath->query($now_price_location);

        /* Create an array containing the sale price */
        if ($now->length > 0)
        {
            foreach ($now as $price)
            {
                $stripped = preg_replace("/[^0-9,.]/", "", $price->nodeValue);
                $stripped = number_format((float)$stripped, 2, '.', '');
                $now_price[] = "&pound;".$stripped;
            }
        }
        else { echo "now price location is wrong<br>";}

        $result = array();

        /* Create an associative array containing all the above values */
        foreach ($product_urls as $i => $product_url)
        {
            $result[] = array(
                'product_url' => $product_url,
                'shop_name' => $shop_name,
                'photo_url' => $photo_url[$i],
                'was_price' => $was_price[$i],
                'now_price' => $now_price[$i]
            );
        }
    }

但是，如果我想要获得第二页，或者如果我每页查看100个file_get_contents($list_url)将始终返回第一页及其24个值，则会出现问题。

我认为页面更改是通过AJAX请求处理的（虽然我无法在源代码中找到任何证据）。有没有办法可以清楚地抓住我在屏幕上看到的内容？

我已经在之前的答案中看过PhantomJS的谈话，但鉴于我在PHP工作，我不确定它是否合适。

Answer 1

// Create DOM from URL or file
$file= file_get_html('http://stackoverflow.com/');

// Find your links
foreach($file->find('a') as $youreEement) {
       echo $yourElement->href . '<br>';
}

Answer 2

这是因为某些js脚本生成的链接中的＃标签。关闭该网站的javascript并检查它生成的输出链接。

例如，对于第二页，它是http://www.hm.com/gb/subdepartment/sale?page=1

用PHP屏幕抓取JavaScript

2 个答案: