Question

制作此脚本以抓取论坛上的某些链接并提取用户名，发布日期和帖子编号。

效果很好，唯一的问题是它会占用内存，大约半小时后它会显着减慢速度。

有人有建议加快速度吗？我一直在我的服务器上运行WGET来启动脚本。

谢谢，尼克

   <?
//this php script is going to download pages and tear them apart from ###

/*
Here's the process:

1. prepare url 
2. get new HTML document from the web
3. extract xpath data
4. input in mysql database
*/


$baseURL="http://www.###.com";

//end viewtopic.php?p=357850
for ($post = 325479; $post <= 357850; $post++) {

//connect to mysql
if (!mysql_connect('localhost','###','###')) echo mysql_error;
mysql_select_db('###');

//check to see if the post is already indexed
$result = mysql_query("SELECT postnumber FROM ### WHERE postnumber = '$post'");
if (mysql_num_rows($result) > 0) {
    //echo "Already in the database." . "<br>";
    mysql_close();
    continue;
}

$url=$baseURL."/viewtopic.php?p=".$post;
//echo $url."<br>";

//get new HTML document
$html = new DOMDocument(); 
$html->loadHTMLFile($url);

$xpath = new DOMXpath($html);

//select the page elements that you want
//I want the parent of the TD class = forumRow
$links = $xpath->query( "//td[@class='forumRow']/parent::tr" ); 

    foreach($links as $results){
        $newDom = new DOMDocument;
        $newDom->appendChild($newDom->importNode($results,true));

        $xpath = new DOMXpath ($newDom);

        //which parts of the selection do you want?
        $time_stamp = trim($xpath->query("//td[2]/table/tr/td/span")->item(0)->nodeValue);
        $user_name = trim($xpath->query("//a[@class='genmed']")->item(0)->nodeValue);
        $post_number = trim($xpath->query("//td/a/@name")->item(0)->nodeValue);

        $return[] = array(
            'time_stamp' => $time_stamp,
            'username' => $user_name,
            'post_number' => $post_number,
            );
    }

    foreach ($return as $output) {
        if (strlen($output['time_stamp']) > 0 && strlen($output['username']) > 0) 
          {
          //$timestamp = substr($output['time_stamp'],8,25);
          //echo $timestamp . "<br>";
          //$unixtimestamp = strtotime($timestamp);
          //echo $unixtimestamp;
          //echo $output['time_stamp']."<br>";
          preg_match("/[Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec]{3} \d{1,2}[,] \d{4} \d{1,2}[:]\d{2}/", $output['time_stamp'],$matches). "<br>";
          $unixtimestamp = strtotime($matches[0]);

          //YYYY-MM-DD HH:MM:SS
          $phpdate=date("Y-m-d H:i:s",$unixtimestamp);
          $username=$output['username'];
          $post_number=$output['post_number'];
          //echo $phpdate ." by ". $username . " #" . $post_number ;

          $result = mysql_query("SELECT postnumber FROM ### WHERE postnumber = '$post_number'");
          if (mysql_num_rows($result) == 0) {         
            if (mysql_query("INSERT INTO ### VALUES('','$url','$username','$phpdate','$post_number')")) echo "Y";
            else echo "N";
            mysql_close();
          }
          echo "<br>";
          }
    }
}
?>

Answer 1

您可能需要查看mysql_free_result。此外，您通过整个脚本维护$ return数组的事实没有帮助。如果你想避免内存问题，你应该抓取十几条记录，插入它们，重置$ return，再抓一打，插入，重置......等等。否则，$ return数组变得很大，这可能是你问题的原因之一（如果不是原因）。

保持我的DOMDocument / DOMXpath PHP脚本不会占用内存？

1 个答案: