使用php从网站获取信息,递归遍历每个HTML节点

时间:2013-04-28 04:49:04

标签: php web-crawler simple-html-dom dom

我做了很多研究,但我没有找到答案。我试图从网页获取一些信息,其中包含以下HTML结构

<div id="xxx" class="some1">
        <h1>This is the time</h1>
    <div class="ti12">
        <div class="sss"></div>
        <div class="sss">
                     <span class="hhh">
            <div class="sded">
                City:
                <span class="sh">CCC</span>
            </div>
                    </span>
        </div>
        </div>
        .
        .
    .
        <div class="pp12"></div>
</div>

现在,我正在做的是以同样的方式获取城市的名称和类似的其他信息。

我必须从上面的代码中找到这些信息。

$arr=array('City', 'Name', 'Address', 'DOB');

如果存在获取其值,则将其留空。

希望我很清楚。

以下代码尝试了:

<?php
include "simple_html_dom.php";
   $html = new simple_html_dom();
   $listItem = array('City', 'Name', 'Address', 'DOB');
$html->load_file('simp.html');
$found=array();
    foreach($listItem as $item){
        $ret = $html->find('div[id=xxx] div',0);
        iterateParentNode($ret, $item);
        }
function iterateParentNode($ret1, $item1){
for ($node=0;$node < count($ret1->children());$node++){
    $child=$ret1->children($node);
    echo count($ret1->children())."<br/>";
    if(count($ret1->children())==1 && strpos($child, '<span class="sh"')!==false ){
        $found[$item1]=$ret1->find('span[class=sh]',0)->plaintext;
        return true;
    }else{
    goThroughChildNode($child, $item1);
    }
}
}
function goThroughChildNode($child1, $item2){
    echo $child1."ITEM:".$item2;
        if(strpos($child1, $item2)!==false){
            iterateParentNode($child1, $item2);
        }else{
            return false ;
        }
        return true;
}
foreach ($found as $structure=>$data){
  echo $structure."=>".$data."<br />";
}
?>

我知道我的PHP方法不好,所以请考虑我的PHP代码,建议我这样做的好方法。

3 个答案:

答案 0 :(得分:0)

使用正则表达式执行此操作可能最简单。当然,如果HTML结构发生变化,它将会中断。

if (ereg('<div.*?h1>(.*?)</h1>.*?City:.*?>(.*?)<', $input, $regs)) {
    $title = $regs[1];
    $city = $regs[2];
} else {
    $title = "";
    $city = "";
}

/*
Match 1 of 1
Matched text: <div id="xxx" class="some1">
        <h1>This is the time</h1>
    <div class="ti12">
        <div class="sss"></div>
        <div class="sss">
                     <span class="hhh">
            <div class="sded">
                City:
                <span class="sh">CCC<
Match offset: 0
Match length: 282
Group 1: This is the time
Group 1 offset: 42
Group 1 length: 16
Group 2: CCC
Group 2 offset: 278
Group 2 length: 3
*/

// <div.*?h1>(.*?)</h1>.*?City:.*?>(.*?)<
// 
// Match the characters "<div" literally «<div»
// Match any single character «.*?»
//    Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the characters "h1>" literally «h1>»
// Match the regular expression below and capture its match into backreference number 1 «(.*?)»
//    Match any single character «.*?»
//       Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the characters "</h1>" literally «</h1>»
// Match any single character «.*?»
//    Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the characters "City:" literally «City:»
// Match any single character «.*?»
//    Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the character ">" literally «>»
// Match the regular expression below and capture its match into backreference number 2 «(.*?)»
//    Match any single character «.*?»
//       Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the character "<" literally «<»

答案 1 :(得分:0)

手动遍历的一种替代方法是查询数据。在DOMDocument中,这通常是使用XPath完成的,XPath是一种专门用于完成该工作的语言。

您使用的库不支持XPath,但PHP确实支持它。 PHP也支持DOMDocument开箱即用,所以我认为我可以安全地建议你这样做。

因此,在您的情况下,您首先要查看带有ID的div:

//div[@id="xxx"]

然后在某个地方的div里面:

   //div

如果没有特定的名字(孩子),那么你想要另一个元素:

       //*

但是那些需要匹配一个特定的模式:这里,包含一个带有“sh”类属性的span,它必须是那里的第一个跨度,在跨度之前必须有一些文本:

            [
                span[@class="sh"]
                and span = span[@class="sh"]
                and span/preceding-sibling::text()
            ]

和那个你想要第一个文本节点子节点的孩子:

                /text()[1]

所以只是一目了然地看到这一点:

    //div[@id="xxx"]
        //div
            //*[
                span[@class="sh"]
                and span = span[@class="sh"]
                and span/preceding-sibling::text()
            ]
                /text()[1]

这将为您提供命名的字符串,如“City:”等。然后,下一个兄弟(span)将包含值。

您需要做的就是将其包装到代码中(这里我加载一个字符串,但您也可以使用 loadHTMLFile() 加载HTML文件,检查上面的DOMDocument链接所有的荣耀细节):

$dom = new DOMDocument();
$dom->loadHTML($string);
$xp = new DOMXPath($dom);

foreach ($xp->query('
        //div[@id="xxx"]
            //div
                //*[
                    span[@class="sh"]
                    and span = span[@class="sh"]
                    and span/preceding-sibling::text()
                ]
                    /text()[1]
                '
         ) as $node
) {
    $name  = trim($node->nodeValue);
    $value = trim($node->nextSibling->nodeValue);
    printf("%s %s\n", $name, $value);
}

示例HTML的输出:

City: CCC

我希望这可以激励您研究DOMDocument并帮助您探索XPath的强大功能。

答案 2 :(得分:0)

我花了一段时间才能做到正确,但是这段代码使用Simple HTML Dom遍历整个DOM。希望有人可以使用它。

<?php
$html = new simple_html_dom();
$html->load('<html><body>'.$text.'</body></html>');

if(method_exists($html,"childNodes")){
    if($html->find('html')) {
    //IF NOT OK, THROW ERROR
}}

$e=$html->find('body',0);
$p=$e->childNodes(0);
if(!$p){
    //BODY HAS NO CHILDNODES< THROW ERROR
    }

    $loop=true;
    //SAFEGUARD, PREVENTS INDEFINITE LOOPS
    $i=$j=0; 
    $i_max=500;
    $j_max=500;


    while($loop==true){
        //SAFEGUARD, PREVENTS INDEFINITE LOOPS
        $i=0;$i++;if($i>$i_max){$loop=false;break;}

        //TEST IF NODE HAS CHILDREN
        $p=$e->childNodes(0);

        //NO CHILDREN
        if(!$p){
            //DO SOMETHING WITH NODE
            clean_dom($e->outertext);

            //TEST IF NODE HAS SIBLING
            $p=$e->next_sibling();
            if(!$p){
                //NO SIBLING
                    //TEST THE PARENT, LOOP TILL WE FIND A SIBLING
                    $j=0;$sib_loop=true;
                    while($sib_loop==true){
                        //SAFEGUARD, PREVENTS INDEFINITE LOOPS
                        $j++;if($j>$j_max){$sib_loop=false;break;}
                        //TEST IF THERE IS A PARENT
                        $e=$e->parent();
                            //NO PARENT, WE'VE REACHED THE TOP AGAIN
                            if(!$e){
                                echo'***THE END***';
                                $sib_loop=$loop=false;break;}
                            //ELSE, TEST IF PARENT HAS SIBLING
                            $p=$e->next_sibling();
                            //THERE IS A SIBBLING, GO THERE
                            if($p){
                                //DO SOMETHING WITH THIS NODE
                                clean_dom($e->outertext);
                                $e=$e->next_sibling();
                                $sib_loop=false;break;
                                }
                            else{
                                $ret=clean_dom($e->outertext,$all);
                                $e->outertext=$ret;
                                }

                        }       
                    }
            else{
                //GOTO SIBLING
                $e=$e->next_sibling();
                }
            }

        else{
        //THERE IS A CHILD
            $e=$e->childNodes(0);
            }
        }
$text=$html->save();
$html->clear(); 
unset($html);

function clean_dom($e){
    //DO SOMETHING HERE        
    }