Question

我有一个php文件从另一个站点抓取一个xml文件，然后将该信息丢入我的数据库。

我遇到的问题是他们的网站在任何1小时内只允许360次请求，所以我试图在抓取文件的同时对其进行编码以检查标题信息。

我使用

检查页面状态

$requesttest = 'http://www.footballwebpages.co.uk/teams.xml';
if($requesttest == NULL) return false;  
$ch = curl_init($requesttest);  
curl_setopt($ch, CURLOPT_TIMEOUT, 5);  
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);  
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);  
$data = curl_exec($ch);  
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);  
curl_close($ch); 

if($httpcode == 429){
    return 'Try again later, too many requests recieved.';
} else if($httpcode>=200 && $httpcode<300){
    /* run code to grab xml file */
    $comps = array (    0 => 1, /* Premier_League */
                    1 => 2 /* Championship */ 
                    );
    $comps_total = count($comps);
    $comps_no = 0;

    while ($comps_no < $comps_total) {
        $url = 'http://www.footballwebpages.co.uk/teams.xml?comp=' . $comps[$comps_no];
        $full_list = simplexml_load_file($url);
        /* Code for grabbing and storing info from XML */
} else {
    return 'Football Web Pages Offline';
}

目前，它会检查主要的“团队”页面以查看是否已达到请求限制，然后抓取每个xml的竞赛集。问题是，如果在第一次检查时，只有一个请求可用，当它进入下一个阶段时，它将失败。如何在加载xml文件时检查标题信息，而不必调用页面来检查标题，然后调用页面来获取xml文件？

如果标题代码在1次调用中介于200和300之间，则基本上加载xml文件，以免浪费2个请求来获取1 xml页面。

Answer 1

您可以使用类似于以下的方法，忘记对基本URL的第一次调用，因为它是冗余的，而是使用函数的返回值来确定是否应该进行进一步处理：

<?php
    /* utility function to get data and return an object */
    function getxml( $comp=1 ){
        global $ch;
        global $url;

        curl_setopt( $ch, CURLOPT_URL, $url . '?comp=' . $comp );
        $data = curl_exec( $ch );
        $status = curl_getinfo( $ch, CURLINFO_HTTP_CODE ); 

        return (object)array(
            'xmldata'   =>  $data,
            'status'    =>  $status
        );
    }
    /* All the comps available - more than specified! */
    $comps=array( 
        'Barclays_Premier_League' => 1,
        'Sky_Bet_Championship' => 2,
        'Sky_Bet_League_One' => 3,
        'Sky_Bet_League_Two' => 4,
        'National_League' => 5,
        'National_League_North' => 6,
        'National_League_South' => 7,
        'Evo-Stik_Southern_League_Premier_Division' => 8,
        'Evo-Stik_Southern_League_Division_One_Central' => 9,
        'Evo-Stik_Southern_League_Division_One_South_&_West' => 10,
        'Ryman_League_Premier_Division' => 11,
        'Ryman_League_Division_One_North' => 12,
        'Ryman_League_Division_One_South' => 13,
        'Evo-Stik_League_Premier_Division' => 14,
        'Evo-Stik_League_Division_One_North' => 15,
        'Evo-Stik_League_Division_One_South' => 16,
        'Scottish_Premiership' => 17,
        'Scottish_Championship' => 18,
        'Scottish_League_One' => 19,
        'Scottish_League_Two' => 20
    );
    /* only interested in first two */
    $comps=array_slice( $comps, 0, 2, true );


    /* I don't use simple_xml() - used to process xml data */
    $dom=new DOMDocument;

    /* base url */
    $url= 'http://www.footballwebpages.co.uk/teams.xml';

    /* 
        initialise curl request object but 
        set the url for each $comp in the function 
    */
    $ch = curl_init();
    curl_setopt( $ch, CURLOPT_TIMEOUT, 5 );  
    curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, 5 );  
    curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );   

    /* 
    If there have been too many requests when launching 
    the 429 condition should break out of the entire loop -
    thus using only 1 request
    */
    foreach( $comps as $key => $comp ){
        $xml=getxml( $comp );
        switch( $xml->status ){
            case 429: echo 'Try again later, too many requests recieved.'; break 2;
            case 200:
                /* if everything is ok, process $xml */
                $dom->loadXML( $xml->xmldata );


                /* example of processing xml data */
                echo '
                <h1>'.$dom->getElementsByTagName('competition')->item(0)->nodeValue.'</h1>
                    <ul>';

                $col=$dom->getElementsByTagName('team');
                if( $col ){
                    foreach( $col as $team ) echo '<li>'.$team->childNodes->item(1)->nodeValue.', '.$team->childNodes->item(3)->nodeValue.'</li>';
                }
                echo '
                    </ul>';
            break;
            default:/* If no response or an unknown response exit */
                echo 'Football Web Pages Offline';
            break 2;
        }
    }

    curl_close( $ch ); 
    $dom=$ch=$comps=null;
?>

加载外部XML文件并在1次调用中获取html标头信息

1 个答案: