使用多重卷曲获取所有网址

时间：2019-01-02 08:49:47

标签： php curl curl-multi

我正在开发一个应用程序，该应用程序从一系列站点中获取所有URL，并以数组形式或JSON显示它。

我可以使用for循环来做到这一点，问题是我尝试10个URL时的执行时间，这给我一个错误，提示exceeds maximum execution time。

搜索后，我发现了这个multi curl

我也找到了这个Fast PHP CURL Multiple Requests: Retrieve the content of multiple URLs using CURL。我试图添加我的代码，但是由于我不知道如何使用该函数而无法正常工作。

希望你能帮助我。

谢谢。

这是我的示例代码。

<?php

$urls=array(
'http://site1.com/',
'http://site2.com/',
'http://site3.com/');


$mh = curl_multi_init();
foreach ($urls as $i => $url) {

        $urlContent = file_get_contents($url);

        $dom = new DOMDocument();
        @$dom->loadHTML($urlContent);
        $xpath = new DOMXPath($dom);
        $hrefs = $xpath->evaluate("/html/body//a");

        for($i = 0; $i < $hrefs->length; $i++){
            $href = $hrefs->item($i);
            $url = $href->getAttribute('href');
            $url = filter_var($url, FILTER_SANITIZE_URL);
            // validate url
            if(!filter_var($url, FILTER_VALIDATE_URL) === false){
                echo '<a href="'.$url.'">'.$url.'</a><br />';
            }
        }

        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

6 个答案:

答案 0 :(得分：2)

这里是一个我整合在一起的函数，可以正确利用curl_multi_init()函数。它与PHP.net几乎具有相同的功能，但有一些细微的调整。我在这方面取得了巨大的成功。

function multi_thread_curl($urlArray, $optionArray, $nThreads) {

  //Group your urls into groups/threads.
  $curlArray = array_chunk($urlArray, $nThreads, $preserve_keys = true);

  //Iterate through each batch of urls.
  $ch = 'ch_';
  foreach($curlArray as $threads) {      

      //Create your cURL resources.
      foreach($threads as $thread=>$value) {

      ${$ch . $thread} = curl_init();

        curl_setopt_array(${$ch . $thread}, $optionArray); //Set your main curl options.
        curl_setopt(${$ch . $thread}, CURLOPT_URL, $value); //Set url.

        }

      //Create the multiple cURL handler.
      $mh = curl_multi_init();

      //Add the handles.
      foreach($threads as $thread=>$value) {

      curl_multi_add_handle($mh, ${$ch . $thread});

      }

      $active = null;

      //execute the handles.
      do {

      $mrc = curl_multi_exec($mh, $active);

      } while ($mrc == CURLM_CALL_MULTI_PERFORM);

      while ($active && $mrc == CURLM_OK) {

          if (curl_multi_select($mh) != -1) {
              do {

                  $mrc = curl_multi_exec($mh, $active);

              } while ($mrc == CURLM_CALL_MULTI_PERFORM);
          }

      }

      //Get your data and close the handles.
      foreach($threads as $thread=>$value) {

      $results[$thread] = curl_multi_getcontent(${$ch . $thread});

      curl_multi_remove_handle($mh, ${$ch . $thread});

      }

      //Close the multi handle exec.
      curl_multi_close($mh);

  }


  return $results;

} 



//Add whatever options here. The CURLOPT_URL is left out intentionally.
//It will be added in later from the url array.
$optionArray = array(

  CURLOPT_USERAGENT        => 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0',//Pick your user agent.
  CURLOPT_RETURNTRANSFER   => TRUE,
  CURLOPT_TIMEOUT          => 10

);

//Create an array of your urls.
$urlArray = array(

    'http://site1.com/',
    'http://site2.com/',
    'http://site3.com/'

);

//Play around with this number and see what works best.
//This is how many urls it will try to do at one time.
$nThreads = 20;

//To use run the function.
$results = multi_thread_curl($urlArray, $optionArray, $nThreads);

完成后，您将拥有一个数组，其中包含网站列表中的所有html。在这一点上，我将遍历它们并提取所有网址。

像这样：

foreach($results as $page){

  $dom = new DOMDocument();
  @$dom->loadHTML($page);
  $xpath = new DOMXPath($dom);
  $hrefs = $xpath->evaluate("/html/body//a");

  for($i = 0; $i < $hrefs->length; $i++){

    $href = $hrefs->item($i);
    $url = $href->getAttribute('href');
    $url = filter_var($url, FILTER_SANITIZE_URL);
    // validate url
    if(!filter_var($url, FILTER_VALIDATE_URL) === false){
    echo '<a href="'.$url.'">'.$url.'</a><br />';
    }

  }

}

值得一提的是，可以增加脚本的运行时间。

如果您使用托管服务，则无论您将最大执行时间设置为多少，都可能只能在两分钟内完成某些事情。值得深思。

这是通过以下方式完成的：

ini_set('max_execution_time', 120);

您总是可以尝试更多的时间，但是直到您确定时间，您才知道。

希望有帮助。

答案 1 :(得分：0)

您可能正在使用无限循环-否则，您可以在php.ini中或通过以下方式增加最长执行时间：

ini_set('max_execution_time', 600); // 600 seconds = 10 minutes

答案 2 :(得分：0)

首先，我知道OP确实询问了multi_curl，但我只是添加了另一种选择，如果OP可能改变主意。我在这里所做的是将网址分为许多请求，因此cpu的使用不会那么大。如果OP仍要使用multi_curl，那么这里的PHP大师可以提供更好的解决方案。

<?php
$num = preg_replace('/[^0-9]/','',$_GET['num']);
$num = empty($num) ? 0 : $num;

$urls=array(
'http://site1.com/',
'http://site2.com/',
'http://site3.com/');

if(!empty($urls[$num]))
{
    /* do your single curl stuff here and store its data here*/

    /*now redirect to the next url. dont use header location redirect, it would ends up too many redirect error in browser*/
    $next_url = !empty($urls[$num+1]) ? $urls[$num+1] : 'done';

    echo '<html>
    <head>
    <meta http-equiv="refresh" content="0;url=http://yourcodedomain.com/yourpath/yourcode.php?num='.$next_url.'" />
    </head>
    <body>
    <p>Fetching: '.$num+1.' / '.count($urls).'</p>
    </body> 
    </html>';
}
elseif($_GET['num'] == 'done')
{
    /*if all sites have been fetched, do something here*/
}
else
{
    /*throws exception here*/
}

?>

答案 3 :(得分：0)

我遇到了同样的问题，然后我使用 usleep（）解决了这个尝试，让我知道了

do {
    usleep(10000);
    $n=curl_multi_exec($mh,$active);
}

答案 4 :(得分：-1)

尝试以下简化版本：

$urls = [
    'https://en.wikipedia.org/',
    'https://secure.php.net/',
];

set_time_limit(0);
libxml_use_internal_errors(true);

$hrefs = [];
foreach ($urls as $url) {
    $html = file_get_contents($url);
    $doc = new DOMDocument;
    $doc->loadHTML($html);

    foreach ($doc->getElementsByTagName('a') as $link) {
        $href = filter_var($link->getAttribute('href'), FILTER_SANITIZE_URL);
        if (filter_var($href, FILTER_VALIDATE_URL)) {
            echo "<a href='{$href}'>{$href}</a><br/>\n";
        }
    }
}

答案 5 :(得分：-1)

这是我在处理代码后获得的结果，它可以工作，但不确定这是否是最佳答案。请检查我的代码。

<?php

$array = array('https://www.google.com/','https://www.google.com/','https://www.google.com/','https://www.google.com/','https://www.google.com/','https://www.google.com/','https://www.google.com/','https://www.google.com/','https://www.google.com/','https://www.google.com/');

print_r (getUrls($array));

function getUrls($array) { 

  $arrUrl = array();
  $arrList = array();
  $url_count = count($array);
  $curl_array = array();
  $ch = curl_multi_init();

  foreach($array as $count => $url) {
      $curl_array[$count] = curl_init($url);
      curl_setopt($curl_array[$count], CURLOPT_RETURNTRANSFER, true);
      curl_multi_add_handle($ch, $curl_array[$count]);
  }


  do{
    curl_multi_exec($ch, $exec);
    curl_multi_select($ch,1);
  }while($exec);


  foreach($array as $count => $url) {


      $arrUrl = array();

      $urlContent = curl_multi_getcontent($curl_array[$count]);

      $dom = new DOMDocument();
        @$dom->loadHTML($urlContent);
        $xpath = new DOMXPath($dom);
        $hrefs = $xpath->evaluate("/html/body//a");

        for($i = 0; $i < $hrefs->length; $i++){
            $href = $hrefs->item($i);
            $url = $href->getAttribute('href');
            $url = filter_var($url, FILTER_SANITIZE_URL);
            // validate url
            if (filter_var($url, FILTER_VALIDATE_URL) !== false) {

              if (strpos($url, 'mailto') === false) {


                      $arrUrl[] = $url;

              }
          }
        }

        array_push($arrList, array_unique($arrUrl));


  }

  foreach($array as $count => $url) {
      curl_multi_remove_handle($ch, $curl_array[$count]);
  }

  curl_multi_close($ch); 

  foreach($array as $count => $url) {
      curl_close($curl_array[$count]);
  }

  return $arrList;

}
相关问题
最新问题