PHP cURL站点地图生成器

Question

大家好我所有我的站点地图生成器有问题它在几千个URL上工作得很好但是当它到达20000 URL时它开始出错了我还有什么可以做的来帮助防止这个

$request_url ="http://www.example.com";//put your url here
$url=str_replace("http://", "", $request_url); // i added this because to check if wwwdomain.com is in the url
$alllinks=array();// create an array for all the links
         $alllinks2=array();// create an array for all the links
          $alllinks3=array();// create an array for all the links
            $newUrl=array();// create an array for all the links
$badUrls=array("basket", "#" ,"mailto", "javascript:document", "reviews.php", "review.php","tab=",".JPG",".jpg","png","PNG","gif","GIF","item_id=","../", "pn_pr");// array of urls we dont want to include
$ch = curl_init(); //search for links
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL, $request_url); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
curl_setopt ($ch,CURLOPT_VERBOSE,false);
curl_setopt($ch, CURLOPT_TIMEOUT, 900);
$result = curl_exec($ch);

谢谢