无法获取特定网址的元标记

时间:2018-02-15 09:50:09

标签: php web-scraping keyword meta-tags simple-html-dom

我正在使用php脚本从特定网站的元标记中获取关键字。但对于某些网址,它无效,当我手动检查该网址的关键字时,我发现网页中存在关键字。

$url = "https://www.washingtonpost.com/news/education/wp/2018/02/14/school-shooting-reported-at-florida-high-school/?tid=pm_pop";
    get_meta_tags($url);

它总是给我警告: -

警告:get_meta_tags(https://www.washingtonpost.com/politics/stormy-danielss-tale-gains-renewed-momentum-with-trump-lawyers-claim-which-raises-new-questions/2018/02/14/e7ce4a16-119d-11e8-9065-e55346f6de81_story.html?tid=pm_pop):无法打开流:已达到重定向限制

任何想法?

1 个答案:

答案 0 :(得分:2)

让我们去

  • 首先 infinty 重定向循环,所以服务器只有在您启用时才会显示该页面 cookies 。 所以我们将使用 curl函数来获取html页面,步骤为2步:

    1. 获取Cookie
    2. 重新发送Cookie并获取页面
  • 第二次:使用 preg_match 解析html以获取元标记:

  • 最后代码将

    https://www.washingtonpost.com/news/education/wp/2018/02/14/school-shooting-reported-at-florida-high-school/?tid=pm_pop');
    //parsing begins here:
    preg_match_all('/<[\s]meta[\s](name|property)="?' . '([^>"])"?[\s]' . 'content="?([^>"])"?[\s][/]?[\s]*>/si', $html, $match);
    $count = count($match[2]);
    for($i = 0; $i < $count; $i++){
        echo($match[2][$i]." : ".$match[3][$i]."
    "); }

    function get_contents($link) { $result =""; try{ $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $link); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); curl_setopt ($ch, CURLOPT_COOKIEJAR, "-"); // <-- see here $result = curl_exec($ch); // remember i didn't close the curl yet!
    // Now make another curl request with the same handle: curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); $result = curl_exec($ch); // if you are done, you can close it. $result = curl_exec($ch); $httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE); $curlerr = curl_error($ch); curl_close($ch); } catch (Exception $e) { $result = "Error1 :". $result."||".$e; } if(strlen($result) < 5){$result = $result."Error :".$httpcode.$curlerr;}
    return $result; } ?>

注意: html无法通过domdocument进行解析

输出

object-hash : 1518960831
referrer : unsafe-url
keywords : Florida school shooting, Marjory Stoneman Douglas High School, Parkland school shooting, Florida shooting, Broward County
news_keywords : Florida school shooting, Marjory Stoneman Douglas High School, Parkland school shooting, Florida shooting, Broward County
twitter:card : summary_large_image
og:type : article
og:site_name : Washington Post
magnet : floridashooting
article:publisher : https://www.facebook.com/washingtonpost
fb:app_id : 41245586762
fb:admins : 4403963
fb:admins : 500835072
article:content_tier : metered
og:url : https://www.washingtonpost.com/news/education/wp/2018/02/14/school-shooting-reported-at-florida-high-school/
og:title : ‘A horrific, horrific day’: At least 17 killed in Florida school shooting
og:description : The suspect, a student who had been expelled, was armed with an AR-15, authorities said.
robots : index,follow
theme : normal
audio_url : 
twitter:creator : @lori_rozsa
article:author : https://www.facebook.com/moriah.balingit
author : https://www.facebook.com/moriah.balingit
twitter:creator : @ByMoriah
twitter:creator : @thewanreport
article:author : https://www.facebook.com/markberman
author : https://www.facebook.com/markberman
twitter:creator : @markberman