解析HTML以获取两个集合元素之间的元素

Question

我正在尝试解析像this one这样的页面，而我只是想在标题之后得到段落，我想这是引言。

我想要<table class="infobox vcard">和<table id="toc">之间的所有内容（包括段落标记）。使用简单的CSS选择器来获得第一段：

div#bodyContent div#mw-content-text.mw-content-ltr p

并不总是有效，因为有时信息框表中的某些内容会有一个段落。此外，介绍段落的数量也会有所不同。如果某人有比我在这里更好的方法，我也会接受这一点。

-

请求的其他代码，尽可能缩短：

require HTTP::Request;
require LWP::UserAgent;

use LWP::Simple;
use HTML::Query 'Query';

my $pageurl = "http://en.wikipedia.org/wiki/Wayne_Rooney";
my $wikiurl = URI->new($pageurl);
my $wikirequest = HTTP::Request->new(GET => $wikiurl);
my $wikiua = LWP::UserAgent->new;
my $wikiresponse = $wikiua->request($wikirequest);
my $pagetoparse = $wikiresponse->content;

my $q2 = Query(text => $pagetoparse);
my @wikiintro = $q2->query('div#bodyContent div#mw-content-text.mw-content-ltr p')->get_elements();
my $pageintro;
if(@wikiintro) {
    if(index($wikiintro[0]->as_text(), "Appearances (Goals)") != -1){
        $pageintro = $wikiintro[1]->as_text();
    } else {
        $pageintro = $wikiintro[0]->as_text();
    }
} else {
    $pageintro = "unavailable";
}

Answer 1

使用非标准模块HTML::TreeBuilder的一种方式。

script.pl的内容：

#!/usr/bin/env perl

use warnings;
use strict;
use HTML::TreeBuilder;

my (@p);

## Read the web page.
my $root = HTML::TreeBuilder->new_from_url( shift ) or die qq|ERROR: Malformed URL\n|;

## Get the table tag with id = 'toc'.
my $table_toc = $root->look_down(
    id => 'toc'
);

## Get inmediate previous siblings <p> tags.
for my $node ( reverse $table_toc->left ) { 
    if ( $node->tag eq 'p' ) { 
        unshift @p, $node;
    }   
    else {
        last;
    }   
}

## Print the content without the HTML tags.
for my $p ( @p ) { 
    printf qq|%s\n|, $p->as_text;
}

运行它，将url作为唯一参数提供：

perl-5.14.2 script.pl http://en.wikipedia.org/wiki/Wayne_Rooney

以下输出（我希望它接近您的期望）：

Wayne Mark Rooney (born 24 October 1985) is an English footballer who plays as a forward for Premier League club Manchester United and the England national team.
Rooney made his senior international debut in 2003 becoming the youngest player to represent England, until he got beaten by Theo Walcott. He is England's youngest ever goalscorer.[4] He played at UEFA Euro 2004 and scored four goals, briefly becoming the competition's youngest goalscorer. Rooney featured at the 2006 and 2010 World Cups and is widely regarded as his country's best player.[5][6][7][8][9][10] He has won the England Player of the Year award twice, in 2008 and 2009. As of October 2012, he has won 78 international caps and scored 32 goals, making him England's fifth highest goalscorer in history.[11] Along with David Beckham, Rooney is the most red carded player for England, having been sent off twice.[12]
Wide character in printf at script.pl line 25.
Aged nine, Rooney joined the youth team of Everton, for whom he made his professional debut in 2002. He spent two seasons at the Merseyside club, before moving to Manchester United for £25.6 million in the 2004 summer transfer window. The same year, Rooney acquired the nickname "Wazza".[13] Since then, with Rooney in the team, United have won the Premier League four times, the 2007–08 UEFA Champions League and two League Cups. He also holds two runner-up medals from the Champions League and has twice finished second in the Premier League. In April 2012, Rooney scored his 180th goal, making him United's fourth-highest goal-scorer of all time.[14]
Wide character in printf at script.pl line 25.
In 2009–10, Rooney was awarded the PFA Players' Player of the Year and the FWA Footballer of the Year. He has won the Premier League Player of the Month award five times, a record he shares with Steven Gerrard. He came fifth in the vote for the 2011 FIFA Ballon d'Or and was named in the FIFPro World 11 for 2011. Rooney has won the 'Goal of the Season' award by the BBC's Match of the Day poll on three occasions, with his bicycle kick against rivals Manchester City winning the 'Premier League Goal of the 20 Seasons' award.[15] Rooney is the third highest-paid footballer in the world after Lionel Messi and Cristiano Ronaldo, with an annual income of €20.7m (£18m) including sponsorship deals.[16]

---

编辑：为了得到结果，标签也使用printf qq|%s\n|, $p->as_HTML;代替$p->as_text。

Answer 2

HTML::TreeBuilder可能是最好的工具。诀窍是学习并选择它为浏览HTML树提供的许多方法。

这个程序似乎可以满足您的需求。它调用look_down来查找具有给定类的表，该类紧接在您需要的输出之前。在此处，对right的调用将在此表中的所有元素返回到HTML层次结构中的同一级别。循环只是打印这些元素中的每一个，直到遇到带有p以外的标记的元素。

我是使用LWP::UserAgent编写的，但如果您更新HTML::TreeBuilder的副本以便可以使用new_from_url构造函数，则代码会更简洁。

use strict;
use warnings;

use LWP;
use HTML::TreeBuilder;

binmode STDOUT, ':utf8';

my $url = 'http://en.wikipedia.org/wiki/Wayne_Rooney';
my $ua = LWP::UserAgent->new;
my $resp = $ua->get($url);
die $resp->status_line unless $resp->is_success;

my $tree = HTML::TreeBuilder->new_from_content($resp->decoded_content);

my $start = $tree->look_down(_tag => 'table', class => 'infobox vcard');

for ($start->right) {
  last if $_->tag ne 'p';
  print $_->as_trimmed_text. "\n\n";
}

<强>输出

Wayne Mark Rooney (born 24 October 1985) is an English footballer who plays as a forward for Premier League club Manchester United and the England national team.

Rooney made his senior international debut in 2003 becoming the youngest player to represent England, until he got beaten by Theo Walcott. He is England's youngest ever goalscorer.[4] He played at UEFA Euro 2004 and scored four goals, briefly becoming the competition's youngest goalscorer. Rooney featured at the 2006 and 2010 World Cups and is widely regarded as his country's best player.[5][6][7][8][9][10] He has won the England Player of the Year award twice, in 2008 and 2009. As of October 2012, he has won 78 international caps and scored 32 goals, making him England's fifth highest goalscorer in history.[11] Along with David Beckham, Rooney is the most red carded player for England, having been sent off twice.[12]

Aged nine, Rooney joined the youth team of Everton, for whom he made his professional debut in 2002. He spent two seasons at the Merseyside club, before moving to Manchester United for £25.6 million in the 2004 summer transfer window. The same year, Rooney acquired the nickname "Wazza".[13] Since then, with Rooney in the team, United have won the Premier League four times, the 2007–08 UEFA Champions League and two League Cups. He also holds two runner-up medals from the Champions League and has twice finished second in the Premier League. In April 2012, Rooney scored his 180th goal, making him United's fourth-highest goal-scorer of all time.[14]

In 2009–10, Rooney was awarded the PFA Players' Player of the Year and the FWA Footballer of the Year. He has won the Premier League Player of the Month award five times, a record he shares with Steven Gerrard. He came fifth in the vote for the 2011 FIFA Ballon d'Or and was named in the FIFPro World 11 for 2011. Rooney has won the 'Goal of the Season' award by the BBC's Match of the Day poll on three occasions, with his bicycle kick against rivals Manchester City winning the 'Premier League Goal of the 20 Seasons' award.[15] Rooney is the third highest-paid footballer in the world after Lionel Messi and Cristiano Ronaldo, with an annual income of €20.7m (£18m) including sponsorship deals.[16]