Question

我有一个页面，我想要所有链接（例如 http://www.stephenfry.com/）。我想将http://www.stephenfry.com/WHATEVER形式的所有链接放入一个数组中。我现在得到的只是以下方法：

#!/usr/bin/perl -w
use strict;
use LWP::Simple;
use HTML::Tree;

# I ONLY WANT TO USE JUST THESE

my $url = 'http://www.stephenfry.com/';

my $doc = get( $url );

my $adt = HTML::Tree->new();
$adt->parse( $doc );

my @juice = $adt->look_down(
    _tag => 'a',
    href => 'REGEX?'
);

不确定如何只放入这些链接。

Answer 1

您需要使用extract_links()方法，而不是look_down()：

use strict;
use warnings;
use LWP::Simple;
use HTML::Tree;

my %seen;
my $url = 'http://www.stephenfry.com/';
my $doc = get($url);

my $adt = HTML::Tree->new();
$adt->parse($doc);
my $links_array_ref = $adt->extract_links('a');

my @links = grep { /www.stephenfry.com/ and !$seen{$_}++ } map $_->[0],
  @$links_array_ref;

print "$_\n" for @links;

部分输出：

http://www.stephenfry.com/
http://www.stephenfry.com/blog/
http://www.stephenfry.com/category/blessays/
http://www.stephenfry.com/category/features/
http://www.stephenfry.com/category/general/
...

使用WWW::Mechanize可能更简单，并且确实会返回更多链接：

use strict;
use warnings;
use WWW::Mechanize;

my %seen;
my $mech = WWW::Mechanize->new();
$mech->get('http://www.stephenfry.com/');
my @links = grep { /www.stephenfry.com/ and !$seen{$_}++ } map $_->url,
  $mech->links();

print $_, "\n" for @links;

部分输出：

http://www.stephenfry.com/wp-content/themes/fry/images/favicon.png
http://www.stephenfry.com/wp-content/themes/fry/style.css
http://www.stephenfry.com/wordpress/xmlrpc.php
http://www.stephenfry.com/feed/
http://www.stephenfry.com/comments/feed/
...

希望这有帮助！

提取某种形式的所有链接

1 个答案: