Question

以下是我在网页上使用的正则表达式匹配代码：

public class RegexTestHarness {

    public static void main(String[] args) {

        File aFile = new File("/home/darshan/Desktop/test.txt");
        FileInputStream inFile = null;
        try {
            inFile = new FileInputStream(aFile);
        } catch (FileNotFoundException e) {
            e.printStackTrace(System.err);
            System.exit(1);
        }

        BufferedInputStream in = new BufferedInputStream(inFile);
        DataInputStream data = new DataInputStream(in);
        String string = new String();
        try {
            while (data.read() != -1) {
                string += data.readLine();
            }

        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        Pattern pattern = Pattern
                .compile("<div class=\"rest_title\">.*?<h1>(.*?)</h1>");
        Matcher matcher = pattern.matcher(string);
        boolean found = false;
        while (matcher.find()) {
            System.out.println("Name: " + matcher.group(1) );
            found = true;
        }
        if(!found){
            System.out.println("Pattern Not found");
        }
    }
}

但是相同的代码对我正在测试正则表达式的crwaler代码不起作用，我的爬虫代码是:(我正在使用Websphinx）

// Our own Crawler class extends the WebSphinx Crawler
public class MyCrawler extends Crawler {

    MyCrawler() {
        super(); // Do what the parent crawler would do
    }

    // We could choose not to visit a link based on certain circumstances
    // For now we always visit the link
    public boolean shouldVisit(Link l) {
        // String host = l.getHost();
        return false; // always visit a link
    }

    // What to do when we visit the page
    public void visit(Page page) {
        System.out.println("Visiting: " + page.getTitle());
        String content = page.getContent();

        System.out.println(content);

        Pattern pattern = Pattern.compile("<div class=\"rest_title\">.*?<h1>(.*?)</h1>");
        Matcher matcher = pattern.matcher(content);
        boolean found = false;
        while (matcher.find()) {
            System.out.println("Name: " + matcher.group(1) );
            found = true;
        }
        if(!found){
            System.out.println("Pattern Not found");
        }
    }
}

这是我运行抓取工具的代码：

public class WebSphinxTest {

    public static void main(String[] args) throws MalformedURLException, InterruptedException {

        System.out.println("Testing Websphinx. . .");

        // Make an instance of own our crawler
        Crawler crawler = new MyCrawler();
        // Create a "Link" object and set it as the crawler's root
        Link link = new Link("http://justeat.in/restaurant/spices/5633/indian-tandoor-chinese-and-seafood/sarjapur-road/bangalore");
        crawler.setRoot(link);

        // Start running the crawler!
        System.out.println("Starting crawler. . .");
        crawler.run(); // Blocking function, could implement a thread, etc.

    }

}

有关抓取工具代码的一些细节。 shouldvisit(Link link)过滤是否访问链接。 visit(Page page)决定在我们到达页面时该怎么做。

在上面的示例中，test.txt和content包含相同的String

Answer 1

在RegexTestHarness中，您正在从文件中读取行并连接没有换行符的行，之后您进行匹配（readLine()会返回行内容，而不会换行！）。

因此，在MyCrawler类的输入中，输入中可能是换行符。由于默认情况下正则表达式字符.与换行符不匹配，因此在MyCrawler中不起作用。

要解决此问题，请在包含(?s)元字符的所有模式中附加.。所以：

Pattern.compile("<div class=\"rest_title\">.*?<h1>(.*?)</h1>")

会变成：

Pattern.compile("(?s)<div class=\"rest_title\">.*?<h1>(.*?)</h1>")

DOT-ALL标志(?s)将使.匹配任何字符，包括换行符。

正则表达式在测试程序上工作但不在WebSphinx爬虫上工作

1 个答案: