Question

我正在尝试将所有具有标题的网址设置为Content-Type：text / html所以我正在检查每个网址的响应标头，如果他们有内容类型：text / html，那么我想要打印出来具有content-type：text / html的url。但是在我的代码中，当我检查标题是否包含Content-Type时，它没有显示任何内容..如果我删除if循环，那么它会打印与我想要抓取的特定网址相关的每个链接及其响应标头..

public class MyCrawler extends WebCrawler {

    Pattern filters = Pattern.compile(".*(\\.(css|js|bmp|gif|jpe?g"
            + "|png|tiff?|mid|mp2|mp3|mp4" + "|wav|avi|mov|mpeg|ram|m4v|pdf"
            + "|rm|smil|wmv|swf|wma|zip|rar|gz))$");


    /*
    Pattern filters = Pattern.compile("(\\.(html))");
*/
    public MyCrawler() {
    }

    public boolean shouldVisit(WebURL url) {
        String href = url.getURL().toLowerCase();
        //System.out.println("Href: " +href);
        /*
        if (filters.matcher(href).matches()) {
            return false;
        }*/
        if (href.startsWith("http://www.somehost.com/")) {
            return true;
        }
        return false;
    }

    public void visit(Page page) {

        int docid = page.getWebURL().getDocid();

        String url = page.getWebURL().getURL();         
        String text = page.getText();
        List<WebURL> links = page.getURLs();
        int parentDocid = page.getWebURL().getParentDocid();


        //HttpGet httpget = new HttpGet(url);


        try {
            URL url1 = new URL(url);
            URLConnection connection = url1.openConnection();

            Map responseMap = connection.getHeaderFields();
      for (Iterator iterator = responseMap.keySet().iterator(); iterator.hasNext();) 
     {
                String key = (String) iterator.next();
                if(key==("Content-Type")) //(Anything wrong with this if loop)
                {
                System.out.print(key + " = ");

                List values = (List) responseMap.get(key);
                for (int i = 0; i < values.size(); i++) {
                    Object o = values.get(i);
                    System.out.print(o + ", ");
                }
                System.out.println("");
System.out.println(url1);
                }

            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }


        //System.out.println("Docid: " + docid);
        //System.out.println("URL: " + url);
        //System.out.println("Text length: " + text.length());
        //System.out.println("Number of links: " + links.size());
        //System.out.println("Docid of parent page: " + parentDocid);
        System.out.println("=============");
    }   
}

Answer 1

键变量包含：

Content-Type=[text/html; charset=ISO-8859-1]

因此无法抓住==或.equals("Content-Type")

如果您尝试运行以下代码，请查看其打印的内容

URLConnection connection = url1.openConnection();

Map responseMap = connection.getHeaderFields();
Iterator iterator = responseMap.entrySet().iterator();
while (iterator.hasNext())
{
    String key = iterator.next().toString();
    if (key.contains("Content-Type"))
    {
        System.out.println(key);
        // Content-Type=[text/html; charset=ISO-8859-1]
        if (filters.matcher(key) != null){
            System.out.println(url1);
            // http://google.com
        }
    }
}

这是输出：

Content-Type=[text/html; charset=ISO-8859-1]
http://google.com

看起来您也可以使用以下一个if语句：

while (iterator.hasNext())
{
    String key = iterator.next().toString();
    if (key.contains("text/html"))
    {
        System.out.println(url1);
        // http://google.com
    }
}

Java is very intuitive中的BTW字符串比较，让我一直都在这里！

Answer 2

对于字符串比较，请使用.equals()。

Answer 3

它应该与

一起使用

if (key != null && key.equals("Content-Type"))

在检查响应标头时仅爬网HTML页面

3 个答案: