Question

我编写代码来抓取并保存网页中的图像。出于某种原因，我得到了错误，我不知道如何解决。

我正在使用一种方法来确保我索引的每个图像都存在，所以我不确定为什么会这样。

这是我的代码：

import org.jsoup.Jsoup;
import org.jsoup.helper.Validate;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.net.*;
import java.awt.Image;
import java.awt.image.RenderedImage;
import java.io.*;

import java.io.IOException;

import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;

public class jsoup {
    public static void main(String[] args) throws IOException {
    crawl("http://www.istockphoto.com/photo");
}

public static void crawl(String crawlurl) throws IOException{
    Document doc = Jsoup.connect(crawlurl).get();
    getImgFromLinks(doc);
}

public static void getImgFromLinks(Document doc) throws IOException{
    Elements links = doc.select("a[href]");
    //System.out.println(links);

    for(int i=0;i<links.size();i++){
        if(exists(links.get(i).attr("href"))){
            System.out.println("crawled: " + links.get(i).attr("href"));
            getImages(doc, links.get(i).attr("href"));
        }else{
            System.out.println("I couldnt crawl: " + links.get(i).attr("href"));
        }
    }
}

public static String smartUrl(String url, String src) {
    if(exists(src)){
        return(src);
    }else{
        return(url + src);
    }
}


public static void getImages(Document doc, String url) throws IOException{



      for(int i=0; i<doc.getElementsByTag("img").size();i++){
            Element image = doc.select("img").get(i);
            String imgsrc = image.attr("src");
            if(imgsrc.toLowerCase().contains("png") || imgsrc.toLowerCase().contains("jpg") || imgsrc.toLowerCase().contains("jpeg") || imgsrc.toLowerCase().contains("gif")){

            int slashIndex = smartUrl(url, imgsrc).lastIndexOf('/');
            String finalUrl = smartUrl(url, imgsrc).substring(slashIndex);

            URL imgurl = new URL(smartUrl(url, imgsrc));

            if(exists(imgurl.toString())){
            Image crawledimg = ImageIO.read(imgurl);


            ImageIO.write((RenderedImage) crawledimg, "gif",new File("/Users/Jonathan/Desktop/crawledimages" + finalUrl));


            System.out.println("I got an image from:" + url + " Image Name: " + finalUrl);
            }

        }
        }


}


public static boolean exists(String URLName) {
    try {
      HttpURLConnection.setFollowRedirects(false);

    //HttpURLConnection.setInstanceFollowRedirects(false);
      HttpURLConnection con =
         (HttpURLConnection) new URL(URLName).openConnection();
      con.setRequestMethod("HEAD");
      return (con.getResponseCode() == HttpURLConnection.HTTP_OK);
    }
    catch (Exception e) {
       return false;
    }
  }
}

这是输出：

crawled: http://www.istockphoto.com/
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /facebook.png
I got an image from:http://www.istockphoto.com/ Image Name: /twitter.png
I got an image from:http://www.istockphoto.com/ Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/ Image Name: /cartWhite.png
I couldnt crawl: #
I couldnt crawl: http://www.istockphoto.com/sign-in/aHR0cCUzQSUyRiUyRnd3dy5pc3RvY2twaG90by5jb20lMkZwaG90bw==
I couldnt crawl: http://www.istockphoto.com/join/aHR0cCUzQSUyRiUyRnd3dy5pc3RvY2twaG90by5jb20lMkZwaG90bw==
crawled: http://www.istockphoto.com/photo
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
I got an image from:http://www.istockphoto.com/photo Image Name: /facebook.png
I got an image from:http://www.istockphoto.com/photo Image Name: /twitter.png
I got an image from:http://www.istockphoto.com/photo Image Name: /blank.gif
 Exception in thread "main" java.lang.IllegalArgumentException: im == null!
at javax.imageio.ImageIO.write(ImageIO.java:1457)
at javax.imageio.ImageIO.write(ImageIO.java:1527)
at jsoup.getImages(jsoup.java:68)
at jsoup.getImgFromLinks(jsoup.java:34)
at jsoup.crawl(jsoup.java:24)
at jsoup.main(jsoup.java:19)

正在保存图像，直到发生错误。

有人知道如何解决这个问题吗？

此外，由于某些原因，页面上的相同图像被多次保存。

感谢您的时间，

Jonathan Oren。

Answer 1

在null

中传递了ImageIO.write()

smartURL函数有一个您需要解决的缺陷。它不是从网页中获取的图像网址中创建预期的网址。

例如： /static/images/cartWhite.png将由smartURL转换为http://www.istockphoto.com/photo/static/images/cartWhite.png，这不是图像，但同时也不是错误页面。因此crawledimg指的是null引起IllegalArgumentException。

快速解决此问题的方法是在URL内创建仅http://www.istockphoto.com的{{1}}。

图像会被多次保存，因为每个页面都有它们。您可以维护一个图像列表，以避免这种情况发生。

我在您的代码中找到了另一个显示停止点，您将无法从您抓取的网页中检索任何其他图像。网站上的图片不会以getImages()或*.jpg等结尾。因此，您需要在开始之前研究网站上图片网址的模式。

java.lang.IllegalArgumentException在Java中使用Jsoup时

1 个答案: