Question

我试图提取数据并将其直接写入我创建的数据库中 phpmyadmin（wampServer）。数据每天都在增长，我必须保留它今天，所以我会在几天的时间内运行我的prorammcode。现在我的问题：如何防止相同的条目？我用名字，ID和 URL。问题是，某些数据具有相同的ID。通过使用它抛出的名字 SQL语句的语法 - 异常原因（许多名称都有字母喜欢：＆＃34 ;;＆＃34;，＆＃34;＆＃34;＆＃34;，＆＃34; \＆＃34;，＆＃34;`＆＃34;，＆＃34;：＆＃34;，这使得相当困难）和我使用时 URL不会抛出任何错误，但后来我有很多双重条目。并非所有存在的数据都已进入数据库，但我只能阻止其中的一些数据。我该如何解决这个问题？

这是我的programmCode：

package htmlParser;
import java.io.File;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import data.DatenBank_Steam_Spiele;
import data.Spiel;

public class Steam_GameID_Links
{
    public ArrayList<Spiel> spiele;
    public final static String DELIMITER = "Trennzeichen";
    static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
    static final String myDatabase = "jdbc:mysql://localhost:3306    /Steam_Spiele";

public static void main(String[] args) throws SQLException, ClassNotFoundException
{
    Connection conn = null;
    Statement stmt = null;

    Steam_GameID_Links wc = new Steam_GameID_Links();
    try
    {
        Class.forName(JDBC_DRIVER);
        conn = DriverManager.getConnection(myDatabase, "root", "");
        System.out.println("Verbindung erfogreich hergestellt");

        String url = "http://store.steampowered.com/search/?sort_by=_ASC&category1=998&page=1";
        Document document = Jsoup.connect(url).get();
        Elements howMuchPages = document.select(".search_pagination_right");
        String[] stuff = howMuchPages.text().split(" ");
        String tmp = stuff[4].replace(" ", "").replace(".", "");
        StringBuilder sb = new StringBuilder();
        for(int i = 0; i < tmp.length(); i++)
        {
            if(Character.isDigit(tmp.charAt(i)))
            {
                sb.append(tmp.charAt(i));
            }
        }
        String last = sb.toString().trim();;
        int lastPages = Integer.parseInt(last);
        int counter = 0;
        wc.spiele = new ArrayList<>();
        for(int i = 1; i < lastPages + 1; i++)

        {
            System.out.println("______________________ SpieleAngebot Seite " + i + " von insgesamt " + lastPages
                    + " Seiten.");
            url = "http://store.steampowered.com/search/?sort_by=_ASC&category1=998&page=" + i;
            // ACHTUNG: ohne timeout(0) kann der AusleseProzess VORHER abbrechen (dauert zu lange)
            document = Jsoup.connect(url).timeout(0).get();
            // waehlt zunaechst den ElternKnoten: <div id="search_result_container">
            Element parentNode = document.getElementById("search_result_container");
            Elements childNodes = parentNode.getElementsByAttribute("data-ds-appid");
            // FUNZT (releasedates)
            Elements releasedates = document.select("div.col.search_released.responsive_secondrow");
            Elements prices = document.select("div.col.search_price.responsive_secondrow");
            for(int j = 0; j < releasedates.size(); j++)
            {
                String name = childNodes.get(j).getElementsByClass("title").text();
                String id = childNodes.get(j).attr("data-ds-appid");
                int ID = Integer.parseInt(id);
                String href = childNodes.get(j).attr("href");
                String releaseDate = releasedates.get(j).text();
                String priceAll = prices.get(j).text();
                String[] getPrice = priceAll.split(" ");
                String price = getPrice[getPrice.length - 1];
                // Liste befuellen WICHTIG fuer Textfile!!
                wc.spiele.add(new Spiel(name, ID, href, price, releaseDate));

                //check if the game is already in database
                String search = "SELECT * FROM spiel WHERE URL LIKE \"" + href + "\"";
                Statement checkIfAlreadyThere = conn.createStatement();
                ResultSet rs = checkIfAlreadyThere.executeQuery(search);
                if(rs.next())
                {
                    System.out.println("Game already in database");
                }
                else
                {
                    String insert = "INSERT INTO spiel (NAME, ID, URL, VERÖFFENTLICHT, PREIS)"
                            + "VALUES (?,?,?,?,?)";
                    PreparedStatement preparedStatement = conn.prepareStatement(insert);
                    preparedStatement.setString(1, name);
                    preparedStatement.setInt(2, ID);
                    preparedStatement.setString(3, href);
                    preparedStatement.setString(4, releaseDate);
                    if(price.equals("Play") || price.equals("Free"))
                    {
                        price = "kostenlos";
                    }
                    preparedStatement.setString(5, price);
                    preparedStatement.executeUpdate();
                }
                counter++;
            }
        }
        System.out.println("Anzahl Spiele: " + counter);

        for(Spiel out : wc.spiele)
        {
            wc.writeSpielNameIDLink("Spiel:" + out.getName() + DELIMITER + "ID:" + out.getID() + DELIMITER
                    + "SpieleLink:" + out.getLink() + DELIMITER + "veröffentlicht:" + out.getReleaseDate()
                    + DELIMITER + "Preis:" + out.getPrice() + "\n \n");
        }
        wc.writeSpielNameIDLink("\n \n Anzahl Spiele: " + counter);
        Steam_Sales_Per_Game spG = new Steam_Sales_Per_Game();

    }
    catch(IOException e)
    {
        e.printStackTrace();
    }
}

Answer 1

您必须找到游戏的唯一标识符。你应该看看Steam是否为游戏提供了一个非常独特的id，并使用这个id作为游戏的标识符。当我在代码中检查您的网址的Steam链接时，我可以看到每个列出的条目的应用ID。 app id在游戏条目的顶部链接<a>中作为tag-attribute存在（/ app / part之后的href）

编辑：id在href属性中是唯一的

Answer 2

这是我的代码提取，用于检查游戏是否已存在于数据库中：

String search = "SELECT * FROM spiel WHERE URL LIKE \"" + href + "\"";
Statement checkIfAlreadyThere = conn.createStatement();
ResultSet rs = checkIfAlreadyThere.executeQuery(search);
if(rs.next())
{
    System.out.println("Game already in database");
 }
 else
  {
      << pseudoCode: insert game into database >>>
  }

如上所述，检查URL会导致没有问题，但是没有效果，使用NAME（这将是最佳解决方案，因为它真正独特）会导致语法错误。也许有更好的方法来创建SQL-Statment？

Answer 3

好的，我做到了，感谢B的@Erik B，你给了我正确的提示。这是我的解决方案代码-extract：

String href = childNodes.get(j).attr("href");
String cutFirstpart = href.replaceAll("http://store.steampowered.com/app/", "");
String[] hrefSplit = cutFirstpart.split("/");
String uniqueID = hrefSplit[0];
int ID = Integer.parseInt(uniqueID);
String releaseDate = releasedates.get(j).text();
String priceAll = prices.get(j).text();
String[] getPrice = priceAll.split(" ");
String price = getPrice[getPrice.length - 1];
//check if the game is already in database
String search = "SELECT * FROM spiel WHERE ID LIKE \"" + ID + "\"";
Statement checkIfAlreadyThere = conn.createStatement();
ResultSet rs = checkIfAlreadyThere.executeQuery(search);
if(rs.next())
{
    System.out.println("Game already in database");
}
else
{
      << insert NEW data into database >>>>
}

Jsoup，MySQL：如何通过JavaCode阻止数据库中的doubleEntries

3 个答案: