Question

public void filterLogins() throws IOException, SQLException{
    for(int i = 0; i<200; ++i){
        BufferedReader bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
        long prev_id = 0;
        boolean contains = false;
        while(bufferedReader.ready()){ //very big file
            String line = bufferedReader.readLine();
            Login login = new Login(line);
            if ( login.userId == prev_id && !contains )
                continue;
            if ( samples.contains(login.userId) ){
                mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
                contains = true;
            }else{
                contains = false;
            }
            prev_id = login.userId;
        }
        bufferedReader.close();
        System.out.println((double)i/2.0);
    }
}

此功能长时间工作，因为数据是更大的文件。 2小时前，这是与OutOfMemory异常的crushess

mysql是

的实例

public class MySQLHandler {
private Connection connection = null;
MySQLHandler() throws ClassNotFoundException, SQLException{
    try{
        Class.forName("com.mysql.jdbc.Driver");
        connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/database", "root", "");
    }catch (ClassNotFoundException e){
        System.out.println("Failed jdbc driver load.");
        throw e;
    }
}


public void close() throws SQLException{
       if ( connection != null)
           connection.close();
   }
   public boolean execute(String sql) throws SQLException{
       Statement statement = connection.createStatement();
       return statement.execute(sql);
   }
}

登录它只是带数据的类。（id，name，value，别的东西）。

Answer 1

似乎，您每次都在创建Statement而不是关闭循环中的语句，这会泄漏内存。执行完成后关闭语句。

 public boolean execute(String sql) throws SQLException{
       Statement statement = connection.createStatement();
       return statement.execute(sql);
   }

喜欢

 public boolean execute(String sql) throws SQLException{
     Statement statement = null;
    try {
        statement = connection.createStatement();
        return statement.execute(sql);
    }finaly{
         if (statement != null) statement.close();
     }

UPDATE

正如@Holger在评论中提到的，如果您使用的是JDK 7及更高版本，则可以使用try-with-resources 如下所示

public boolean execute(String sql) throws SQLException{
       try(Statement s=connection.createStatement()) {
          return s.execute(sql);
       }
 }

Answer 2

尝试增加JVM的最大堆大小例如：

-Xms<size>        set initial Java heap size
-Xmx<size>        set maximum Java heap size
-Xss<size>        set java thread stack size

java -Xms16m -Xmx64m ClassName

还可以优化您的代码并重用变量并在使用后关闭/清理。例如：将它们移出循环： BufferedReader bufferedReader，String line和Login login

public void filterLogins() throws IOException, SQLException{
    BufferedReader bufferedReader;
    long prev_id;
    boolean contains;
    String line;
    Login login;
    for(int i = 0; i<200; ++i){
        bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
        prev_id = 0;
        contains = false;
        while(bufferedReader.ready()){ //very big file
            line = bufferedReader.readLine();
            login = new Login(line);
            if ( login.userId == prev_id && !contains )
                continue;
            if ( samples.contains(login.userId) ){
                mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
                contains = true;
            }else{
                contains = false;
            }
            prev_id = login.userId;
        }
        bufferedReader.close();
        System.out.println((double)i/2.0);
    }
}

为什么这个功能会导致内存泄漏？

2 个答案: