我正在尝试创建一个批处理程序。问题是表可能有两百万行，这可能会产生内存问题。在我的工作中，我有DB2，在我的家里测试我使用的是mysql。
我已经使用eclipse进行了调试，并且我查看了结果集中的所有行。
我在网页上搜索过，但我找不到解决方案。例如，我的测试代码是：
package misCodigos;
import java.io.FileWriter;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class testLimitDB {
// JDBC driver name and database URL
static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
static final String DB_URL = "jdbc:mysql://localhost/testing";
// Database credentials
static final String USER = "root";
static final String PASS = "8182";
private static FileWriter fw=null;
public static void main(String[] args) {
// Clase que pretende mostrar cómo insertar filas en java
Connection conn = null;
Statement stmt = null;
try{
//STEP 2: Register JDBC driver
Class.forName("com.mysql.jdbc.Driver");
//STEP 3: Open a connection
System.out.println("Connecting to database...");
conn = DriverManager.getConnection(DB_URL,USER,PASS);
//STEP 4: Execute a query
// System.out.println("Creating statement...");
stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
java.sql.ResultSet.CONCUR_READ_ONLY);
String sql = "select * from tablaTest";
PreparedStatement pstmt;
pstmt = conn.prepareStatement(sql);
conn.setAutoCommit(false);
pstmt.setFetchSize(200);
ResultSet rs = pstmt.executeQuery(sql);
System.out.println("Starting to retrieve data. Memory Used: " + getUsedMemorySize());
//System.out.println(rs.getString(rs.last()));
/* rs.last();
int numRows = rs.getRow();
System.out.println("Numero de filas: " + numRows);*/
fw = new FileWriter("test.out");
while (rs.next()) {
fw.write(rs.getInt(1) + ";" + rs.getString(2) + "\n");
}
System.out.println("Done retrieving data => Memory Used: " + getUsedMemorySize());
pstmt.close();
conn.close();
fw.flush();
fw.close();
}catch(Exception e){
//Handle errors for Class.forName
e.printStackTrace();
}finally{
//finally block used to close resources
try{
if (fw != null){
fw.flush();
fw.close();
}
if(stmt!=null) {
stmt.close();
}
}catch(SQLException | IOException se2){
}// nothing we can do
try{
if(conn!=null) {
conn.close();
}
}catch(SQLException se){
se.printStackTrace();
}//end finally try
}//end try
System.out.println("Finalizado!");
}
public static long getUsedMemorySize() {
long freeSize = 0L;
long totalSize = 0L;
long usedSize = -1L;
try {
Runtime info = Runtime.getRuntime();
freeSize = info.freeMemory();
totalSize = info.totalMemory();
usedSize = totalSize - freeSize;
} catch (Exception e) {
e.printStackTrace();
}
return usedSize;
}
}
Could anyone help me?
由于
PD：我为db2创建了代码。我不确定结果集是否只获取我喜欢的行，因为我在eclipse中使用调试模式和结果变量，现在它是非常不同的，它显示加密。
import java.io.FileWriter;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class testLimitDB {
// JDBC driver name and database URL
//static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
static final String JDBC_DRIVER = "com.ibm.db2.jcc.DB2Driver";
//static final String DB_URL = "jdbc:mysql://localhost/testing";
static final String DB_URL = "jdbc:db2://localhost:50000/SAMPLE";
// Database credentials MYSQL
//static final String USER = "root";
static final String USER = "db2admin";
static final String PASS = "8182";
private static FileWriter fw=null;
public static void main(String[] args) {
// Clase que pretende mostrar cómo insertar filas en java
Connection conn = null;
Statement stmt = null;
try{
//STEP 2: Register JDBC driver
Class.forName(JDBC_DRIVER);
//STEP 3: Open a connection
System.out.println("Connecting to database...");
conn = DriverManager.getConnection(DB_URL,USER,PASS);
//STEP 4: Execute a query
// System.out.println("Creating statement...");
stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
java.sql.ResultSet.CONCUR_READ_ONLY);
String sql = "select * from FELIPE.TABLATEST";
PreparedStatement pstmt;
pstmt = conn.prepareStatement(sql);
conn.setAutoCommit(false);
pstmt.setFetchSize(999);
ResultSet rs = pstmt.executeQuery();
System.out.println("Starting to retrieve data. Memory Used: " + getUsedMemorySize());
//System.out.println(rs.getString(rs.last()));
/* rs.last();
int numRows = rs.getRow();
System.out.println("Numero de filas: " + numRows);*/
fw = new FileWriter("test.out");
while (rs.next()) {
fw.write(rs.getInt(1) + ";" + rs.getString(2) + "\n");
}
System.out.println("Done retrieving data => Memory Used: " + getUsedMemorySize());
pstmt.close();
conn.close();
fw.flush();
fw.close();
}catch(Exception e){
//Handle errors for Class.forName
e.printStackTrace();
}finally{
//finally block used to close resources
try{
if (fw != null){
fw.flush();
fw.close();
}
if(stmt!=null) {
stmt.close();
}
}catch(SQLException | IOException se2){
}// nothing we can do
try{
if(conn!=null) {
conn.close();
}
}catch(SQLException se){
se.printStackTrace();
}//end finally try
}//end try
System.out.println("Finalizado!");
}
public static long getUsedMemorySize() {
long freeSize = 0L;
long totalSize = 0L;
long usedSize = -1L;
try {
Runtime info = Runtime.getRuntime();
freeSize = info.freeMemory();
totalSize = info.totalMemory();
usedSize = totalSize - freeSize;
} catch (Exception e) {
e.printStackTrace();
}
return usedSize;
}
}
我再次尝试，我的结论是驱动程序（mysql和db2）正常工作。我用jvisualvm进行了测试，我清楚地看到内存随着光标的大小而变化。我会写一篇不同结论的文档，我会发布它。
答案 0 :(得分：1)
我认为你的问题来自于没有服务器端游标的MySQL测试，选择大型结果集无论是设置FETCH大小还是不设置它都不可避免地导致OOM。
由于游标，DB2没有相同的问题，驱动程序不会抛出OOM。您需要定义一个非常适合的获取大小。如果你有一个大的批量作业，我会使用一个大的值。换句话说，当您针对相同的环境进行测试时：）