Question

有一个表event_logs表中有大约1600万个条目。数据库是MySQL。

在应用程序中，需要逐个读取所有行[特定于我的应用程序要求]

我只是计算检索速度

statement = connection.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
                        java.sql.ResultSet.CONCUR_READ_ONLY);
statement.setFetchSize(Integer.MIN_VALUE);   
statement.executeQuery("select * from test.event_logs");

long startTime = 0;
long endTime = 0;
int counter = 0;

while (resultSet.next()) {  

if (counter == 0){
    startTime = System.currentTimeMillis();         
}

if(counter == 100000){
    endTime = System.currentTimeMillis();           
    long diff = endTime - startTime;
    System.out.println(diff);
    long diffMinutes = diff / (60 * 1000) % 60;
    System.out.println("diffMinutes" + " " + diffMinutes);
    counter = 0;        
}
counter++;
}

对于第一个100k记录，它花了不到一分钟接下来的100k记录花了大约一分钟接下来的100k记录花了大约2分钟接下来的100k记录花了大约11分钟

之后，我正在接受以下异常

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 195 milliseconds ago.  The last packet sent successfully to the server was 152,824 milliseconds ago.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3673)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3562)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4113)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:927)
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2006)
    at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:420)
    at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:399)
    at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7012)
    at com.google.io.InsertCampaignLogs.main(InsertCampaignLogs.java:55)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
    at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
    at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3116)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3629)
    ... 8 more

我的问题

1.为什么检索结果的时间会增加相同数量的记录

2.为什么我在正在运行的查询中间获得通信失败异常异常

提前致谢

Answer 1

您可能需要增加事务长度或读取容器中的行。阅读前10k，然后是下一个10k，每个都是单独的声明。

从MySql中读取大集合

1 个答案: