我正在研究一个项目,用于解析文件,然后将解析后的数据插入数据库。由于记录数量以百万计,我决定使用执行程序服务进行多线程,使用c3p0进行数据库连接池。但是,在某个地方似乎有一个数据库连接泄漏,我无法弄清楚在哪里,但是当我在数据库上运行show full processlist时,它显示所有连接都处于休眠状态。具体来说,我认为这种泄漏发生在run方法的person部分,因为我得到了所有的出版物,并且当我评论该部分时,整个事情都没有失败。通常大约有200,000个条目进入人员处理泄漏的时间。这是我的主题'运行方法:
protected Connection DB;
private ComboPooledDataSource source;
protected static final Object lock = new Object();
public void openDB() {
synchronized (lock) {
try {
this.DB = source.getConnection();
this.DB.setAutoCommit(false);
} catch (SQLException ex) {
SynchronizedPrinter.printErr(ex.getMessage());
}
}
}
public void closeDB() {
try {
if (!DB.isClosed()) {
DB.close();
}
} catch (SQLException ex) {
SynchronizedPrinter.printErr(ex.getMessage());
}
}
public static boolean executeBatches(PreparedStatement ps) {
try {
ps.executeBatch();
} catch (SQLException ex) {
String exm = ex.getMessage();
if (exm.contains("try restarting transaction")) {
return true;
}
}
return false;
}
@Override
public void run() {
this.openDB();
SynchronizedPrinter.printOut(Thread.currentThread() + " open");
try {
boolean executePerson = !personList.isEmpty();
while (executePerson) {
ps_person = DB.prepareStatement(INSERT_PERSON);
for (Element e : personList) {
ps_person.setString(1, e.getKey());
ps_person.setInt(2, getAuthorId(e.getFirstAuthor().getContent()));
List<Field> insertList = e.getFieldListForInsert();
for (int i = 0; i < insertList.size(); i++) {
ps_person.setString(i + 3, insertList.get(i) == null ? null : insertList.get(i).getContent());
}
ps_person.addBatch();
}
SynchronizedPrinter.printOut(Thread.currentThread() + " execute person");
executePerson = executeBatches(ps_person);
SynchronizedPrinter.printOut(Thread.currentThread() + " executed person");
ps_person.close();
if (executePerson) {
DB.rollback();
} else {
DB.commit();
}
SynchronizedPrinter.printOut(Thread.currentThread() + " committed person");
}
boolean executeSynonym = !synonymList.isEmpty();
while (executeSynonym) {
ps_synonym = DB.prepareStatement(AUTHOR_SYNONYM_MAPPER);
for (Element e : synonymList) {
for (Field f : e.getAuthorList()) {
if (!f.equals(e.getFirstAuthor())) {
ps_synonym.setInt(1, getAuthorId(e.getFirstAuthor().getContent()));
ps_synonym.setString(2, f.getContent());
ps_synonym.addBatch();
}
}
}
executeSynonym = executeBatches(ps_synonym);
ps_synonym.close();
if (executeSynonym) {
DB.rollback();
} else {
DB.commit();
}
}
boolean executePublication = !pubList.isEmpty();
while (executePublication) {
ps_pub = DB.prepareStatement(INSERT_PUB);
for (Element e : pubList) {
ps_pub.setString(1, e.getKey());
ps_pub.setString(2, e.elementType.toString());
List<Field> insertList = e.getFieldListForInsert();
for (int i = 0; i < insertList.size(); i++) {
ps_pub.setString(i + 3, insertList.get(i) == null ? null : insertList.get(i).getContent());
}
ps_pub.addBatch();
}
executePublication = executeBatches(ps_pub);
ps_pub.close();
if (executePublication) {
DB.rollback();
} else {
DB.commit();
}
}
} catch (SQLException ex) {
SynchronizedPrinter.printErr("Something terribly wrong has occurred:\n" + ex.getMessage());
System.exit(1);
}
SynchronizedPrinter.printOut(Thread.currentThread() + " closing");
this.closeDB();
SynchronizedPrinter.printOut(Thread.currentThread() + " closed");
}
这是打开线程的方法。它传递ComboPooledDataSource和要插入的元素(一次1024个元素,然后根据它们是出版物,人物还是同义词进行划分):
if (currentCounter >= PROCESSED_AT_ONCE) {
try {
while (source.getNumBusyConnections() >= CONNECTION_LIMIT-1) Thread.yield();
} catch (SQLException ex) {
SynchronizedPrinter.printErr(ex.getMessage());
}
service.submit(new ElementProcessor(source, currentElements));
currentCounter = 0;
}
最后,这是我在所有线程被卡住后生成的一些jstack转储。所有的线程都被卡在了看起来相同的位置:
"pool-1-thread-63" #79 prio=5 os_prio=31 tid=0x00007fa849439000 nid=0xdc03 waiting on condition [0x000070000599e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c2c39588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
我无法弄清楚我的生活是什么造成了这种情况。该出版物和同义词插入工作完美无问题,但这些人一直非常有问题,总是造成某种我永远无法通过的锁定。然而,逻辑与其他两个几乎完全相同。
这里是我生成的输出日志中的最后几行,它开始变得不稳定:
Thread[pool-1-thread-60,5,main] open
Thread[pool-1-thread-33,5,main] executed person
Thread[pool-1-thread-33,5,main] committed person
Thread[pool-1-thread-33,5,main] closing
Thread[pool-1-thread-33,5,main] closed
Thread[pool-1-thread-46,5,main] open
Thread[pool-1-thread-46,5,main] execute person
Thread[pool-1-thread-54,5,main] executed person
Thread[pool-1-thread-54,5,main] committed person
Thread[pool-1-thread-54,5,main] closing
Thread[pool-1-thread-54,5,main] closed
Thread[pool-1-thread-40,5,main] open
Thread[pool-1-thread-46,5,main] executed person
Thread[pool-1-thread-46,5,main] committed person
Thread[pool-1-thread-46,5,main] closing
Thread[pool-1-thread-46,5,main] closed
Thread[pool-1-thread-16,5,main] open
我真的迷失了导致所有这一切的原因,任何人都能提供的任何帮助都会令人惊叹。