Question

我已经设置了SOLR。我曾经将数据（1万分之一）导入solr索引，该索引正常工作。

现在我正在尝试将大数据（10万十万条记录）导入solr索引。我已经设置了data-config.xml然后我开始完全导入但是它失败了并且给出了下面提到的错误

索引失败。回滚所有更改。

我将solr start log存储在solr.err文件中，该文件出现错误：

Jul 31, 2012 10:19:33 AM org.apache.solr.update.DirectUpdateHandler2 rollback
SEVERE: Ignoring Error when closing connection
    at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:637)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4749)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2627)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:841)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:681)
    ... 6 more
Jul 31, 2012 10:19:44 AM org.apache.solr.update.DirectUpdateHandler2 rollback
SEVERE: Ignoring Error when closing connection
    at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:637)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2709)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2709)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2627)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:841)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:681)
    ... 6 more
Jul 31, 2012 10:19:44 AM org.apache.solr.update.DirectUpdateHandler2 rollback

SEVERE: Ignoring Error when closing connection at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:637) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)

Jul 31, 2012 10:19:44 AM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback
Jul 31, 2012 10:19:44 AM org.apache.solr.handler.dataimport.JdbcDataSource closeConnection
SEVERE: Ignoring Error when closing connection

com.mysql.jdbc.PacketTooBigException: Packet for query is too large (7959657 > 1048576). You can change this value on the server by setting the max_allowed_packet' variable.
at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:637)
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:420)
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3118)
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2288)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2709)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4863)
    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4749)
    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4358)
    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1607)

@ ataConfig.java:342）在org.apache.solr.handler.dataimport.DataImporter.doFullImport（DataImporter.java:383）在org.apache.solr.handler.dataimport.DataImporter.runCmd（DataImporter.java:445）在org.apache.solr.handler.dataimport.DataImporter $ 1.run（DataImporter.java:426）

我使用4核处理器和4GB RAM。

解决此问题需要哪些mysql / java配置更改？

Answer 1

我想这与Mysql有关，而与Solr有关，因为mysql jdbc驱动程序存在问题。试试this workaround here，它只是提升了mysql可以处理的数据限制。

对于SOLR中的大数据，Dataimport失败

1 个答案: