Question

我正在使用 Spring框架的 JdbcTemplate 机制。我也有调度程序类来解析大型XML文件（> 500mb），这些文件是在cron表达式中以pool-size = 10的xml配置进行调度的。当调度程序读取XML文件时，解析每个元素并插入mariadb数据库中，插入大约8000条记录，但之后将冻结或跳过某些记录，但有以下异常：

!important

我的xml解析器类代码段如下：

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=localhost)(port=3307)(type=master) : Address already in use: connect
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:245)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy36.findAccountByACCust(Unknown Source)
    at com.intersect.aml.service.impl.AccountMasterServiceImpl.findAccountByACCust(AccountMasterServiceImpl.java:147)
    at com.intersect.aml.schedule.CbsAcctMasterXmlScheduler.xmlParsingToEntity(CbsAcctMasterXmlScheduler.java:182)
    at com.intersect.aml.schedule.CbsAcctMasterXmlScheduler.execute(CbsAcctMasterXmlScheduler.java:83)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=localhost)(port=3307)(type=master) : Address already in use: connect
    at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:234)
    at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.connException(ExceptionMapper.java:95)
    at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1132)
    at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:560)
    at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:174)
    at org.mariadb.jdbc.Driver.connect(Driver.java:92)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:153)
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:144)
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:155)
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:120)
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:204)
    ... 24 more
Caused by: java.net.BindException: Address already in use: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:406)
    at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1124)
    ... 34 more

我的调度程序配置xml：

public class ModelXmlScheduler {
........
            File fXmlFile = new File(xmlFile);
                    DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
                    DocumentBuilder dBuilder;
                    Document doc = null;
                    try {
                        dBuilder = dbFactory.newDocumentBuilder();
                        doc = dBuilder.parse(fXmlFile);
                    } catch (ParserConfigurationException e) {
                        System.out.println("ParserConfigurationException:" + e.getMessage());
                        e.printStackTrace();
                    } catch (SAXException ee) {
                        System.out.println("SAXException" + ee.getMessage());
                        ee.printStackTrace();
                    }
                    doc.getDocumentElement().normalize();
                    System.out.println("Root element :" + doc.getDocumentElement().getNodeName());
                    NodeList dataElement = doc.getElementsByTagName("row");
                    if (dataElement.getLength() > 0) {

                        xmlParsingToEntity(dataElement);
                    }
    .
    .
    .
    .
    .
            boolean xmlParsingToEntity(NodeList nodeList) {
                Modal dto = null;
                for (int i = 0; i < nodeList.getLength(); i++){
                    dto = new Modal();        
     dto.setField(nodeElement.getElementsByTagName("tag_name").item(0).getTextContent());


       int exists = modalService.findByNumber(dto.getNumber());
        if(exists>0)
           modalService.updateModel(dto);
        else
           modalService.insertModel(dto);
        dto=null;
        }

schedulerConfig.java：

.
.
.
<task:scheduled-tasks scheduler="xmlSchedulers">
...
<task:scheduled ref="ModalXmlScheduler" method="execute" cron="0 01 21 * * *" />
...
</task:scheduled-tasks>
<task:scheduler id="cbsXmlSchedulers" pool-size="1" />

我已经将Java和tomcat的堆内存设置为2gb，并且对于相应的xml文件有五个这样的计划，可以读取数据并将其插入数据库。所有线程都进入 TIME_WAIT 状态< strong>在这种情况下，请帮助解决已经使用的地址

Answer 1

切换到准备好的语句以查找，批量插入和批量更新记录
使用了池数据源，并根据application.properties文件中的属性设置了最大和最小空闲，等待时间和活动时间
datasource.type =org.apache.tomcat.jdbc.pool.DataSource datasource.initial-size=15 datasource.max-wait=6000 datasource.max-active=50 datasource.max-idle=3600 datasource.min-idle=8
使用 StAX解析器方法读取大型XML文件

JournalDev StAX Parser example

SQLNonTransientConnectionException：地址已在使用中

1 个答案: