如何使用hiberate优化插入查询时间?

时间:2017-02-08 08:20:19

标签: java sql-server spring hibernate wildfly-10

我有大约140万条记录的数据,但插入它们需要3个多小时。我似乎无法找到它的问题。

我读了并从身份变为序列。它只有一点点改进,但​​完成插入仍需要很长时间。

我正在使用:

  • Hibernate 5
  • 春季4
  • mssql 2014
  • Wildfly 10

的applicationContext-hibernate.xml

<tx:advice id="txAdvice">
        <!-- the transactional semantics... -->
        <tx:attributes>
            <tx:method name="*_TransNew" propagation="REQUIRES_NEW" />
            <tx:method name="*_NoTrans" propagation="NEVER" />

            <tx:method name="create*" propagation="REQUIRED" />
            <tx:method name="update*" propagation="REQUIRED" />
            <tx:method name="delete*" propagation="REQUIRED" />
            <tx:method name="add*" propagation="REQUIRED" />

            <tx:method name="generate*" propagation="REQUIRED" />
            <tx:method name="get*" propagation="REQUIRED" />
            <tx:method name="is*" propagation="REQUIRED" />

            <!-- other methods use the default transaction settings (see below) -->
            <tx:method name="*" read-only="true" />
        </tx:attributes>
    </tx:advice>


    <!-- ensure that the above transactional advice runs for any execution of 
        an operation defined by the following -->
    <aop:config>
        <aop:pointcut id="demoServiceOperations"
            expression="execution(* com.test.*.*.*(..))" />
        <aop:advisor advice-ref="txAdvice" pointcut-ref="demoServiceOperations" />
    </aop:config>

<bean id="sessionFactory"
        class="org.springframework.orm.hibernate5.LocalSessionFactoryBean">
        <property name="dataSource" ref="dataSource" />
        <property name="hibernateProperties">
            <props>
                <prop key="hibernate.show_sql">true</prop>
                <prop key="hibernate.dialect">org.hibernate.dialect.SQLServer2012Dialect</prop>
                <prop key="hibernate.cache.region.factory_class">org.hibernate.cache.ehcache.SingletonEhCacheRegionFactory</prop>
                <prop key="hibernate.cache.use_second_level_cache">true</prop>
                <prop key="hibernate.cache.use_query_cache">true</prop>
                <prop key="hibernate.jdbc.batch_size">50</prop>
                <prop key="hibernate.order_inserts">true</prop>
                <prop key="hibernate.order_updates">true</prop>

                <prop key="hibernate.c3p0.min_size">5</prop>
                <prop key="hibernate.c3p0.max_size">20</prop>
                <prop key="hibernate.c3p0.timeout">1800</prop>
                <prop key="hibernate.c3p0.max_statements">50</prop>


            </props>
        </property>         
    </bean>

Umts.hbm.xml:

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">
<!-- Generated Nov 22, 2016 11:36:21 AM by Hibernate Tools 5.2.0.Beta1 -->
<hibernate-mapping>
    <class name="com.test.domain.Umts" table="TBLDM_UMTS" schema="dbo" catalog="DEMO" optimistic-lock="version" dynamic-update="true">
        <id name="umtsId" type="java.lang.Integer">
            <column name="UMTS_ID" />
            <generator class="org.hibernate.id.enhanced.SequenceStyleGenerator">
                <param name="optimizer">pooled-lo</param>
                <param name="increment_size">1</param>
                <param name="sequence_name">UMTS_SEQ</param>
            </generator>
        </id>
        <property name="cid" type="java.lang.Integer">
            <column name="CI" not-null="true" />
        </property>
        <property name="channelNo" type="java.lang.Integer">
            <column name="UARFCN" />
        </property>
        <property name="signalStrength" type="java.lang.Double">
            <column name="EC_IO" precision="53" scale="0" />
        </property>
        <property name="sc" type="java.lang.Integer">
            <column name="SC" />
        </property>
        <property name="latitude" type="java.lang.Double">
            <column name="LATITUDE" precision="53" scale="0" />
        </property>
        <property name="longitude" type="java.lang.Double">
            <column name="LONGITUDE" precision="53" scale="0" />
        </property>
         <property name="mcc" type="java.lang.Integer">
            <column name="MCC" not-null="true" />
        </property>
        <property name="mnc" type="java.lang.Integer">
            <column name="MNC" not-null="true" />
        </property>
        <property name="recvDate" type="date">
            <column name="RECV_DATE" length="10" />
        </property>
        <property name="recvTime" type="time">
            <column name="RECV_TIME" length="16" />
        </property>
    </class>
</hibernate-mapping>

服务类:

public void process(List<Umts> umtsList)
{
  for (int i = 0; i < umtsList.size(); i = i + PropertiesUtil.MAX_COMMIT_COUNT)
        {
            int min = i;
            int max = i + PropertiesUtil.MAX_COMMIT_COUNT;

            if (max > umtsList.size())
            {
                max = umtsList.size();
            }

            createUmts_TransNew(umtsList.subList(min, max));
        }
}
    @Override
        public void createUmts_TransNew(Collection list) 
        {
            // TODO Auto-generated method stub

            umtsDAO.saveAll(list);  
        }

DAO课程:

@Transactional
    public void saveAll(Collection collection)
    {
        log.debug("** save all");
        try
        {
            if (collection != null && collection.size() > 0)
            {
                for (Object obj : collection)
                {
                    sessionFactory.getCurrentSession().saveOrUpdate(obj);                   
                }
            }
        }
        catch (RuntimeException re)
        {
            log.error("** save all failed", re);
            throw re;
        }
    }

**编辑 连接池是否在此处起作用?连接池有什么意义有助于提高性能?我是否需要将jar文件添加到wildfly 10或应用程序本身?

1 个答案:

答案 0 :(得分:0)

首先,尝试增加pooled-lo值。由于您将其设置为1,因此不会进行优化/池化 - 因为需要获取的每个ID都需要调用DB才能获得实际值。如果你有一个更大的增量大小,休眠将预取/保留一个ID块用于新实体,而无需每个实体往返。

不确定您发布的代码是如何执行的,但我假设您在单个线程中按顺序插入它们。你可以:

  • 使用一个线程池,其中每个线程从列表/队列中取出一些项目以进行插入。
  • 一个线程在单个事务中插入的项目与配置的休眠批处理大小理想地大小相同,以便最小化往返。
  • 确保池中的线程数与连接池大小相同,因此在等待连接时不要阻止工作线程。
  • 确保连接池大小对于服务器可以承担的负载是合理的,并使用良好的连接池(例如HikariCP)。这是一个有趣的writeup on connection pool size.