Hive Remote Metastore＆amp;交易

时间：2015-08-22 11:29:58

标签： transactions hive metastore

我通过编辑hive-site.xml启用了Transaction并在Hive中使用了Remote Metastore（MySQL-MariaDB），如下所示：

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<script src="http://codemirror.net/lib/codemirror.js"></script>
<script src="http://codemirror.net/javascripts/code-completion.js"></script>
<script src="http://codemirror.net/javascripts/css-completion.js"></script>
<script src="http://codemirror.net/javascripts/html-completion.js"></script>
<script src="http://codemirror.net/mode/javascript/javascript.js"></script>
<script src="http://codemirror.net/mode/xml/xml.js"></script>
<script src="http://codemirror.net/mode/css/css.js"></script>
<script src="http://codemirror.net/mode/htmlmixed/htmlmixed.js"></script>
<script src="http://codemirror.net/addon/edit/closetag.js"></script>
<script src="http://codemirror.net/addon/edit/matchbrackets.js"></script>
<script src="http://codemirror.net/addon/selection/active-line.js"></script>
<script src="http://codemirror.net/keymap/extra.js"></script>
<script src="http://codemirror.net/addon/fold/foldcode.js"></script>
<script src="http://codemirror.net/addon/fold/foldgutter.js"></script>
<script src="http://codemirror.net/addon/fold/brace-fold.js"></script>
<script src="http://codemirror.net/addon/fold/xml-fold.js"></script>
<script src="http://codemirror.net/addon/fold/comment-fold.js"></script>
<script src="https://codemirror.net/addon/lint/lint.js"></script>
<script src="https://rawgithub.com/stubbornella/csslint/master/release/csslint.js"></script>
<script src="http://htmlhint.com/js/htmlhint.js"></script>
<script src="https://codemirror.net/addon/lint/css-lint.js"></script>
<script src="https://codemirror.net/addon/lint/html-lint.js"></script>

<div id="code"></div>

当我运行configuration> <property> <name>hive.support.concurrency</name> <value>true</value> </property> <property> <name>hive.enforce.bucketing</name> <value>true</value> </property> <property> <name>hive.exec.dynamic.partition.mode</name> <value>nonstrict</value> </property> <property> <name>hive.txn.manager</name> <value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value> </property> <property> <name>hive.compactor.initiator.on</name> <value>true</value> </property> <property> <name>hive.compactor.worker.threads</name> <value>1</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://master/metastore</value> <description>the URL of the MySQL database</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>hive</value> </property> <property> <name>datanucleus.autoCreateSchema</name> <value>false</value> </property> <property> <name>datanucleus.fixedDatastore</name> <value>true</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://master:9083</value> <description>IP address (or fully-qualified domain name) and port of the metastore host</description> </property> <property> <name>hive.in.test</name> <value>true</value> </property> </configuration> 时，它显示错误：

  

回滚时出错：autocommit = true时无法调用回滚

当我在Hive CLI hive --service metastore中运行查询时，它显示错误：

  

失败：LockException [错误10280]：与...通信时出错   metastore

当我只启用Transaction或仅使用Remote Metastore时，没有错误。

这是Hive日志的一部分：

show databases

这些是我正在使用的版本：

2015-08-22 17:44:12,043 INFO  [pool-3-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 1: source:172.28.10.148 get_all_databases
2015-08-22 17:44:12,049 INFO  [pool-3-thread-1]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=hduser   ip=172.28.10.148    cmd=source:172.28.10.148 get_all_databases  
2015-08-22 17:44:12,053 INFO  [pool-3-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(589)) - 1: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2015-08-22 17:44:12,055 INFO  [pool-3-thread-1]: metastore.ObjectStore (ObjectStore.java:initialize(289)) - ObjectStore, initialize called
2015-08-22 17:44:12,128 INFO  [pool-3-thread-1]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is MYSQL
2015-08-22 17:44:12,128 INFO  [pool-3-thread-1]: metastore.ObjectStore (ObjectStore.java:setConf(272)) - Initialized ObjectStore
2015-08-22 17:44:12,159 INFO  [pool-3-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 1: source:172.28.10.148 get_functions: db=default pat=*
2015-08-22 17:44:12,159 INFO  [pool-3-thread-1]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=hduser   ip=172.28.10.148    cmd=source:172.28.10.148 get_functions: db=default pat=*    
2015-08-22 17:44:20,027 INFO  [pool-3-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 1: source:172.28.10.148 get_all_databases
2015-08-22 17:44:20,028 INFO  [pool-3-thread-1]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=hduser   ip=172.28.10.148    cmd=source:172.28.10.148 get_all_databases  
2015-08-22 17:44:28,015 INFO  [Thread-3]: metastore.HiveMetaStore (HiveMetaStore.java:run(5898)) - Shutting down hive metastore.
2015-08-22 18:03:55,331 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(5881)) - Starting hive metastore on port 9083
2015-08-22 18:03:55,559 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(589)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2015-08-22 18:03:55,594 INFO  [main]: metastore.ObjectStore (ObjectStore.java:initialize(289)) - ObjectStore, initialize called
2015-08-22 18:03:56,815 INFO  [main]: metastore.ObjectStore (ObjectStore.java:getPMF(370)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2015-08-22 18:03:58,280 INFO  [main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is MYSQL
2015-08-22 18:03:58,283 INFO  [main]: metastore.ObjectStore (ObjectStore.java:setConf(272)) - Initialized ObjectStore
2015-08-22 18:03:58,626 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles_core(663)) - Added admin role in metastore
2015-08-22 18:03:58,628 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles_core(672)) - Added public role in metastore
2015-08-22 18:03:58,712 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:addAdminUsers_core(712)) - No user is added in admin role, since config is empty
2015-08-22 18:03:58,962 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6009)) - Starting DB backed MetaStore Server with SetUGI enabled
2015-08-22 18:03:58,965 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6027)) - Started the new metaserver on port [9083]...
2015-08-22 18:03:58,965 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6029)) - Options.minWorkerThreads = 200
2015-08-22 18:03:58,965 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6031)) - Options.maxWorkerThreads = 1000
2015-08-22 18:03:58,965 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6033)) - TCP keepalive = true
2015-08-22 18:03:59,974 INFO  [Thread-4]: metastore.HiveMetaStore (HiveMetaStore.java:initializeAndStartThread(6156)) - Starting metastore thread of type org.apache.hadoop.hive.ql.txn.compactor.Initiator
2015-08-22 18:03:59,999 INFO  [Thread-4]: txn.TxnHandler (TxnHandler.java:checkQFileTestHack(1277)) - Hacking in canned values for transaction manager
2015-08-22 18:04:00,276 ERROR [Thread-4]: metastore.HiveMetaStore (HiveMetaStore.java:run(6104)) - Failure when starting the compactor, compactions may not happen, java.lang.RuntimeException: Unable to set up transaction database for testing: Can't call rollback when autocommit=true
    at org.apache.hadoop.hive.metastore.txn.TxnHandler.checkQFileTestHack(TxnHandler.java:1285)
    at org.apache.hadoop.hive.metastore.txn.TxnHandler.<init>(TxnHandler.java:120)
    at org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.<init>(CompactionTxnHandler.java:43)
    at org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.init(CompactorThread.java:78)
    at org.apache.hadoop.hive.ql.txn.compactor.Initiator.init(Initiator.java:165)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.initializeAndStartThread(HiveMetaStore.java:6159)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startCompactorInitiator(HiveMetaStore.java:6119)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.access$400(HiveMetaStore.java:236)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$4.run(HiveMetaStore.java:6100)

1 个答案:

答案 0 :(得分：0)

当我删除此属性时，不再有错误

<property>
  <name>hive.in.test</name>
  <value>true</value>
</property>
相关问题
最新问题