Question

我目前有一个问题，间歇性的oozie工作流将无法连接到我的hive Metastore。好像它与蜂房 - Metastore的连接耗尽了？

引起：MetaException（消息：无法使用连接到元存储提供的任何URI。最近的失败： org.apache.thrift.transport.TTransportException： java.net.ConnectException：连接被拒绝在org.apache.thrift.transport.TSocket.open（TSocket.java:185）在org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open（HiveMetaStoreClient.java:277）在org.apache.hadoop.hive.metastore.HiveMetaStoreClient。（HiveMetaStoreClient.java:163） at sun.reflect.NativeConstructorAccessorImpl.newInstance0（Native Method） at sun.reflect.NativeConstructorAccessorImpl.newInstance（NativeConstructorAccessorImpl.java:39） at sun.reflect.DelegatingConstructorAccessorImpl.newInstance（DelegatingConstructorAccessorImpl.java:27） at java.lang.reflect.Constructor.newInstance（Constructor.java:513）在org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance（MetaStoreUtils.java:1082）在org.apache.hadoop.hive.metastore.RetryingMetaStoreClient。（RetryingMetaStoreClient.java:51）在org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy（RetryingMetaStoreClient.java:61）在org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient（Hive.java:2140）在org.apache.hadoop.hive.ql.metadata.Hive.getMSC（Hive.java:2151）在org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern（Hive.java:1013）在org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern（Hive.java:1000）在org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable（SemanticAnalyzer.java:8732）在org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal（SemanticAnalyzer.java:8097）在org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze（BaseSemanticAnalyzer.java:258）在org.apache.hadoop.hive.ql.Driver.compile（Driver.java:443）在org.apache.hadoop.hive.ql.Driver.compile（Driver.java:347）在org.apache.hadoop.hive.ql.Driver.run（Driver.java:908）在org.apache.hadoop.hive.cli.CliDriver.processLocalCmd（CliDriver.java:259）在org.apache.hadoop.hive.cli.CliDriver.processCmd（CliDriver.java:216）在org.apache.hadoop.hive.cli.CliDriver.processLine（CliDriver.java:412）在org.apache.hadoop.hive.cli.CliDriver.processLine（CliDriver.java:347）在org.apache.hadoop.hive.cli.CliDriver.processReader（CliDriver.java:445）在org.apache.hadoop.hive.cli.CliDriver.processFile（CliDriver.java:455）在org.apache.hadoop.hive.cli.CliDriver.run（CliDriver.java:711）在org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613）在org.apache.oozie.action.hadoop.HiveMain.runHive（HiveMain.java:261）在org.apache.oozie.action.hadoop.HiveMain.run（HiveMain.java:238）在org.apache.oozie.action.hadoop.LauncherMain.run（LauncherMain.java:37）在org.apache.oozie.action.hadoop.HiveMain.main（HiveMain.java:49） at sun.reflect.NativeMethodAccessorImpl.invoke0（Native Method） at sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:39） at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:25）在java.lang.reflect.Method.invoke（Method.java:597）在org.apache.oozie.action.hadoop.LauncherMapper.map（LauncherMapper.java:491）在org.apache.hadoop.mapred.MapRunner.run（MapRunner.java:50）在org.apache.hadoop.mapred.MapTask.runOldMapper（MapTask.java:418）在org.apache.hadoop.mapred.MapTask.run（MapTask.java:333）在org.apache.hadoop.mapred.Child $ 4.run（Child.java:268） at java.security.AccessController.doPrivileged（Native Method）在javax.security.auth.Subject.doAs（Subject.java:396）在org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1408）在org.apache.hadoop.mapred.Child.main（Child.java:262）引起：java.net.ConnectException：连接被拒绝 at java.net.PlainSocketImpl.socketConnect（Native Method）在java.net.PlainSocketImpl.doConnect（PlainSocketImpl.java:351）在java.net.PlainSocketImpl.connectToAddress（PlainSocketImpl.java:213）在java.net.PlainSocketImpl.connect（PlainSocketImpl.java:200）在java.net.SocksSocketImpl.connect（SocksSocketImpl.java:366）在java.net.Socket.connect（Socket.java:529）在org.apache.thrift.transport.TSocket.open（TSocket.java:180） ......还有44个）在org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open（HiveMetaStoreClient.java:323）在org.apache.hadoop.hive.metastore.HiveMetaStoreClient。（HiveMetaStoreClient.java:163） ......还有42个 FAILED：元数据错误：java.lang.RuntimeException：无法实例化 org.apache.hadoop.hive.metastore.HiveMetaStoreClient

我可以从命令行运行配置单元。使用hue与hive-Metoreore进行通信并执行查询。这似乎只发生在我的oozie工作流程的一半。我提交的每个hive操作都将hive-site.xml配置为oozie.hive.defaults。

蜂房-site.xml中

 <property>   <name>hive.metastore.uris</name>  
 <value>thrift://localhost:9083</value>   <description>Thrift uri for
 the remote metastore. Used by metastore client to connect to remote
 metastore.</description> </property>

 <property>   <name>datanucleus.fixedDatastore</name>
     <value>false</value> </property>

 <property>   <name>javax.jdo.option.ConnectionURL</name>  
 <value>jdbc:mysql://localhost:3306/metastore</value>  
 <description>JDBC connect string for a JDBC metastore</description>
 </property>

 <property>   <name>javax.jdo.option.ConnectionDriverName</name>  
 <value>com.mysql.jdbc.Driver</value>   <description>Driver class name
 for a JDBC metastore</description> </property>

 <property>   <name>javax.jdo.option.ConnectionUserName</name>  
 <value>hive</value>   <description>username to use against metastore
 database</description> </property>

 <property>   <name>javax.jdo.option.ConnectionPassword</name>  
 <value>hive</value>   <description>password to use against metastore
 database</description> </property>

hive-metastore，hive-server2，mysql-server和oozie目前都在同一台主机上运行，因此localhost可以运行。有任何想法吗？我有oozie share lib default true，并且已经创建了sharelibs。

CDH 4.2.1包

-oozie：oozie-3.3.0

-hive-Metoreore：hive-metastore-0.10.0

-hive-server2：hive-server2-0.10.0

-mysql-server：mysql-server-5.1.69-1

非常感谢任何帮助

Answer 1

Hive操作中不推荐使用

oozie.hive.defaults。你能尝试让Job Xml指向HDFS上传的hive-site.xml吗？

Oozie Hive工作流间歇性

1 个答案: