WSO2 BAM问题与hdb查询到jdbc到MySQL外部DB：错误

时间：2014-06-14 11:54:32

标签： wso2 hive

我正在测试通过REST API将数据从Perl推送到BAM中，这看起来效果很好。现在正在进行下一步，即将聚合测试到另一台主机上的单独MySQL数据库，我收到一个我似乎无法通过的错误。请参阅下面的堆栈跟踪.. 

Hive history file=/Users/<user>/local/wso2bam-2.4.1/tmp/hive/wso2-querylogs/hive_job_log_jskogsta_201406141932_1331503753.txt
Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
[2014-06-14 19:33:00,180] ERROR {org.apache.hadoop.hive.ql.exec.Task} -  Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
    at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:278)
    at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3360)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
    at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
    at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.executeHiveQuery(HiveExecutorServiceImpl.java:577)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:287)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:190)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
    at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:151)
    at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:273)
    ... 17 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
    at org.wso2.carbon.ui.JspClassLoader.findClass(JspClassLoader.java:113)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at org.wso2.carbon.ui.JspClassLoader.loadClass(JspClassLoader.java:108)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:249)
    at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:144)
    ... 18 more

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[2014-06-14 19:33:00,180] ERROR {org.apache.hadoop.hive.ql.Driver} -  FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[2014-06-14 19:33:00,183] ERROR {org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl} -  Error while executing Hive script.
Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:189)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.executeHiveQuery(HiveExecutorServiceImpl.java:577)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:287)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:190)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)
[2014-06-14 19:33:00,200] ERROR {org.wso2.carbon.analytics.hive.ui.client.HiveExecutionClient} -  Error while executing Hive script.Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
org.wso2.carbon.analytics.hive.stub.HiveExecutionServiceHiveExecutionException: HiveExecutionServiceHiveExecutionException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at java.lang.Class.newInstance0(Class.java:357)
    at java.lang.Class.newInstance(Class.java:310)
    at org.wso2.carbon.analytics.hive.stub.HiveExecutionServiceStub.executeHiveScript(HiveExecutionServiceStub.java:216)
    at org.wso2.carbon.analytics.hive.ui.client.HiveExecutionClient.executeScript(HiveExecutionClient.java:66)
    at org.apache.jsp.hive_002dexplorer.queryresults_jsp._jspService(org.apache.jsp.hive_002dexplorer.queryresults_jsp:92)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.ui.JspServlet.service(JspServlet.java:155)
    at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:80)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
    at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:605)
    at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:544)
    at org.eclipse.equinox.http.servlet.internal.RequestDispatcherAdaptor.include(RequestDispatcherAdaptor.java:37)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor$RequestDispatcherAdaptor.include(ContextPathServletAdaptor.java:369)
    at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:1015)
    at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:700)
    at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.tiles.jsp.context.JspUtil.doInclude(JspUtil.java:87)
    at org.apache.tiles.jsp.context.JspTilesRequestContext.include(JspTilesRequestContext.java:88)
    at org.apache.tiles.jsp.context.JspTilesRequestContext.dispatch(JspTilesRequestContext.java:82)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:465)
    at org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:140)
    at org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:117)
    at org.apache.tiles.jsp.taglib.RenderTagSupport.execute(RenderTagSupport.java:171)
    at org.apache.tiles.jsp.taglib.RoleSecurityTagSupport.doEndTag(RoleSecurityTagSupport.java:75)
    at org.apache.tiles.jsp.taglib.ContainerTagSupport.doEndTag(ContainerTagSupport.java:80)
    at org.apache.jsp.admin.layout.template_jsp._jspx_meth_tiles_insertAttribute_7(org.apache.jsp.admin.layout.template_jsp:603)
    at org.apache.jsp.admin.layout.template_jsp._jspService(org.apache.jsp.admin.layout.template_jsp:335)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.ui.JspServlet.service(JspServlet.java:155)
    at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:80)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
    at org.eclipse.equinox.http.servlet.internal.RequestDispatcherAdaptor.forward(RequestDispatcherAdaptor.java:30)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor$RequestDispatcherAdaptor.forward(ContextPathServletAdaptor.java:362)
    at org.apache.tiles.servlet.context.ServletTilesRequestContext.forward(ServletTilesRequestContext.java:198)
    at org.apache.tiles.servlet.context.ServletTilesRequestContext.dispatch(ServletTilesRequestContext.java:185)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:419)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:370)
    at org.wso2.carbon.ui.action.ActionHelper.render(ActionHelper.java:52)
    at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:101)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
    at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
    at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
    at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)

我甚至下载了MySQL jar文件并放入/ lib：

[19:31:00] [<user>@enterprise ../local/wso2bam-2.4.1/lib]$ ls
ant-contrib-1.0b3.jar           org.wso2.carbon.logging-4.2.0.jar   tomcat
commons-lang-2.6.0.wso2v1.jar       org.wso2.carbon.server-4.2.0.jar    transactions.properties
endorsed                org.wso2.ciphertool-1.0.0-wso2v2.jar
mysql-connector-java-5.1.31.tar     runtimes
[19:45:34] [<user>@enterprise ../local/wso2bam-2.4.1/lib]$

这是我试图在BAM admin gui中执行的查询，它失败了：

CREATE EXTERNAL TABLE IF NOT EXISTS Summarized_Table ( payload_symbol STRING, payload_min INT, payload_price STRING, payload_volume INT, payload_max INT)
STORED BY 'org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler'
TBLPROPERTIES ( 'mapred.jdbc.driver.class' = 'com.mysql.jdbc.Driver' ,
'mapred.jdbc.url' = 'jdbc:mysql://<IP>:<PORT>/testdb' ,
'mapred.jdbc.username' = '<USER>' ,
'mapred.jdbc.password' = '<PASS>' ,
'hive.jdbc.update.on.duplicate' = 'true' ,
'hive.jdbc.table.create.query' = 'CREATE TABLE testtable
                                (payload_symbol VARCHAR(10), payload_min INT, payload_price VARCHAR(20), payload_volume INT, payload_max INT) ');

以下查询，我在此之前所做的工作正常：

CREATE EXTERNAL TABLE IF NOT EXISTS TestTable (Row_ID STRING, payload_symbol STRING,
          payload_min INT, payload_price STRING, payload_volume INT, payload_max INT)
STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
WITH SERDEPROPERTIES ( "cassandra.host" = "127.0.0.1" ,
"cassandra.port" = "9160" ,
"cassandra.ks.name" = "EVENT_KS" ,
"cassandra.ks.username" = "<USER>" ,
"cassandra.ks.password" = "<PASS>" ,
"cassandra.cf.name" = "stockquote_stream" ,
"cassandra.columns.mapping" = ":key,payload_symbol,payload_min,payload_price,payload_volume,payload_max" );

我已经确认我也可以从这台电脑连接到远程数据库：

[19:51:22] [<USER>@enterprise ../local/wso2bam-2.4.1/lib]$ mysql -u currentcost -h <IP> --port=<PORT> -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 43
Server version: 5.5.34 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit
Bye
[19:51:46] [<USER>@enterprise ../local/wso2bam-2.4.1/lib]$

这是BAM的标准默认安装，根本没有任何操作。

任何人都可以指出解决问题的方法吗？非常感谢！

[解决方案更新]我设法解决了这个问题，不管你信不信...... $ CARBON_CLASSPATH问题与第三方.jar for MySQL。结束将mysql-connector-java-5.1.31-bin.jar文件放在WSO2 BAM安装中的各种/ lib /文件夹中并且它有效。现在是执行的控制台日志，现在数据显示在远程MySQL数据库中。 

Hive history file=/Users/<USER>/local/wso2bam-2.4.1/tmp/hive/wso2-querylogs/hive_job_log_<USER>_201406151106_798138044.txt
OK
OK
OK
Copying data from file:/Users/<USER>/projects/ccost/prod.txt
Copying file: file:/Users/<USER>/projects/ccost/prod.txt
Loading data to table default.producttable
OK
OK
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Execution log at: /Users/<USER>/local/wso2bam-2.4.1/repository/logs//wso2carbon.log
2014-06-15 11:13:07.935 java[2137:1003] Unable to load realm info from SCDynamicStore
[2014-06-15 11:13:08,473]  WARN {org.apache.hadoop.mapred.JobClient} -  Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2014-06-15 11:13:11,720 null map = 0%,  reduce = 0%
2014-06-15 11:13:14,722 null map = 100%,  reduce = 0%
2014-06-15 11:13:17,725 null map = 100%,  reduce = 100%
Ended Job = job_local_0001
Execution completed successfully
Mapred Local Task Succeeded . Convert the Join into MapJoin
OK

因此 - 到目前为止一切顺利。现在谈到下一个问题。

1 个答案:

答案 0 :(得分：0)

您必须将任何第三方库复制到<BAM_HOME>/repository/components/lib文件夹。

相关问题
最新问题