Question

我按照说明here在单个节点中安装了CDH5.4，同样，我使用这些instructions将hive-metastore放在localmode中，一切正常，除非我尝试过将pig与metastore：

联系起来

➜  ~  pig -useHCatalog
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2015-05-01 15:45:08,657 [main] INFO  org.apache.pig.Main - Apache Pig version 0.12.0-cdh5.4.0 (rUnversioned directory) compiled Apr 21 2015, 12:19:15
2015-05-01 15:45:08,658 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/itam/pig_1430495108571.log
2015-05-01 15:45:09,035 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:09,035 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:09,035 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:8020
2015-05-01 15:45:09,940 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:09,941 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost:8021
2015-05-01 15:45:09,941 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:09,999 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:10,001 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:10,088 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:10,089 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:10,125 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:10,126 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:10,160 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:10,162 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:10,194 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:10,195 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:10,227 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:10,228 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:10,261 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:10,262 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-05-01 15:45:10,295 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-01 15:45:10,296 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

当我试图访问该表时：

grunt> a = load 'ufos' using org.apache.hcatalog.pig.HCatLoader();
2015-05-01 15:46:11,656 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Details at logfile: /home/itam/pig_1430495108571.log
grunt>

Hadoop版本

➜  ~  hadoop version
Hadoop 2.6.0-cdh5.4.0
Subversion http://github.com/cloudera/hadoop -r c788a14a5de9ecd968d1e2666e8765c5f018c271 
Compiled by jenkins on 2015-04-21T19:16Z
Compiled with protoc 2.5.0
From source with checksum cd78f139c66c13ab5cee96e15a629025
This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.4.0.jar

更新：我刚尝试使用Impala，它既没有看到任何内容：

➜  ~  impala-shell                                                                                                                                                                                                                             
/usr/lib/python2.7/dist-packages/pkg_resources.py:1049: UserWarning: /home/itam/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extracti
on_path or the PYTHON_EGG_CACHE environment variable).
  warnings.warn(msg, UserWarning)
Starting Impala Shell without Kerberos authentication
Connected to 6b512e41337d:21000
Server version: impalad version 2.2.0-cdh5 RELEASE (build 2ffd73a4255cefd521362ffe1cfb37463f67f75c)
Welcome to the Impala shell. Press TAB twice to see a list of available commands.

Copyright (c) 2012 Cloudera, Inc. All rights reserved.

(Shell build version: Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015)
[6b512e41337d:21000] > invalidate metadata;
Query: invalidate metadata
[6b512e41337d:21000] > show tables;
Query: show tables

Fetched 0 row(s) in 0.00s

但来自beeline：

~  beeline -u jdbc:hive2://
scan complete in 2ms
Connecting to jdbc:hive2://
Connected to: Apache Hive (version 1.1.0-cdh5.4.0)
Driver: Hive JDBC (version 1.1.0-cdh5.4.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.4.0 by Apache Hive
0: jdbc:hive2://> show tables;
OK
+-----------+--+
| tab_name  |
+-----------+--+
| ufos      |
+-----------+--+
1 row selected (0.701 seconds)

它有效......发生了什么事？

更新：我正在运行hcatalog

➜  ~  sudo service hive-webhcat-server status
 * WEBHCat server is running

➜  ~  hcat -e "desc ufos"                    
OK
timestamp               string                  from deserializer   
city                    string                  from deserializer   
state                   string                  from deserializer   
shape                   string                  from deserializer   
duration                string                  from deserializer   
summary                 string                  from deserializer   
posted                  string                  from deserializer   
Time taken: 1.314 seconds

更新：impala的问题是由于我没有将hive-site.xml复制到/etc/impala/conf，一旦完成，impala-shell工作得很好。

Answer 1

您弃用的加载程序已弃用。您需要使用org.apache.hcatalog.pig.HCatLoader。

，而不是使用org.apache.hive.hcatalog.pig.HCatLoader

来自org.apache.hcatalog.pig.HCatLoader：

已过时。使用/修改HCatLoader代替

Answer 2

我在HDP 2.3和Pig 0.15中遇到了这个问题。

HCatLoader（）类的包名称在Hortonworks发行版中有所不同。

以下为我工作

使用org.apache.hive.hcatalog.pig.HCatLoader（）

而不是使用org.apache.hcatalog.pig.HCatLoader（）;

Answer 3

就像你开始看到hive-site.xml文件所带来的问题一样 - 你需要把它放在类路径中

提到here：

与HCatalog交互的工作流操作需要以下内容   类路径中的jar：hcatalog-core.jar，webhcat-java-client.jar，   hive-common.jar，hive-exec.jar，hive-metastore.jar，hive-serde.jar和   libfb303.jar。 hive-site.xml，它具有与之通信的配置   HCatalog服务器也需要在类路径中。正确的版本   应该将HCatalog和hive jar放在基于类的路径中   集群上安装的HCatalog版本。

可以使用其中一个将jar添加到操作的类路径中   以下方式。


您可以将jar和hive-site.xml放在系统共享库中。   可以覆盖pig，hive或java操作的共享库   包括hcatalog共享库以及共享的操作   图书馆。有关更多信息，请参阅共享库。该   oozie-sharelib- [版本] .tar.gz在oozie发行版中捆绑了   在hcatalog sharelib中需要HCatalog jar。如果使用不同的   HCatalog的版本比sharelib中捆绑的版本复制一样   需要将HCatalog jar从此类版本导入到sharelib中。

你可以   将jar和hive-site.xml放在工作流应用程序lib /中   路径。

您可以在archive标签中指定jar文件的位置   相应的pig，hive或java中的文件标记中的hive-site.xml   动作。

如果您打算使用Oozie协调员，请将它们上传到HDFS协调员路径

标记-useHCatalog无效

3 个答案: