Question

我必须在hive中添加以下UDF：

package com.hadoopbook.hive;

import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public class Strip extends UDF {
  private Text result = new Text();

  public Text evaluate(Text str) {
    if (str == null) {
      return null;
    }
    result.set(StringUtils.strip(str.toString()));
    return result;
  }

  public Text evaluate(Text str, String stripChars) {
    if (str == null) {
      return null;
    }
    result.set(StringUtils.strip(str.toString(), stripChars));
    return result;
  }
}

这是“Hadoop：The definitive guide”一书中的一个例子

我使用以下命令创建了上述java文件的.class文件：

hduser@nb-VPCEH35EN:~/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive$ javac Strip.java

然后我使用以下命令创建了jar文件：

hduser@nb-VPCEH35EN:~/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive$ jar cvf Strip.jar Strip Strip.class 
Strip : no such file or directory
added manifest
adding: Strip.class(in = 915) (out= 457)(deflated 50%)

我将带有geenrated的jar文件添加到hdfs目录：

hduser@nb-VPCEH35EN:~/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive$ hadoop dfs -copyFromLocal /home/hduser/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive/Strip.jar /user/hduser/input

我尝试使用以下命令创建UDf：

hive> create function strip as 'com.hadoopbook.hive.Strip' using jar 'hdfs://localhost/user/hduser/input/Strip.jar';

但是我收到了如下错误：

转换为本地hdfs：//localhost/user/hduser/input/Strip.jar已添加 [/tmp/hduser_resources/Strip.jar]到类路径添加资源： [hdfs：//localhost/user/hduser/input/Strip.jar]注册失败 default.strip using class com.hadoopbook.hive.Strip FAILED：Execution 错误，从org.apache.hadoop.hive.ql.exec.FunctionTask返回代码1

我也尝试创建临时功能。所以我首先使用：

将jar文件添加到hive

hive> add jar hdfs://localhost/user/hduser/input/Strip.jar;
converting to local hdfs://localhost/user/hduser/input/Strip.jar
Added [/tmp/hduser_resources/Strip.jar] to class path
Added resources: [hdfs://localhost/user/hduser/input/Strip.jar]

然后我尝试添加临时功能：

hive> create temporary function strip as 'com.hadoopbook.hive.Strip';

但是我收到了以下错误：

FAILED：类com.hadoopbook.hive.Strip not found FAILED：执行错误，从org.apache.hadoop.hive.ql.exec.FunctionTask返回代码1

成功创建了jar文件并将其添加到hive中。它显示未找到该类。任何人都可以告诉它有什么问题吗？

Answer 1

是的，使用像eclipse这样的IDE很容易，然后从CLI制作jar。

从命令行创建jar文件，您必须按照以下步骤操作：

首先在项目目录ch17-hive下制作项目目录：

bin - 将存储.class（Strip.class）文件
lib - 将存储所需的外部罐子

traget - 将存储您将创建的罐子

[ch17-hive]$ mkdir bin lib traget
[ch17-hive]$ ls
bin  lib  src  target

将所需的外部广告投放到ch170hive/lib dir：

[ch17-hive]$ cp /usr/lib/hive/lib/hive-exec.jar lib/.
[ch17-hive]$ cp /usr/lib/hadoop/hadoop-common.jar lib/.

现在从您的类com.hadoopbook.hive.Strip所在的目录编译java，在您的情况下为ch17-hive/src/main/java：

[java]$ pwd
/home/cloudera/ch17-hive/src/main/java
[java]$ javac  -d ../../../bin -classpath ../../../lib/hive-exec.jar:../../../lib/hadoop-common.jar com/hadoopbook/hive/Strip.java

创建清单文件：

[ch17-hive]$ cat MENIFEST.MF 
Main-Class: com.hadoopbook.hive.Strip
Class-Path: lib/hadoop-common.jar  lib/hive-exec.jar

创建jar

[ch17-hive]$ jar cvfm target/strip.jar MENIFEST.MF -C bin .added manifest
adding: com/(in = 0) (out= 0)(stored 0%)
adding: com/hadoopbook/(in = 0) (out= 0)(stored 0%)
adding: com/hadoopbook/hive/(in = 0) (out= 0)(stored 0%)
adding: com/hadoopbook/hive/Strip.class(in = 915) (out= 456)(deflated 50%)

现在您的项目结构应如下所示：

[ch17-hive]$ ls *
MENIFEST.MF

bin:
com

lib:
hadoop-common.jar  hive-exec.jar

src:
main

target:
strip.jar

将创建的jar复制到hdfs：

hadoop fs -put /home/cloudera/ch17-hive/target/strip.jar /user/cloudera/.

在HIVE中使用它

hive> create function strip_new as 'com.hadoopbook.hive.Strip' using jar 'hdfs:/user/cloudera/strip.jar';
converting to local hdfs:/user/cloudera/strip.jar
Added [/tmp/05a13d23-8051-431f-a354-793abac66160_resources/strip.jar] to class path
Added resources: [hdfs:/user/cloudera/strip.jar]
OK
Time taken: 0.071 seconds
hive>

无法在配置单元

1 个答案: