Question

我正在检查如何在Hive中创建自定义UDF。我创建了一个不做任何事情的自定义UDF，只是按原样返回给定的文本。

我能够在没有任何问题的情况下在hive中加载此JAR
我也可以从这个jar创建函数，我可以执行这个功能。

问题

如果添加了这个Jar，我就无法从另一个表执行load table / hdfs。以下简单查询失败。

insert into demo1 select * from demo;

堆栈跟踪：

Vertex failed, vertexName=Map 1, vertexId=vertex_1523501275422_0010_3_00, diagnostics=[Task failed, taskId=task_1523501275422_0010_3_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_1523501275422_0010_01_000007 finished with diagnostics set to [Container completed. ]], TaskAttempt 1 failed, info=[Container container_1523501275422_0010_01_000008 finished with diagnostics set to [Container completed. ]], TaskAttempt 2 failed, info=[Container container_1523501275422_0010_01_000009 finished with diagnostics set to [Container completed. ]], TaskAttempt 3 failed, info=[Container container_1523501275422_0010_01_000010 finished with diagnostics set to [Container completed. ]]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1523501275422_0010_3_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0

请注意，两个表都具有相同的结构。

如果我删除jar，那么我可以执行上述查询而不会出现任何问题。

代码： 的 ca.abc.demo

public class demo extends UDF {

    public String evaluate(String s) {
        if (s == null) {
            return null;
        }else{
            return s;
        }

    }
}

的pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>ca.cantire</groupId>
    <artifactId>hive-normalize</artifactId>
    <version>1.0</version>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.hive/hive-exec -->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>2.3.3</version>
        </dependency>



    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.8</version>
                </plugin>
                <plugin>
                    <artifactId>maven-assembly-plugin</artifactId>
                    <configuration>
                        <archive>
                            <manifest>
                                <mainClass>ca.abc.demo</mainClass>
                            </manifest>
                        </archive>
                        <descriptorRefs>
                            <descriptorRef>jar-with-dependencies</descriptorRef>
                        </descriptorRefs>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>

Hive自定义UDF jar为其他插入查询

0 个答案: