Question

自定义功能我所做的并不适用于EMR。

我想创建一个只返回42的简单UDAF。

首先，我的自定义函数是我编写的一个简单函数，但在presto中没有工作。

presto-cli：

中出现错误

presto> select answer_to_life('the universe');
Query 20180324_120433_00000_7n6s6 failed: answer_to_life(varchar):bigint not found
com.facebook.presto.spi.PrestoException: answer_to_life(varchar):bigint not found
        at com.facebook.presto.metadata.FunctionRegistry.doGetSpecializedFunctionKey(FunctionRegistry.java:972)
        at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:146)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
        at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:5153)
        at com.facebook.presto.metadata.FunctionRegistry.getSpecializedFunctionKey(FunctionRegistry.java:898)
        at com.facebook.presto.metadata.FunctionRegistry.getAggregateFunctionImplementation(FunctionRegistry.java:875)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.buildAccumulatorFactory(LocalExecutionPlanner.java:1973)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.planGlobalAggregation(LocalExecutionPlanner.java:1984)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:955)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:1919)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:196)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:952)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:638)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:393)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:324)
        at com.facebook.presto.execution.SqlTaskExecution.<init>(SqlTaskExecution.java:161)
        at com.facebook.presto.execution.SqlTaskExecution.createSqlTaskExecution(SqlTaskExecution.java:121)
        at com.facebook.presto.execution.SqlTaskExecutionFactory.create(SqlTaskExecutionFactory.java:71)
        at com.facebook.presto.execution.SqlTask.updateTask(SqlTask.java:340)
        at com.facebook.presto.execution.SqlTaskManager.updateTask(SqlTaskManager.java:321)
        at com.facebook.presto.server.TaskResource.createOrUpdateTask(TaskResource.java:128)
        at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
        at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:454)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:564)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
        at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
        at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
        at java.lang.Thread.run(Thread.java:748)

AggregationFunction代码如下。

我参考了这个presto code

package sample;  

import com.facebook.presto.spi.block.BlockBuilder;  
import com.facebook.presto.spi.function.*;  
import com.facebook.presto.spi.type.BigintType;  
import com.facebook.presto.spi.type.StandardTypes;  
import io.airlift.slice.Slice;  

@AggregationFunction("answer_to_life")  
public final class AnswerToLife {  
    private AnswerToLife() {  

    }  

   @InputFunction  
   public static void input(@AggregationState NullState state, @SqlType(StandardTypes.VARCHAR) Slice value) {  

    }  

   @CombineFunction  
   public static void combine(@AggregationState NullState state, @AggregationState NullState other) {  

    }  

    @OutputFunction(StandardTypes.BIGINT)  
    public static void output(@AggregationState NullState state, BlockBuilder out) {  
        BigintType.BIGINT.writeLong(out, 42);  
   }  
}

详细代码在这里（https://github.com/asari-mtr/presto-udaf/tree/stackoverflow）。

我部署的结构如下：

$ ls -1 /usr/lib/presto/plugin/my-udaf/
commons-codec-1.4.jar
guava-21.0.jar
hive-udf-1.0-SNAPSHOT.jar
presto-array-0.197.jar
stats-0.155.jar
presto-udaf-1.0-SNAPSHOT.jar

我使用emr-5.12.0（Presto 0.188）

感谢您的时间。

Edited1

列在jar文件中。

% jar -tf target/presto-udaf-1.0-SNAPSHOT.jar 
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
META-INF/
META-INF/MANIFEST.MF
META-INF/services/
sample/
META-INF/services/com.facebook.presto.spi.Plugin
sample/AnswerToLife.class
sample/AnswerToLifePlugin.class
sample/NullState.class
META-INF/maven/
META-INF/maven/sample/
META-INF/maven/sample/presto-udaf/
META-INF/maven/sample/presto-udaf/pom.xml
META-INF/maven/sample/presto-udaf/pom.properties

sever.log

$ grep answer -i  /mnt/var/log/presto/server.log
2018-03-26T06:37:14.213Z        INFO    main    com.facebook.presto.server.PluginManager        Installing sample.AnswerToLifePlugin
2018-03-26T06:37:14.214Z        INFO    main    com.facebook.presto.server.PluginManager        Registering functions from sample.AnswerToLife

执行show funcstions

presto> show functions;
            Function             |                     Return Type                     |                                 Argument Types
---------------------------------+-----------------------------------------------------+------------------------------------------------------------------------------
 ST_Area                         | double                                              | Geometry
 ST_AsText                       | varchar                                             | Geometry
 ...
 answer_to_life                  | bigint                                              | varchar
 ...

Edited2

删除src/main/resources/META-INF/services/com.facebook.presto.spi.Plugin

将<packaging>presto-plugin</packaging>添加到pom.xml

https://github.com/asari-mtr/presto-udaf/commit/f2a2ddbf0339e08f418b378a7ead511020e98a3b

我在/usr/lib/presto/plugin/下部署了由Maven制作的拉链。

但是，错误的内容不会改变。

Edited3

我在我的Mac上从github（分支0.188）获得了源代码并构建了presto。

当我们将上面的UDAF放在它的presto上时，它工作得很好。

对于EMR上的presto安装过程可能存在错误。

Answer 1

解决!!

Custom插件不起作用，因为它不是一个插件创建，但是在EMR上部署的方法是错误的。

由于我没有在我执行的过程中部署到所有节点，因此我通过将以下shell注册为引导操作来确认操作。

#/bin/bash
aws s3 cp s3://hogehoge/presto-udaf-1.0-SNAPSHOT.zip /tmp/
sudo mkdir -p /usr/lib/presto/plugin/
sudo unzip -d /usr/lib/presto/plugin/ /tmp/presto-udaf-1.0-SNAPSHOT.zip

10.1. SPI Overview — Presto 0.198 Documentation

必须在Presto群集中的所有节点上安装插件（协调员和工人）。

Create Bootstrap Actions to Install Additional Software - Amazon EMR

您可以使用引导操作将对象从Amazon S3复制到每个对象安装应用程序之前群集中的节点。 AWS CLI 安装在群集的每个节点上，因此您的引导操作可以调用AWS CLI命令。

Answer 2

根据你所写的内容，有两件事情遗失了。

确保您在pom.xml

presto-plugin

还实现com.facebook.presto.spi.Plugin，getFunctions方法会返回您的函数。

查看presto-geospatial Presto模块，查看com.facebook.presto.plugin.geospatial.GeoPlugin及其pom.xml作为参考。

为什么在presto的自定义UDAF中找不到返回类型

2 个答案: