Question

我正在尝试将我的Glue目录连接到EMR中的Presto和Hive。在presto-cli中运行查询时，我收到NullPointerException，而同一查询在hive-cli中成功。

像下面一样开始cli

presto-cli --catalog hive

执行查询时出现异常：

Query 20180814_174636_00003_iika5 failed: java.lang.NullPointerException: parameters is null

EMR配置如下：

[
    {
        "classification": "presto-connector-hive",
        "properties": {
            "hive.metastore": "glue"
        },
        "configurations": []
    },
    {
        "classification": "hive-site",
        "properties": {
            "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
        },
        "configurations": []
    }
]

EMR版本： 5.16.0 Presto版本： 0.203

参考文档：https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-presto-glue.html

调试日志

Query 20180816_060942_00001_m9i52 failed: java.lang.NullPointerException: parameters is null
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException: parameters is null
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2052)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3943)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3967)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4952)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4958)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.get(CachingHiveMetastore.java:207)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.getPartitionNamesByParts(CachingHiveMetastore.java:499)
    at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.doGetPartitionNames(SemiTransactionalHiveMetastore.java:467)
    at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.getPartitionNamesByParts(SemiTransactionalHiveMetastore.java:445)
    at com.facebook.presto.hive.HivePartitionManager.getFilteredPartitionNames(HivePartitionManager.java:284)
    at com.facebook.presto.hive.HivePartitionManager.getPartitions(HivePartitionManager.java:146)
    at com.facebook.presto.hive.HiveMetadata.getTableLayouts(HiveMetadata.java:1305)
    at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.getTableLayouts(ClassLoaderSafeConnectorMetadata.java:73)
    at com.facebook.presto.metadata.MetadataManager.getLayouts(MetadataManager.java:346)
    at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout.planTableScan(PickTableLayout.java:203)
    at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout.access$200(PickTableLayout.java:61)
    at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout$PickTableLayoutWithoutPredicate.apply(PickTableLayout.java:186)
    at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout$PickTableLayoutWithoutPredicate.apply(PickTableLayout.java:153)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.transform(IterativeOptimizer.java:168)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreNode(IterativeOptimizer.java:141)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:104)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:193)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:106)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:193)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:106)
    at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.optimize(IterativeOptimizer.java:95)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:140)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:129)
    at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:327)
    at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:312)
    at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:268)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: parameters is null
    at java.util.Objects.requireNonNull(Objects.java:228)
    at com.facebook.presto.hive.metastore.Partition.<init>(Partition.java:54)
    at com.facebook.presto.hive.metastore.Partition$Builder.build(Partition.java:180)
    at com.facebook.presto.hive.metastore.glue.converter.GlueToPrestoConverter.convertPartition(GlueToPrestoConverter.java:141)
    at com.facebook.presto.hive.metastore.glue.GlueHiveMetastore.lambda$getPartitions$8(GlueHiveMetastore.java:558)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at com.facebook.presto.hive.metastore.glue.GlueHiveMetastore.getPartitions(GlueHiveMetastore.java:558)
    at com.facebook.presto.hive.metastore.glue.GlueHiveMetastore.getPartitionNamesByParts(GlueHiveMetastore.java:541)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadPartitionNamesByParts(CachingHiveMetastore.java:504)
    at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
    at com.google.common.cache.CacheLoader$1.load(CacheLoader.java:188)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2273)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2156)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2046)
    ... 33 more

Answer 1

好像presto 0.203就有这个bug，我也遇到了，我换了一个新的版本就可以了。

在我编写此答案时，EMR 5.17已发布，并且已存储为0.206，从而解决了此问题。

Presto-Glue-EMR集成：presto-cli提供NullPointerException

1 个答案: