Question

我正在尝试通过我们的EMR启动配置JSON部署EMR。我已经决定了github issue of presto中建议的配置属性。我在启动配置中添加了以下presto属性

{
    "Classification": "presto-connector-hive",
    "Properties": {
        "hive.metastore.glue.datacatalog.enabled": "true",
        "hive.table-statistics-enabled": "true"
    },
    "Configurations": []
},
{
    "Classification": "presto-config",
    "Properties": {
        "query.max-memory": "150G",
        "query.max-memory-per-node": "20G",
        "query.max-total-memory-per-node": "30G",
        "memory.heap-headroom-per-node": "10G",
        "query.initial-hash-partitions": "15"
    },
    "Configurations": []
}

EMR集群已创建，但由于以下错误导致presto失败

1) Explicit bindings are required and com.facebook.presto.memory.LowMemoryKiller is not explicitly bound.
  while locating com.facebook.presto.memory.LowMemoryKiller
    for parameter 7 at com.facebook.presto.memory.ClusterMemoryManager.<init>(ClusterMemoryManager.java:123)
  at com.facebook.presto.server.CoordinatorModule.setup(CoordinatorModule.java:189) (via modules: com.facebook.presto.server.ServerMainModule -> com.facebook.presto.server.CoordinatorModule)

2) Error: Could not coerce value '150G' to io.airlift.units.DataSize (property 'query.max-memory') in order to call [public com.facebook.presto.memory.MemoryManagerConfig com.facebook.presto.memory.MemoryManagerConfig.setMaxQueryMemory(io.airlift.units.DataSize)]

3) Error: Could not coerce value '20G' to io.airlift.units.DataSize (property 'query.max-memory-per-node') in order to call [public com.facebook.presto.memory.NodeMemoryConfig com.facebook.presto.memory.NodeMemoryConfig.setMaxQueryMemoryPerNode(io.airlift.units.DataSize)]

4) Configuration property 'memory.heap-headroom-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

5) Configuration property 'query.max-memory' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

6) Configuration property 'query.max-memory-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

7) Configuration property 'query.max-total-memory-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

7 errors
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466)
    at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
    at com.google.inject.Guice.createInjector(Guice.java:96)
    at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:241)
    at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:114)
    at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:66)

我的config.properties文件

coordinator=true
node-scheduler.include-coordinator=false
discovery.uri=X.X.X.X:YYYY
http-server.threads.max=500
discovery-server.enabled=true
sink.max-buffer-size=1GB
query.max-memory=150G
query.max-memory-per-node=20G
query.max-history=40
query.min-expire-age=30m
http-server.http.port=8889
http-server.log.path=/var/log/presto/http-request.log
http-server.log.max-size=67108864B
http-server.log.max-history=5
log.max-size=268435456B
log.max-history=5
query.initial-hash-partitions = 15
memory.heap-headroom-per-node = 10G
query.max-total-memory-per-node = 30G

Answer 1

安装失败，因为

设置数据大小配置属性时，您需要使用“ GB”（而非“ G”）作为单位
您的版本（0.194）不支持您正在设置的某些属性（memory.heap-headroom-per-node或query.max-total-memory-per-node）。

通过emr启动配置进行presto配置

1 个答案: