https://pastebin.com/1AGTVSjx

Question

当我跑步时：

SELECT anum as anum, COUNT(*) as callCount, SUM(dur*1) as callDuration, COUNT(distinct bnum) as voiceBnumCount, COUNT(distinct cdrref) as longCallCount
FROM hdfs.`/data/files/{171221}*` as t1 WHERE `t1`.`direction` IN ('up') AND `t1`.`anum_code` IN ('FIRSTANUM') 
AND CONCAT(sdate, ' ', stime) BETWEEN '17.12.21 14.39.58' AND '17.12.21 17.40.00' 
GROUP BY (anum) HAVING COUNT(*) >= 30 AND SUM(dur*1)/60 >= 150
LIMIT 10

它举起

org.apache.drill.common.exceptions.UserRemoteException：系统错误：IllegalStateException：未为该向量容器设置记录计数片段0：0 [错误ID：master上的错误编码为：1c199e3b-d680-4264-88c5-437a97516a75：31010]

简单的错误日志：

2018-11-29 20:38:42,552 [2400141c-c721-c327-9199-19ec95ea5b62:foreman] INFO  o.a.drill.exec.work.foreman.Foreman - Query text for query id 2400141c-c721-c327-9199-19ec95ea5b62: SELECT anum as anum, COUNT(*) as callCount, SUM(dur*1) as callDuration, COUNT(distinct bnum) as voiceBnumCount, COUNT(distinct cdrref) as longCallCount
FROM hdfs.`/data/files/{171221}*` as t1 WHERE `t1`.`direction` IN ('moc') AND `t1`.`anum_code` IN ('62_XL') 
AND CONCAT(sdate, ' ', stime) BETWEEN '17.12.21 14.39.58' AND '17.12.21 17.40.00' 
GROUP BY (anum) HAVING COUNT(*) >= 30 AND SUM(dur*1)/60 >= 150
LIMIT 10
2018-11-29 20:38:42,665 [2400141c-c721-c327-9199-19ec95ea5b62:frag:4:1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:4:1: State change requested AWAITING_ALLOCATION --> RUNNING
...
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:5:3] INFO  o.a.d.e.w.f.FragmentStatusReporter - 2400141c-c721-c327-9199-19ec95ea5b62:5:3: State to report: FINISHED
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO  o.a.d.e.p.impl.join.HashJoinBatch - Memory limit 40 MB bytes
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO  o.a.d.e.p.impl.join.HashJoinBatch - Memory limit 40 MB bytes
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter - 2400141c-c721-c327-9199-19ec95ea5b62:0:0: State to report: RUNNING
...
2018-11-29 20:38:44,218 [2400141c-c721-c327-9199-19ec95ea5b62:frag:2:0] INFO  o.a.d.e.w.f.FragmentStatusReporter - 2400141c-c721-c327-9199-19ec95ea5b62:2:0: State to report: FINISHED
2018-11-29 20:38:44,232 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO  o.a.d.e.p.i.u.UnorderedReceiverBatch - Informing senders of request to terminate sending.
2018-11-29 20:38:44,232 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] WARN  o.a.d.e.w.b.ControlMessageHandler - Dropping request for early fragment termination for path 2400141c-c721-c327-9199-19ec95ea5b62:1:0 -> 2400141c-c721-c327-9199-19ec95ea5b62:0:0 as path to executor unavailable.
2018-11-29 20:38:44,406 [2400141c-c721-c327-9199-19ec95ea5b62:frag:6:1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:6:1: State change requested RUNNING --> FINISHED
2018-11-29 20:38:44,406 [2400141c-c721-c327-9199-19ec95ea5b62:frag:6:1] INFO  o.a.d.e.w.f.FragmentStatusReporter - 2400141c-c721-c327-9199-19ec95ea5b62:6:1: State to report: FINISHED
...
2018-11-29 20:38:44,963 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:0:0: State change requested RUNNING --> FAILED
2018-11-29 20:38:44,965 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:0:0: State change requested FAILED --> FINISHED
2018-11-29 20:38:44,965 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Record count not set for this vector container

Fragment 0:0

[Error Id: 1c199e3b-d680-4264-88c5-437a97516a75 on master:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Record count not set for this vector container

Fragment 0:0

[Error Id: 1c199e3b-d680-4264-88c5-437a97516a75 on master:31010]
    at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) [drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) [drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) [drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0.jar:1.14.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalStateException: Record count not set for this vector container
    at com.google.common.base.Preconditions.checkState(Preconditions.java:173) ~[guava-18.0.jar:na]
    at org.apache.drill.exec.record.VectorContainer.getRecordCount(VectorContainer.java:394) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.RecordBatchSizer.<init>(RecordBatchSizer.java:714) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.RecordBatchSizer.<init>(RecordBatchSizer.java:698) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.getActualSize(HashTableTemplate.java:442) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.common.HashTableTemplate.getActualSize(HashTableTemplate.java:901) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.common.HashTableTemplate.makeDebugString(HashTableTemplate.java:910) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.common.HashPartition.makeDebugString(HashPartition.java:580) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.join.HashJoinBatch.makeDebugString(HashJoinBatch.java:1017) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.join.HashJoinBatch.executeBuildPhase(HashJoinBatch.java:829) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:415) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:87) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:87) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:294) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:281) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_181]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[hadoop-common-2.7.1.jar:na]
    at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:281) [drill-java-exec-1.14.0.jar:1.14.0]
    ... 4 common frames omitted
2018-11-29 20:38:44,969 [qtp717308094-635] ERROR o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Record count not set for this vector container

Fragment 0:0

[Error Id: 1c199e3b-d680-4264-88c5-437a97516a75 on master:31010]
    at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:72) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:87) ~[drill-java-exec-1.14.0.jar:1.14.0]
    at sun.reflect.GeneratedMethodAccessor189.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.8.jar:na]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) [jersey-common-2.8.jar:na]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023) [jersey-server-2.8.jar:na]
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372) [jersey-container-servlet-core-2.8.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382) [jersey-container-servlet-core-2.8.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:345) [jersey-container-servlet-core-2.8.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:220) [jersey-container-servlet-core-2.8.jar:na]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.Server.handle(Server.java:462) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2018-11-29 20:38:44,973 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] WARN  o.a.d.e.w.f.QueryStateProcessor - Dropping request to move to COMPLETED state as query is already at FAILED state (which is terminal).

完整的错误日志：

https://pastebin.com/1AGTVSjx

那只是在开发机器中。我认为这是因为仅内存配置问题（？）。

在生产模式下，使用相同的查询，错误是（无法提供更多，没有访问权限）：

org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Unable to allocate buffer of size 8192 due to memory limit (41943040). Current allocation: 41943040 Fragment 3:0 [Error Id: 45ab7e51-b563-4fff-9908-33208b895ee6 on MASTER:31010]

如果我做这样的事情（将列划分为join），它将起作用：

SELECT t1.anum as anum, COUNT(*) as callCount, SUM(dur*1) as callDuration, COUNT(distinct bnum) as voiceBnumCount, t2.longCallCount
FROM hdfs4.`/cdr2p2/{181129}*` as t1 
JOIN(
    SELECT anum as anum, COUNT(distinct cdrref) as longCallCount
    FROM hdfs4.`/cdr2p2/{181129}*` as t1 WHERE `t1`.`direction` IN ('UP') AND `t1`.`anum_code` IN ('FIRSTANUM') 
    AND CONCAT(sdate, ' ', stime) BETWEEN '18.11.29 14.39.58' AND '18.11.29 17.40.00' 
    GROUP BY (anum) HAVING COUNT(*) >= 30 AND SUM(dur*1)/60 >= 150
) as t2 ON t2.anum = t1.anum
WHERE `t1`.`direction` IN ('UP') AND `t1`.`anum_code` IN ('FIRSTANUM') 
AND CONCAT(sdate, ' ', stime) BETWEEN '18.11.29 14.39.58' AND '18.11.29 17.40.00' 
GROUP BY (t1.anum, t2.longCallCount) HAVING COUNT(*) >= 30 AND SUM(dur*1)/60 >= 150
LIMIT 10

是什么原因造成的？是因为仅1个分组时聚合过多？我该如何解决？

Answer 1

看起来像个错误。此外，DRILL-6027之后可以进行回归。有必要在Drill 1.13上进行验证。

请创建一张Jira票证，并提供日志，逻辑计划和重现该问题的最少数据。 https://issues.apache.org/jira/projects/DRILL

未为此向量容器设置记录计数

https://pastebin.com/1AGTVSjx

1 个答案: