Question

我正在尝试从存储在mysql中的表创建一个镶木地板文件。源包含数百万行，并在几分钟后得到GC Overhead限制异常。

是否可以在允许操作暂时使用磁盘的情况下配置apache drill以防止没有更多RAM可用？

这是我在收到错误之前的步骤：

将mysql jdbc连接器放在jars / 3rdparty
执行sqlline.bat -u“jdbc：drill：zk = local”
导航至http://localhost:8047/storage
配置新的存储插件以连接到mysql
导航至http://localhost:8047/query并执行以下查询
ALTER SESSION SET`store.format` ='parquet';
ALTER SESSION SET`store.parquet.compression` ='snappy';
create table dfs.tmp .bigtable.parquet` as（select * from mysql.schema.bigtable）

然后我收到错误并且申请结束：

节点用完堆内存，退出。 java.lang.OutOfMemoryError：超出了GC开销限制在com.mysql.jdbc.MysqlIO.nextRowFast（MysqlIO.java:2149）在com.mysql.jdbc.MysqlIO.nextRow（MysqlIO.java:1956）在com.mysql.jdbc.MysqlIO.readSingleRowSet（MysqlIO.java:3308）在com.mysql.jdbc.MysqlIO.getResultSet（MysqlIO.java:463）在com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate（MysqlIO.java:3032）在com.mysql.jdbc.MysqlIO.readAllResults（MysqlIO.java:2280）在com.mysql.jdbc.MysqlIO.sqlQueryDirect（MysqlIO.java:2673）在com.mysql.jdbc.ConnectionImpl.execSQL（ConnectionImpl.java:2546）在com.mysql.jdbc.ConnectionImpl.execSQL（ConnectionImpl.java:2504）在com.mysql.jdbc.StatementImpl.executeQuery（StatementImpl.java:1370）在org.apache.commons.dbcp.DelegatingStatement.executeQuery（DelegatingStatement.java:208）在org.apache.commons.dbcp.DelegatingStatement.executeQuery（DelegatingStatement.java:208）在org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup（JdbcRecordReader.java:177）在org.apache.drill.exec.physical.impl.ScanBatch。（ScanBatch.java:101）在org.apache.drill.exec.physical.impl.ScanBatch。（ScanBatch.java:128）在org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch（JdbcBatchCreator.java:40）在org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch（JdbcBatchCreator.java:33）在org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch（ImplCreator.java:151）在org.apache.drill.exec.physical.impl.ImplCreator.getChildren（ImplCreator.java:174）在org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch（ImplCreator.java:131）在org.apache.drill.exec.physical.impl.ImplCreator.getChildren（ImplCreator.java:174）在org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch（ImplCreator.java:131）在org.apache.drill.exec.physical.impl.ImplCreator.getChildren（ImplCreator.java:174）在org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch（ImplCreator.java:131）在org.apache.drill.exec.physical.impl.ImplCreator.getChildren（ImplCreator.java:174） at org.apache.drill.exec.physical.impl.ImplCreator.getRootExec（ImplCreator.java:105）在org.apache.drill.exec.physical.impl.ImplCreator.getExec（ImplCreator.java:79）在org.apache.drill.exec.work.fragment.FragmentExecutor.run（FragmentExecutor.java:230）在org.apache.drill.common.SelfCleaningRunnable.run（SelfCleaningRunnable.java:38）在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1145） at java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:615）在java.lang.Thread.run（Thread.java:745）

任何帮助都将是apreciated

干杯

Answer 1

点击drill-env.sh

中的<drill_installation_directory>/conf

默认值为：

DRILL_MAX_DIRECT_MEMORY="8G"
DRILL_HEAP="4G"

Drillbit的默认内存为8G，但Drill更喜欢16G或更多，具体取决于工作负载。

如果您有足够的RAM，可以将其配置为16G

您可以在Drill's documentation中详细阅读。

如何创建大于节点的已分配内存的镶木地板文件？

1 个答案: