Question

I m using sqoop 1.4.5-cdh5.2.1 and oracle .

I m importing a small set of records of 115k from oracle . Sqoop command works fine on setting --num-mappers to 5. But when i set it to more than 5 , I get an error of JAVA HEAP SPACE.

Can any one tell this ,that why its happening so.

LOG Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.math.BigInteger.(BigInteger.java:394) at java.math.BigDecimal.bigTenToThe(BigDecimal.java:3380) at java.math.BigDecimal.bigDigitLength(BigDecimal.java:3635) at java.math.BigDecimal.precision(BigDecimal.java:2189) at java.math.BigDecimal.compareMagnitude(BigDecimal.java:2585) at java.math.BigDecimal.compareTo(BigDecimal.java:2566) at org.apache.sqoop.mapreduce.db.BigDecimalSplitter.split(BigDecimalSplitter.java:138) at org.apache.sqoop.mapreduce.db.BigDecimalSplitter.split(BigDecimalSplitter.java:69) at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:171) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:498) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:515) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:399) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313) at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:198) at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:171) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:268) at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:721) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:499) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) 2015-06-25 13:48:59 STATUS: 1 2015-06-25 13:48:59 ERROR Error (1) Sqoop failed. 2015-06-25 13:48:59 ERROR Error (1) run_sqoop

Answer 1

默认情况下，每个map和reduce任务都在自己的JVM中运行。因此，每个映射器将消耗一定量的物理内存。随着映射器数量的不断增加，内存需求也将不断增长。如果java进程无法分配足够的内存，则抛出java.lang.OutOfMemoryError

在您的情况下，系统（或VM，如果您正在运行VM）可能只有最多5个映射器的内存。

您可以在启动＆gt; 5个映射器并监视可用内存时运行top命令。

Answer 2

尝试在$ HADOOP_HOME / conf / mapred-site.xml上添加属性，如下所示

<property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>4.2</value>
</property>

调整您的环境，可能加上或减去该值。记得修改每个节点。

或修改纱线网站上的虚拟内存限制

Shape Component

默认2.1G

Increasing number of mappers in sqoop command gives java heap space error

2 个答案: