Question

我正在尝试使用oozie并行运行两个sqoop作业。但两个工作岗位在95％之后停滞不前，其他两个工作岗位处于接受状态。我也增加了纱线资源的最大内存。还添加了

<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
 in mapred-site.xml , but nothing helped. please help.

Yarn Cluster Metrix:
Apps Submitted          4
Apps Pending            2
Apps Running            2
Apps Completed          0
Containers Running      4
Memory Used             10GB
Memory Total            32GB
Memory Reserved         0B
VCores Used             4
VCores Total            24
VCores Reserved         0
Active Nodes            4
Decommissioned Nodes    0
Lost Nodes              0
Unhealthy Nodes         0
Rebooted Nodes          0
----------
Sysout Log 
========================================================================
3175 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
  3198 [main] INFO  org.apache.sqoop.Sqoop  - Running Sqoop version: 1.4.5-cdh5.2.0
  3212 [main] WARN  org.apache.sqoop.tool.BaseSqoopTool  - Setting your password on the command-line is insecure. Consider using -P instead.
  3213 [main] INFO  org.apache.sqoop.tool.BaseSqoopTool  - Using Hive-specific delimiters for output. You can override
  3213 [main] INFO  org.apache.sqoop.tool.BaseSqoopTool  - delimiters with --fields-terminated-by, etc.
  3224 [main] WARN  org.apache.sqoop.ConnFactory  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
  3280 [main] INFO  org.apache.sqoop.manager.oracle.OraOopManagerFactory  - Data Connector for Oracle and Hadoop is disabled.
  3293 [main] INFO  org.apache.sqoop.manager.SqlManager  - Using default fetchSize of 1000
  3297 [main] INFO  org.apache.sqoop.tool.CodeGenTool  - Beginning code generation
  3951 [main] INFO  org.apache.sqoop.manager.OracleManager  - Time zone has been set to GMT
  4023 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM PT_PRELIM_FINDING_V t WHERE 1=0
  4068 [main] INFO  org.apache.sqoop.orm.CompilationManager  - HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hadoop-mapreduce
  5925 [main] INFO  org.apache.sqoop.orm.CompilationManager  - Writing jar file: /tmp/sqoop-nobody/compile/0dab11f6545d8ef69d6dd0f6b9041a50/PT_PRELIM_FINDING_CYTOGEN_V.jar
  5937 [main] INFO  org.apache.sqoop.mapreduce.ImportJobBase  - Beginning import of PT_PRELIM_FINDING_V
  5962 [main] INFO  org.apache.sqoop.manager.OracleManager  - Time zone has been set to GMT
  5981 [main] WARN  org.apache.sqoop.mapreduce.JobBase  - SQOOP_HOME is unset. May not be able to find all job dependencies.
  6769 [main] INFO  org.apache.sqoop.mapreduce.db.DBInputFormat  - Using read commited transaction isolation
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat 
  Heart beat

Answer 1

谢谢@abeaamase。

我要求我们的DBA将oracle数据库最大进程增加到750，并将max session pool增加到大约1.5倍的进程大小，即1125。这解决了这个问题。这与纱线记忆无关。不幸的是，在sqoop2中，此异常未得到处理。如果您认为此解释不合适，请随时添加更多答案。

cloudera Oozie sqoop工作挂起永远的心跳心跳

1 个答案: